Evaluation Process

  

 

 

  

In order to consider the various activities taking place throughout an evaluation activity, it is important to understand the whole process it involves. We present some aspects related to the approach linked to learning evaluation activity; here are some aspects:

 

  • Types of assessment
  • Stages of an evaluation cycle
  • Information processing using appropriate software

A.   Types of Assessment

 

There are several types of evaluation to better reflect the function that we want to consider with the establishment of an evaluation mechanism:

  • formative assessment to assist, support and guide the learning in order to achieve the objectives;
  • summative assessment for deciding the certification of learning or promotion;
  • assessing the school system in order to measure its performance or another aspect of its operation or its performance.

It should be noted that an evaluation process has three components:

  • the measurement;
  • the judgement;
  • the decision.

The first component of an evaluation mechanism (the measurement) will identify, collect and produce relevant and valid information. This activity is performed using instruments adapted to the objectives of the evaluation process.

 

Secondly, school officials interpret (the judgment) the results achieved according to the established criteria or by reference to performance obtained in this evaluation activity, previously or in other education systems.

 

Finally, officials have to choose the consequent actions to select (the decision). The results of the evaluation have to be helpful to the different agents of the school system. Note that in terms of diagnostic assessment and summative assessment, the results directly affect evaluated people. The results obtained in a formative evaluation itself or the one concerning an entire school system will mainly be associated to pilot activities.

 

B.   Stages of an evaluation cycle

 

We can describe an evaluation activity in relation to different stages of its development. We propose the following four steps:

 

  1. preparation of the assessment activity;
  1. data collection;
  1. data entry using suitable support;
  1. data analysis.

 

We present the operations to be performed in relation to each stage.

 

  1.   The preparation of the assessment activity

During this step, we perform the following operations:

 

  • The determination of the objectives of the operation – We have to know what we want or what we must assess with the mechanism chosen and for what purpose. Following the operations depend on.
  • The definition of the design – Before preparing the instrumentation to be used, we must specify its internal organization.
  • Preparation of instruments – Instruments for collecting data must meet a number of criteria to ensure the rigor of the operation.
  • Experimentation of mechanism – An experiment on a limited sample will help to adjust instruments and administrative procedures.

 

  2.   Data Collection

Here are the main operations related to data collection for completing the evaluation:

 

  • The definition of the population or sample choice – It is important to clarify the population to be evaluated. If the operation must be carried out using a sample, we have to ensure that it is representative of the population that will be evaluated.
  • Contact schools – Different persons must be informed on the operation taking place in their organisations. It is also important to inform and sometimes train staff associated to the evaluation.
  • The administration of instruments – We must determine who will be the person in charge of data collection. Administrative procedures should be exactly the same from one place to another.
  • Control of administrative procedures – It is necessary that the national authorities ensure that the adopted procedures are respected by all. Control mechanisms should be provided for this purpose.

 

  3.   The data entry on an appropriate support

In order to conduct an assessment of learning, several technical operations must be performed in the processing of the data:

 

  • Codification and marking responses – If the type of instruments must be filled, we must make the marking and codify this information.
  • Data entry – Information provided by students is then input using a suitable support for data processing.
  • Preparation of data files – Usually, the data is processed using computer tools. Also, we must make the data format compatible to the tools that will be used in statistical compilations.
  • The data quality control – Managers must provide mechanisms to note the quality of data used to achieve the statistical analysis.

 

  4.   Data Analysis

Data analysis turns out to be a milestone in the realization of an evaluation. Following the review of the available information, it will be possible to make appropriate judgments and take better decisions:

 

  • Statistical analysis – This is a very important step that must be carried according to the objectives pursued by the planned evaluation. To achieve statistical compilations, the operation's leaders must have the technical and appropriate tools.
  • Preparation of reports – It is important to prepare reports according to the results of the evaluation as soon as possible and in formats tailored to different target audiences.
  • Dissemination of results – The results of the evaluation should be distributed to the various target audiences (e.g., students, parents, teachers, school principals, officials in the areas concerned and Ministry); this information should be disclosed as soon as possible.
  • The additional analyzes – An assessment often collects a set of information that can be analyzed in more detail in a second time. More specialized research can thus be achieved, without delay the dissemination of the overall results obtained.

 

C.   Information Processing

 

In the processing of the information collected, it is necessary to retain a set of computer tools to carry out the various operations taking place in an assessment of learning outcomes. Some of these tools (especially Excel and SPSS) are sometimes used in pedagogical evaluation process.

 

EduStat software is used since many years for the realization of certain evaluation in education activities. It can be considered as a management and control tool for educational assessment activities. This information tool can be well integrated with other software used for data processing. Thus it is possible to use data stored in databases, "SPSS" or "EXCEL" (among others) for the planned compilations. Moreover, this software can import data or results of compilation made by the software to perform complimentary analysis using a software best suited to particular needs (e.g., SPSS, EXCEL, X- CALIB).

 

We do not believe that it appropriate to describe the set of operations that can be performed with one software. However, we provide a summary linked to major tasks to perform during the administration of an educational assessment:

 

  1. First, it is important to define the specifications or terms of the desired evaluation. There are several techniques to clearly identify the area to look and abilities or skills preferred. It may be mentioned here preparing "specification tables" that will have an impact on the tools for the collection of information (particularly the tests and complementary questionnaires). In addition to know the "field" to analyze the skills or abilities to be examined, must be able to establish the necessary links with the IT tools that will be used to ensure the realization of future technical operations. This is a very important planning step in providing a software including all parameters necessary to carry out other activities.
  1. Before the actual administration of the evaluation mechanism, we may choose samples to be covered. If this is the situation, these samples must comply with recognized standards to be able to make the "inference" statistics from the collected data, that is, the application to the whole population of the characteristics and results obtained using the selected sample. The head of the holding of the assessment activity will have to rely on software including the options for establishing a sampling design respecting the characteristics of the proposed activity and the realization of tasks related to the choice of the appropriate milieu.
  1. When there is administration of the instruments used to gather relevant information, it may be desirable or at best a need for a monitoring mechanism of this activity. If one opts for the integration of quality control activities of the administration, it will provide an information gathering tool collected in this framework and its treatment. This is a complementary instrument to predict; one or more software will then be used to accomplish the tasks of input information and its compilation.
  1. Evaluation of learning sometimes requires the collection of information that should be treated by assessment experts before submitting it to software that can perform in various compilations. This occurs when the response from the student must be "appreciated” by a marking; the result of such a correction or "rating" may take the form of a binary code (correct / incorrect, true / false) or an appreciation in the form of a ranking providing several possibilities (answers that refer to "partial credit"). In such a situation, we need to establish the rating of quality control mechanisms to ensure that the operation performed by human proofreaders meets the criteria. Software comprising data compilation options should be used. It may be mentioned here some mechanisms used to decide on the severity of the notation: "double-marking", "group correction" and establishing "marking profiles."
  1. After preparing the data collection instruments (field testing or proper administration), it is prudent to provide immediately the data entry mechanism that will be used. Most data processors provide the information input options. The selected software should include options to ensure data entry with the features of the instrument used. It must also be able to choose from multiple input modalities of data those suits the specifications of the driving evaluation or based on available resources. It may be mentioned here at least three input options that allow, in varying degrees, to provide a check on this: i) a single entry without validation, that is to say the input data without the intervention of the software to check the validity of the provided information; ii) a single validated entry, that is to say the input data with software intervention when information provided does not respect the values ​​defined as "valid"; iii) a double entry, that is to say the input data by two different operators or even operation at two different times.
  1. Following the entry of the collected data, we will create the databases necessary for the implementation of relevant statistical compilations. Software includes this possibility to save data in relation to a particular computing structure. Several additional works must often be made; we note here the extraction of information, merging databases or data files, creating files in relation to various formats.
  1. When databases are available, it is possible to undertake a series of statistical compilations tailored to the information gathered. The information in the databases usually comes from different sources: i) the assessed student; ii) the teachers of these students; iii) the directors of selected schools; iv) the quality control (administering and scoring). It is desirable to establish links between these different sources of information. It is also necessary to retain analytical techniques adapted to each data set. The successful processing software should contain options of compilations affecting particularly the evaluation of learning and their relationship to information from various sources (e.g., school context, information on students, teachers and principals, opinions or attitudes of respondents). It is also necessary to perform analyzes to decide on the quality of selected instruments (items analysis, analysis of differential item functioning, for example).
  1. We must quickly produce reports that present results. It is important to provide people who have the mandate to draw up these reports the results of compilations. It is usually the production of one or more technical reports that will be used as an information source or by the report writers. The processors of the selected information should produce tables easily integrated in a report. We must avoid capture information that will be integrated in a report; we should be able to get this technical information from productions prepared by a software.
  1. It is necessary to ensure the archiving of the data and information generated under the holding of an evaluation activity. It is usually necessary to consult the used documentation to make a prior evaluation of learning to perform further analysis or for comparison between the results and then those of another evaluation cycle. There should be recording any useful information on a suitable IT support and implement an interrogation procedure for effective consultation.

 

Regardless of the IT tools selected by the official’s in charge of an educational assessment, we should provide training or development activities affecting both the technical aspects of software and on the choice and meaning appropriate statistical indices. As will be seen by looking at other sections of this website, the EduStat software includes options for performing the work described in this document.

 

Email address: info@mcpythagore.com