Quality Control

It is important to introduce various mechanisms to check the quality of operations in progress or already carried out as part of an evaluation activity. These should be present at each planned stage as part of the realization of an activity. EduStat software includes some options to establish quality control on data included in databases and examining the rigor and standardisation in administering an evaluation of learning.

This document is a reminder of some mechanisms involved in monitoring the quality of evaluation at the planning stage or already completed.

A. Planning the assessment activity

At the planning phase of the evaluation mechanism, we have to select the techniques and mechanisms to facilitate realization of the evaluation respecting quality. Thus the availability of tables of specification will better ensure the quality of the instruments that we will be used to gather the required information. Always in relation to the quality of data collection instruments, it is desirable and often necessary to make a field testing; analyzes of data collected during a field testing will help to make the corrections necessary to use items and questions tailored to the pursued objectives.

If the evaluation mechanism must be made from a sample representing a population, we must ensure that the chosen sampling plan suits the situation. We know that there are several techniques; we must select what is most suitable for the intended assessment. Initially, it is important to know the evaluated population. We will carefully consider the databases on the affected population. Sometimes it is necessary to establish links between different sources of information in order to provide information that will be required at the time of selection of the sample used for the administration of the assessment.

B. Administration of the evaluation

Ordinarily administering an assessment activity requires the participation of several agents. It is necessary to ensure consistency in the adopted procedures. It will require that administrative instructions are clear; moreover, it is desirable to provide training activities for the benefit of all those involved in administering the mechanism. The presence of observers in a sample of assessed sites (this is usually schools) collects information on compliance with the agreed guidelines.

C. Treatment of information

Following the administration of the assessment, we must ensure the treatment of the collected information. Several activities must be performed during this important stage of a learning evaluation cycle. Sometimes it is necessary to mark some students’ responses. Several techniques can be used to guarantee the quality of this operation; here is the following ones: double marking, group marking, and marking profile. The information gathered will be incorporated into data files that will make up databases. We must ensure that the input activities are free of errors that may influence the quality of data available to perform the required compilations.

D. Statistical Analysis

The statistical analysis used to process the provided information must be relevant to the current assessment. Also, we must make sure that the results are free of miscalculations. It may be useful to use additional techniques to verify the accuracy of the calculations. Ideally it would be appropriate to use the participation of several people to run in parallel similar compilations; this approach provides guarantees linked to the accuracy of compilations performed.

E. Reporting results

The evaluation reports are usually written by experts in the evaluated area. It is possible that these writers are not experts in the interpretation of statistical evidence available following realized compilations. It is prudent to entrust statisticians in the field of learning assessment the task of verifying the conclusions drawn in the reports. It must guarantee the accuracy of comments and statements contained in an assessment report. It may influence the credibility of the entire evaluation process.

F. Summary

Here is a summary of the quality control activities associated to the key steps of an evaluation of learning.

1. Planning the evaluation operation and preparation of instruments	preparation of the evaluation plan preparation of the sampling plan (if applicable) experimentation of the evaluation mechanism review of procedures and instruments backwards translation (if several languages) quality control data collection instruments
2. Administration of the assessment	administrative control (by site visits, for example) training session and drafting administrative instructions
3. Marking and coding information	definition of the rules of codification and marking staff training in coding and marking double marking group marking staff marking profiles
4. Preparation of databases	choice of the type of data entry comparing with a sampled file produce frequency distributions in connection with valid values visual inspection of files (comparison between data saved in the data collection form and the data file) validation of files recodification, if required
5. Realization of statistical analyses	classic item analysis bias analysis calculation of mean scores production of tables of mean scores involving contextual variables complementary statistical analyzes
6. Production and editing reports	identifying target audiences understanding, by writers, of the statistical analysis results linking different data request experts to assist in the interpretation of results use of various techniques to present the results (tables, diagrams, graphs)

Email address: info@mcpythagore.com