Evaluation Process

Evaluation Process and EduStat

This document summarizes the steps usually adopted for the establishment of an educational assessment. Relative to each identified step, this document presents the tasks that can be performed by the EduStat software.

I.      Definition of the objectives of the assessment activity

Conducting research, study, investigation, survey or educational assessment is held to confirm or disprove assumptions by school officials. Instruments prepared as sources of consulted information must be able to answer the questions then raised. It is at this stage that officials inventory the sources of available information and the most appropriate information collection methods. We must take into account the available resources.

Managers must, from the start of discussions on whether to conduct an evaluation activity, consider the technical aspects of such an operation. This approach can avoid doing any work that will prove unhelpful or impossible to carry it out. Among the technical aspects to consider, it is good to include the software available to perform operations related to data entry, the preparation of files, quality control, and statistical analysis. We must also consider the equipment available to accomplish these tasks. EduStat is a software that can assist in achieving a set of operations related to the execution of an assessment, survey or research.

II.     Definition of the Work Plan

Before preparing the instruments that will be used to gather information, we must clarify its internal organization. Usually it is possible to group the information gathering according to some dimensions. It can also be helpful to connect these dimensions to parts or areas of analysis. For example, items of a maths test could be attached simultaneously to dimensions reflecting the related content and dimensions describing skills or competencies.

The database definition is an important step for the further use of information processing software (EduStat, for example). The study design must be included in the software. There are few limits on the number of dimensions to be included in such design. If using EduStat, these dimensions can be grouped according to three different components. Thus, the same question can be linked to three different dimensions (e.g. affected areas, skills or competencies concerned, objectives).

III.    Preparation of Data Collection Instruments

It is possible that persons decide to use multiple data collection instruments. Questions must meet a set of criteria. Moreover, several types of questions can be retained such as multiple choice or short answer. It may be helpful to use items banks when preparing the information collection instruments.

EduStat helps provide the parameters characterizing each question or requested information: the type of information collected, the type of selected issues, valid answers, the correct answer if necessary, an identifier, a brief description of the information collected. This information is provided during the development of databases and used to validate the data collected during the administration. It is also possible to create and run items banks or question banks with EduStat.

IV.   Field Testing

Usually the instruments drawn up to conduct an assessment or survey are checked with a limited sample. To perform testing, operations usually conducted as part of an evaluation will include: administration of questionnaires, coding if required, data entry, creation of data files, statistical compilations. Given the results, it is often necessary to make changes to the instruments or to the rules governing the conduct of the operation.

EduStat can be used for various operations related to the field testing of instruments. Thus it is possible to use the input options to enter the information. The utilities included in EduStat may be useful to perform a set of operations on data files: data extraction, file merging, sorting of information, backup. However, EduStat is particularly useful for the statistical analysis of data. Three options are well suited here: frequency distributions and through item analysis and bias analysis. The statistics produced by these options will detect problems that may be related to the formulation of questions.

V.    Completion of the Instrumentation

Given the results of the field testing, the instruments should be amended accordingly. It is also possible that it is necessary to also make changes to the procedures governing the conduct of the operation.

We need to update the databases describing the instruments used. It can be put aside some information, to add or make changes. EduStat includes specialized editors that can perform these operations in order to have databases reflecting changes.

VI.   Sampling

An assessment or survey is often achieved with a sample. If this is the case, we have to make sure that the sample meets certain criteria such as representativeness and size to make compilations for statistical inference.

The EduStat software contains options to perform the various operations required to determine a sample: calculating the sample size, the creation of the sampling frame, the choice of sampling design, selection of sample, replacing cancellations, registration of participation, weighting the production of various reports.

VII.  Contact with Persons in Charge of Evaluation

Managers should usually inform people who will be involved in the assessment or survey. Such contact helps to ensure participation in the administration of the mechanism, in particular data collection.

Some sample options in EduStat can produce different lists that can be used to contact the environment targeted by the operation. EduStat can also produce various lists from the information in the database.

VIII. Administration of the Instrumentation

The administration of the instrumentation is often entrusted to third parties. For example, a learning evaluation instrument could be administered by staff working in the school environment.

The gathering of information can be realized by those responsible for administering a survey. For example, if a survey is performed by interviews (telephone or other), EduStat can then be used for the entry of information provided by respondents.

IX.   Coding and Marking

Completed questionnaires should be checked to ensure compliance with prescribed guidelines. It may happen that some information must be coded prior to analysis with a treatment program. This occurs mainly when the instrumentation includes constructed response questions or inviting the respondent to provide comments. Moreover, it is possible that a test includes questions other than multiple choice, such as those in "partial credit". In this case, we must correct answers using the instructions provided in order to ensure the rigor of the operation.

If there are coding some responses, these codes can only be set during this operation. It will then make the necessary adjustments to EduStat databases in order to have all the expected codes when the data files are validated. Such flexibility is offered by EduStat. In addition, the software enables the use of several techniques that can exercise control of the quality of the marking.

X.    Data Entry

Several techniques can be used to enter data collected using a questionnaire or test. Sometimes, leaders of a study, assessment or survey entrust a firm specialized in data entry. In this case, we must specify the entry rules attached to the used instruments. It is also possible that some instruments can be "read" by an optical reader. This is a procedure that minimizes errors due to typing and that accelerates the process. Finally, managers may wish to perform this operation themselves. Care must be taken to select a suitable mechanism.

EduStat has options to enter data: the double entry, single entry or single entry with validation during the data input. Moreover, multiple formats of presentation of input windows are available: all objects on the same window, displaying a limited number of variables in a single window, the display of all available information on each variable. If we wish to use scanning, EduStat includes options that assist in the preparation of codification of records that can be "read by computer."

XI.   Preparing Files

After entering the collected information using a questionnaire or a test, it is necessary to prepare the data files that will be used by a computer program for the purpose of making compilations. For example, it is possible that those responsible for the data entry have made multiple files. We must merge these different files to perform the analysis.

EDUSTAT contains a set of utilities to perform operations on data files. This is, among other things, sorting, merging, extraction and recodification of certain information.

XII.  Quality Control

It is important to use mechanisms to ensure quality control on all administration procedures and data collected through a survey or evaluation. This often guarantees the credibility of the results. Control procedures should be provided concerning both the administration of the mechanism and the activities related to the marking or coding of responses. We must also ensure that the created databases contain valid data. Finally, it is necessary to certify the accuracy of statistical compilations done.

Among the checks to be performed to ensure quality control, it appears important to confirm that the data files are consistent with the facts and that compilations are free of calculation errors. Several strategies can achieve such control. The EduStat software offers several affecting both data entry and the consistency of the marking of the answers.

XIII. Statistical Analysis

The compilation of the collected data is a very important step in the completion of a study, survey or evaluation. The results will confirm or invalidate the assumptions made at the beginning of the process. However, we must retain the statistical analysis techniques adapted to the operation and data type.

It is often necessary to use several computer programs to perform all analysis that can provide answers to questions raised by the research. For its part, EduStat allows for statistical analysis at the first level. Thus it is possible to compile frequency distributions. When the project includes administering a test, this software allows performing items analysis and calculate proportions of success. It also allows to draw crosstabs involving two sets of variables or average results. EduStat may produce some graphic illustrations. This program also facilitates linkages with other specialized software when required.

XIV. Preparing Reports

At the time of preparation of reports detailing the conducted study, it is often important to perform compilations to verify either hypothesis resulting from a first set of statistical analyzes. The leaders therefore must have flexible tools for achieving these additional treatments. Moreover, it may be useful to have statistical analysis results easily integrated in electronic form to the report in preparation. Such an approach may avoid errors due to transcription while decreasing production time.

EduStat may generate additional information from available information. It is possible to test new hypotheses set after the first statistical analysis. Moreover, the reports produced by EduStat are in electronic form readily usable by a word processor, desktop publishing or spreadsheet.

Email address: info@mcpythagore.com