Collecting data for performance measurement: general guidelines
Abstract
Data are the numbers and other raw information obtained through systematic measurement and serve as the “raw materials” for assessing program results. Data collection begins once performance measures have been defined, and methods and instruments have been identified. This effective practice provides general guidelines for data collection and was submitted by Project STAR in March 2007.
Issue
Data collected in an inconsistent way leads to the problem of "garbage in, garbage out." Data collection protocols should be developed and training should be provided for each instrument used, regardless of how simple the instrument may be. In developing the data collection protocol, consider how the data should be collected, who will collect the data, and what data collection training is necessary.
Action
According to Project STAR, keep in mind the following key points when collecting data:- Clearly define data needed. Describe which data to include. If the data will come from another agency, determine if the agency can provide data covering the correct population, locality, and timeframe. For example, if you need statistics on school dropout rates, do you want these statistics for the entire state, or do you need data on dropout rates in your school district?
- Do a dry run. Don't wait until the end of the year to try out data collection processes. For example, collect last semester's grades or last quarter's statistics at the beginning of the program year. This will help you work out the kinks in your data collection process before results are needed. Collecting pre-service data also provides baseline data for assessing change.
- Keep it confidential. Sometimes the data you need are sensitive and therefore are not publicly available. For example, individual student records on attendance, grades, or disciplinary incidents may be available only to school personnel. Work with agency personnel at the planning stage to ensure that information can be obtained in a legal and ethical manner. The agency may share information on individuals after removing names and other identifiers, or in aggregate form. If data will be shared in aggregate form (e.g., dropout rates reported by school district), make sure this format meets your needs.
- Provide information to data sources (persons providing data). In addition to protecting confidentiality, it may be appropriate and advantageous to provide an advance letter of notification to data sources as part of your protocol, especially if you are collecting survey data. The letter should include a brief explanation of: the survey topic and types of questions, who is conducting or sponsoring the survey, the purpose of the survey and how data will be used, why participation is important and any potential benefits, an explanation of any risk involved, how confidentiality will be protected, that participation is voluntary and will not affect service access, and who to contact for further questions.
- Get the timing right. Clearly describe the timeframe for when the data are to be collected (e.g., grades for all four semesters, or just the first and last semester). Many types of data from other agencies, such as grades and crime statistics, may be available only on set schedules. In addition, the people who can provide access to these data (for example, teachers or camp counselors) may not be available at certain times of the year. To avoid timing problems, involve the people who can provide access to the data during the planning process, and consult agency schedules and calendars.
- When appropriate, be sure to collect information on the "dosage" of service received. It is often important to separate individuals who received little or no service from those who received a lot of service. For example, results for students who attend tutoring sessions regularly should be more compelling than results for students who only participate once or twice. Including individuals who received little or no service in your data set can throw off your results and understate the impact of your program.
- Integrate the instrument into the service. Some instruments that provide data on program results, such as goal-setting logs and journals, may also be integral to the process of service delivery. For example, goal setting can serve to plan tutoring sessions as well as measure their impact. Journals can act as a forum for learning writing skills, as well as a data source for measuring changes in skills and attitudes.
- Take advantage of captive audiences. When appropriate, collect data when the audience is most accessible to you. For example, if students have just completed a computer training course, test their skills before they leave.
Context
For each instrument, it is important to identify who will be the best available person to collect the information (e.g. member, staff person, site personnel such as a teacher, supervisor, parent). Many programs have members or program staff collect data and it is important to provide these individuals with information regarding timing and methods of collecting data. If possible, provide training in person and give data collectors a chance to practice filling out forms. Also, provide feedback on the data they collect, to ensure effective data collection practices.
Outcome
- Collecting data in a consistent, systematic manner helps to ensure that data are high-quality and meaningful.
- Implementing data collection of the highest quality provides programs with confidence when reporting whether outcome targets were met.
- When appropriate, improved data collection methods should be developed for the next program year.
Evidence
If data collection protocols meet the highest quality, programs can be assured that the results reported are indeed evidence of program success.