EvaluATE’s external evaluators at The Rucks Group reviewed the evaluation plans in a random sample of 169 ATE proposals across 14 years. They found that the amount of information in evaluation plans about evaluation data collection increased significantly over time. In this blog post, we share tips about what to do and not to do when describing the evaluation data collection plan for the ATE project you are proposing to NSF.
Describing Your Data Collection Plan: What to Do
The current NSF ATE program solicitation states that ATE evaluation plans should include “the specific data sources, data collection instruments, and methods that will be employed to address the evaluation questions or criteria.” In addition, EvaluATE recommends specifying indicators¾what information or conditions will be used to answer each evaluation question. For example, a survey (a method) is one way of measuring student interested in nanotechnology (an indicator).
Here’s an example of an ATE proposal evaluation plan that includes all the essential information about data collection, including what will be measured, how, and from what sources:
Outcome: Increased enrollment of rural students in certificate program.
Target: 14 new students will enroll in the certificate program each year of the project (three cohorts).
Type of Measurement: Review of student enrollment data provided by registrar’s office
Person(s) Responsible: Project Coordinator
Timeline: Student enrollment data will be analyzed at the end of each semester during all three project years.
This proposal presents each project objective along with corresponding outcomes, targets, types of measurement, persons responsible for collecting data, and a data collection timeline, as shown above for one outcome.
In this example, it’s clear that the indicator is increased enrollment of rural students in certificate program and that the source for that data is the college’s registrar’s office. (In this case, no special data collection methods need to be specified, since the data of interest already exist within the institution.)
Describing Your Data Collection Plan: What Not to Do
A common problem in descriptions of data collection plans is the omission of a description of what will be measured. Suppose an evaluation plan states:
Evaluation methods will include surveys and focus groups with students.
The statement explains how data will be gathered, but it’s not clear what will be measured or why (e.g., students opinions’ of the program, or of their instructors? Their sense of belonging in the program? Their confidence about entering the workforce?) This approach will leave reviewers guessing about what will be investigated and how that will inform your project.
For more tips about describing evaluation plans in ATE proposals, check out the Evaluation Plan Checklist for ATE Proposals and related resources in EvaluATE’s Evaluation Plan Toolkit for ATE Proposals. To learn more about EvaluATE’s review of ATE proposal evaluation plans, view the overview of findings, the scoring rubric, or our article in the American Journal of Evaluation.
Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.