EvaluATE’s external evaluators at The Rucks Group reviewed the evaluation plans in a random sample of 169 ATE proposals across 14 years. They found that the amount of information in evaluation plans about evaluation data collection increased significantly over time. In this blog post, we share tips about what to do and not to do when describing the evaluation data collection plan for the ATE project you are proposing to NSF.
Describing Your Data Collection Plan: What to Do
The current NSF ATE program solicitation states that ATE evaluation plans should include “the specific data sources, data collection instruments, and methods that will be employed to address the evaluation questions or criteria.” In addition, EvaluATE recommends specifying indicators¾what information or conditions will be used to answer each evaluation question. For example, a survey (a method) is one way of measuring student interested in nanotechnology (an indicator).
Here’s an example of an ATE proposal evaluation plan that includes all the essential information about data collection, including what will be measured, how, and from what sources:
Outcome: Increased enrollment of rural students in certificate program.
Target: 14 new students will enroll in the certificate program each year of the project (three cohorts).
Type of Measurement: Review of student enrollment data provided by registrar’s office
Person(s) Responsible: Project Coordinator
Timeline: Student enrollment data will be analyzed at the end of each semester during all three project years.
This proposal presents each project objective along with corresponding outcomes, targets, types of measurement, persons responsible for collecting data, and a data collection timeline, as shown above for one outcome.
In this example, it’s clear that the indicator is increased enrollment of rural students in certificate program and that the source for that data is the college’s registrar’s office. (In this case, no special data collection methods need to be specified, since the data of interest already exist within the institution.)
Describing Your Data Collection Plan: What Not to Do
A common problem in descriptions of data collection plans is the omission of a description of what will be measured. Suppose an evaluation plan states:
Evaluation methods will include surveys and focus groups with students.
The statement explains how data will be gathered, but it’s not clear what will be measured or why (e.g., students opinions’ of the program, or of their instructors? Their sense of belonging in the program? Their confidence about entering the workforce?) This approach will leave reviewers guessing about what will be investigated and how that will inform your project.
Resources
For more tips about describing evaluation plans in ATE proposals, check out the Evaluation Plan Checklist for ATE Proposals and related resources in EvaluATE’s Evaluation Plan Toolkit for ATE Proposals. To learn more about EvaluATE’s review of ATE proposal evaluation plans, view the overview of findings, the scoring rubric, or our article in the American Journal of Evaluation.
Kelly Robertson 
Principal Research Associate, The Evaluation Center, Western Michigan University
Kelly has a Ph.D. in evaluation and more than eight years of experience in the field of evaluation. She works as a Senior Research Associate at The Evaluation Center at Western Michigan University. Dr. Robertson has worked on evaluations at the local, regional, national, and international levels, spanning a wide variety of sectors (e.g., STEM education, adult education, career and technical education, and evaluation capacity development). Her research interests primarily focus on evaluation as it relates to equity, cultural competence, and making evaluation more user-friendly. She, along with Dr. Lori Wingate, led the development of The Evaluation Center's online training program, Valeo (valeoeval.com).
Lori Wingate
Executive Director, The Evaluation Center, Western Michigan University
Lori has a Ph.D. in evaluation and more than 20 years of experience in the field of program evaluation. She is co-principal investigator of EvaluATE and leads a variety of evaluation projects at WMU focused on STEM education, health, and higher education initiatives. Dr. Wingate has led numerous webinars and workshops on evaluation in a variety of contexts, including CDC University and the American Evaluation Association Summer Evaluation Institute. She is an associate member of the graduate faculty at WMU. She, along with Dr. Kelly Robertson, led the development of The Evaluation Center's online training program, Valeo (valeoeval.com)
Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
EvaluATE is supported by the National Science Foundation under grant number 2332143. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.