EvaluATE’s external evaluators at The Rucks Group examined the evaluation plan sections of 169 randomly selected ATE proposals funded between 2004 and 2017. The study’s main purpose was to investigate EvaluATE’s influence on ATE evaluation practice. However, it also produced some useful insights about ATE evaluation plans.
This blog post is the first in a series of seven in which we share tips about communicating compelling evaluation plans in ATE proposals. Subsequent posts will focus on various aspects of evaluation plans: evaluators and their qualifications, evaluation focus, data collection, data analysis and interpretation, reporting, and evaluation use.
In this installment, we share the big takeaways from the study:
- Over time, ATE proposers dedicated more space to evaluation. In 2004, the mean length of an ATE proposal evaluation section was 0.9 pages; in 2017, it was 1.3¾a notable increase, given that project descriptions are only 15 pages long.
- Over time, ATE proposals included more details about all aspects of their evaluation plans. The most notable (and statistically significant) new details expanded on identifying who would evaluate the projects, describing how data would be collected and analyzed, and describing plans for sharing results.
- In smaller ways, proposals also did a better job, in later years, of explaining what their evaluations would focus on and how they would be used. These changes weren’t statistically significant; however, proposals were already strong in these areas in the earlier years, so there was less room for improvement.
Two independent raters used the following scale to rate the degree to which the evaluation plan elements were present in the proposals: 0 = Missing; 1 = Partially Present; 2 = Mostly Present; 3 = Fully Present. The chart below shows the average ratings of evaluation plans in 2004 and 2017, illustrating change over time.
Check out EvaluATE’s Evaluation Plan Checklist for ATE Proposals to make sure you don’t overlook important evaluation plan elements in your ATE proposal. The ATE Proposal Evaluation Plan Template will help you organize the information efficiently.
If you want to learn more about the study, here are some resources:
- Are you interested in the study’s methods and detailed findings? Read our article in the American Journal of Evaluation. (If you have trouble accessing it, email kelly.robertson@wmich.edu and ask for a copy.).
- Do you want a quick overview of the study’s main findings? Review this one-page snapshot.
- Are you curious about how evaluation plans were rated? Look at the rubric the study used.
Kelly Robertson 
Principal Research Associate, The Evaluation Center, Western Michigan University
Kelly has a Ph.D. in evaluation and more than eight years of experience in the field of evaluation. She works as a Senior Research Associate at The Evaluation Center at Western Michigan University. Dr. Robertson has worked on evaluations at the local, regional, national, and international levels, spanning a wide variety of sectors (e.g., STEM education, adult education, career and technical education, and evaluation capacity development). Her research interests primarily focus on evaluation as it relates to equity, cultural competence, and making evaluation more user-friendly. She, along with Dr. Lori Wingate, led the development of The Evaluation Center's online training program, Valeo (valeoeval.com).
Lori Wingate
Executive Director, The Evaluation Center, Western Michigan University
Lori has a Ph.D. in evaluation and more than 20 years of experience in the field of program evaluation. She is co-principal investigator of EvaluATE and leads a variety of evaluation projects at WMU focused on STEM education, health, and higher education initiatives. Dr. Wingate has led numerous webinars and workshops on evaluation in a variety of contexts, including CDC University and the American Evaluation Association Summer Evaluation Institute. She is an associate member of the graduate faculty at WMU. She, along with Dr. Kelly Robertson, led the development of The Evaluation Center's online training program, Valeo (valeoeval.com)
Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
EvaluATE is supported by the National Science Foundation under grant number 2332143. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.