EvaluATE’s external evaluators at The Rucks Group examined the evaluation plan sections of 169 randomly selected ATE proposals funded between 2004 and 2017. The study’s main purpose was to investigate EvaluATE’s influence on ATE evaluation practice. However, it also produced some useful insights about ATE evaluation plans.
This blog post is the first in a series of seven in which we share tips about communicating compelling evaluation plans in ATE proposals. Subsequent posts will focus on various aspects of evaluation plans: evaluators and their qualifications, evaluation focus, data collection, data analysis and interpretation, reporting, and evaluation use.
In this installment, we share the big takeaways from the study:
- Over time, ATE proposers dedicated more space to evaluation. In 2004, the mean length of an ATE proposal evaluation section was 0.9 pages; in 2017, it was 1.3¾a notable increase, given that project descriptions are only 15 pages long.
- Over time, ATE proposals included more details about all aspects of their evaluation plans. The most notable (and statistically significant) new details expanded on identifying who would evaluate the projects, describing how data would be collected and analyzed, and describing plans for sharing results.
- In smaller ways, proposals also did a better job, in later years, of explaining what their evaluations would focus on and how they would be used. These changes weren’t statistically significant; however, proposals were already strong in these areas in the earlier years, so there was less room for improvement.
Two independent raters used the following scale to rate the degree to which the evaluation plan elements were present in the proposals: 0 = Missing; 1 = Partially Present; 2 = Mostly Present; 3 = Fully Present. The chart below shows the average ratings of evaluation plans in 2004 and 2017, illustrating change over time.
Check out EvaluATE’s Evaluation Plan Checklist for ATE Proposals to make sure you don’t overlook important evaluation plan elements in your ATE proposal. The ATE Proposal Evaluation Plan Template will help you organize the information efficiently.
If you want to learn more about the study, here are some resources:
- Are you interested in the study’s methods and detailed findings? Read our article in the American Journal of Evaluation. (If you have trouble accessing it, email firstname.lastname@example.org and ask for a copy.).
- Do you want a quick overview of the study’s main findings? Review this one-page snapshot.
- Are you curious about how evaluation plans were rated? Look at the rubric the study used.
Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.