Lori Wingate
Western Michigan University
What problem does this study address?
This study helps inform whether EvaluATE is achieving its ultimate mission of improving evaluation capacity within NSFs ATE program. Before this study, we relied solely on self-reported data to measure if evaluation capacity was improving. We didn’t have a direct measure of whether evaluation practice was changing.
Why is this study important?
This study examines whether evaluation capacity within NSF’s ATE program has improved over time. This study is unique within the field of evaluation in two ways. First, it is one of the few empirical evaluations of evaluation capacity-building efforts. Second, this study looks at artifacts of evaluation practice to determine whether practice changed, rather than relying on self-reported data, which is more commonly done. This study provides evidence of EvaluATE’s impact on the ATE community, as well as serves as a model for conducting research on evaluation using artifacts of evaluation practice.
How will EvaluATE (or others) use the study findings?
We will share study findings with our project stakeholders as evidence about whether EvaluATE is fulfilling its ultimate mission of improving evaluation capacity within NSF’s ATE program. Data collected as part of this study (project descriptions from ATE proposals) will be de-identified and used as educational resources.
How are the researchers conducting the study?
We used a scoring rubric to determine the degree to which key evaluation elements were present in project descriptions of ATE proposals. See our scoring rubric here.
The research questions that guided this study were:
- How, if at all, have characteristics of NSF ATE proposal evaluation plans changed over time?
- To what extent, if any, has EvaluATE influenced changes in NSF ATE proposal evaluation plans?
Research Team
Takara Tsuzaki
Western Michigan University
Lana Rucks
The Rucks Group
Kelly Robertson
Western Michigan University
Carla Clasen
The Rucks Group
Jeremy Schwob
The Rucks Group
Michael FitzGerald
The Rucks Group
Study Findings
Thinking outside of the self-report: Using evaluation plans to assess evaluation capacity building
In this study, we investigated the impact of the evaluation capacity building (ECB) efforts of an organization by examining the evaluation plans included in funding proposals over a 14-year period. Specifically, we sought to determine the degree to which and how evaluation plans in proposals to one National Science Foundation (NSF) program changed over time and the extent to which the organization dedicated to ECB in that program may have influenced those changes.
EvaluATE is supported by the National Science Foundation under grant number 2332143. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.