EvaluATE’s external evaluators at The Rucks Group reviewed the evaluation plans in a random sample of 169 ATE proposals across 14 years. They found that across the years, ATE proposals consistently conveyed key information about the focus of their evaluations. In this blog post, we share tips about what to do and not to do when describing the evaluation focus for the ATE project you are proposing to NSF.

Identifying Your Evaluation’s Focus: What to Do

The current NSF ATE program solicitation directs proposers to identify the aspects of their projects that will be evaluated. It explains that this focus “is best expressed as evaluation questions that frame the evaluative inquiry or criteria that define the dimensions of project quality that will be investigated.” The solicitation also states that the evaluation “should address both project implementation and outcomes.”

While an evaluation’s focus can be expressed in a variety of ways, EvaluATE recommends using evaluation questions for maximum clarity. Here are some tips for writing evaluation questions in ATE proposals:

  • List key questions—ideally, about three to seven—that the evaluation will address.
  • Include questions about both project implementation (what the project does) and outcomes (what changes it brings about).
  • Ensure that the questions align with the project’s goals and activities as described in the proposal.
  • Ensure that the questions address the project’s intellectual merit (contributions to advancing knowledge) and broader impact (contributions to the betterment of society).

Including details about your evaluation’s focus demonstrates you’ve thought carefully about the evaluation’s focus. It also helps reviewers put the rest of the information in the plan into context.

Here’s an example from a proposal that includes all the key information about an evaluation’s focus:

Formative evaluation (which includes implementation and process evaluations) will examine the development of the program (including internship, faculty mentor program, summer institute) and will be used to identify strengths and weaknesses. Formative questions will include: (1) To what extent do the activities and strategies match those described in the plan? (2) To what extent were the activities conducted according to the proposed timeline? … Summative evaluation (outcome or impact evaluations) will examine what the project has actually accomplished in terms of its stated goals. Summative evaluation questions will include the following: (1) To what extent did the project meet its overall goals? (2) Was the project equally effective for all participants?

These details about the evaluation’s focus show that the proposers thought deeply about where and how they would concentrate their evaluation efforts. This group specifically notes their evaluation will assess both project implementation and outcomes, showing responsiveness to the solicitation’s guidance.

Identifying Your Evaluation’s Focus: What Not to Do

EvaluATE’s study found that sometimes ATE proposers make general statements about their evaluation’s focus, omitting details specific to their project.

Here’s an example of this common mistake:

The assessment process will include both formative and summative evaluation utilizing quantitative and qualitative data sources.

This description does not specifically identify aspects of the project that will be evaluated. A good rule of thumb to follow is this: If a statement that describes an evaluation’s focus could appear in any other proposal and work just as well, it is too broad. A description of an evaluation’s focus should be tailored to your project’s goals, activities, and intended outcomes.

Resources

For more tips about describing evaluation plans in ATE proposals, check out the Evaluation Plan Checklist for ATE Proposals and related resources in EvaluATE’s Evaluation Plan Toolkit for ATE Proposals. To learn more about EvaluATE’s review of ATE proposal evaluation plans, view the overview of findings, the scoring rubric, or our article in the American Journal of Evaluation.

About the Authors

Kelly Robertson

Kelly Robertson box with arrow

Principal Research Associate, The Evaluation Center, Western Michigan University

Kelly has a Ph.D. in evaluation and more than eight years of experience in the field of evaluation. She works as a Senior Research Associate at The Evaluation Center at Western Michigan University. Dr. Robertson has worked on evaluations at the local, regional, national, and international levels, spanning a wide variety of sectors (e.g., STEM education, adult education, career and technical education, and evaluation capacity development). Her research interests primarily focus on evaluation as it relates to equity, cultural competence, and making evaluation more user-friendly.

Lori Wingate

Lori Wingate box with arrow

Executive Director, The Evaluation Center, Western Michigan University

Lori has a Ph.D. in evaluation and more than 20 years of experience in the field of program evaluation. She is co-principal investigator of EvaluATE and leads a variety of evaluation projects at WMU focused on STEM education, health, and higher education initiatives. Dr. Wingate has led numerous webinars and workshops on evaluation in a variety of contexts, including CDC University and the American Evaluation Association Summer Evaluation Institute. She is an associate member of the graduate faculty at WMU.

Creative Commons

Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Related Blog Posts

Nation Science Foundation Logo EvaluATE is supported by the National Science Foundation under grant number 1841783. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.