Communicating a Compelling Evaluation Plan in Your ATE Proposal, Part 2: Evaluation Focus

EvaluATE’s external evaluators at The Rucks Group reviewed the evaluation plans in a random sample of 169 ATE proposals across 14 years. They found that across the years, ATE proposals consistently conveyed key information about the focus of their evaluations. In this blog post, we share tips about what to do and not to do when describing the evaluation focus for the ATE project you are proposing to NSF.

Identifying Your Evaluation’s Focus: What to Do

The current NSF ATE program solicitation directs proposers to identify the aspects of their projects that will be evaluated. It explains that this focus “is best expressed as evaluation questions that frame the evaluative inquiry or criteria that define the dimensions of project quality that will be investigated.” The solicitation also states that the evaluation “should address both project implementation and outcomes.”

While an evaluation’s focus can be expressed in a variety of ways, EvaluATE recommends using evaluation questions for maximum clarity. Here are some tips for writing evaluation questions in ATE proposals:

• List key questions—ideally, about three to seven—that the evaluation will address.
• Include questions about both project implementation (what the project does) and outcomes (what changes it brings about).
• Ensure that the questions align with the project’s goals and activities as described in the proposal.
• Ensure that the questions address the project’s intellectual merit (contributions to advancing knowledge) and broader impact (contributions to the betterment of society).

Including details about your evaluation’s focus demonstrates you’ve thought carefully about the evaluation’s focus. It also helps reviewers put the rest of the information in the plan into context.

Here’s an example from a proposal that includes all the key information about an evaluation’s focus:

Formative evaluation (which includes implementation and process evaluations) will examine the development of the program (including internship, faculty mentor program, summer institute) and will be used to identify strengths and weaknesses. Formative questions will include: (1) To what extent do the activities and strategies match those described in the plan? (2) To what extent were the activities conducted according to the proposed timeline? … Summative evaluation (outcome or impact evaluations) will examine what the project has actually accomplished in terms of its stated goals. Summative evaluation questions will include the following: (1) To what extent did the project meet its overall goals? (2) Was the project equally effective for all participants?

These details about the evaluation’s focus show that the proposers thought deeply about where and how they would concentrate their evaluation efforts. This group specifically notes their evaluation will assess both project implementation and outcomes, showing responsiveness to the solicitation’s guidance.

Identifying Your Evaluation’s Focus: What Not to Do

EvaluATE’s study found that sometimes ATE proposers make general statements about their evaluation’s focus, omitting details specific to their project.

Here’s an example of this common mistake:

The assessment process will include both formative and summative evaluation utilizing quantitative and qualitative data sources.

This description does not specifically identify aspects of the project that will be evaluated. A good rule of thumb to follow is this: If a statement that describes an evaluation’s focus could appear in any other proposal and work just as well, it is too broad. A description of an evaluation’s focus should be tailored to your project’s goals, activities, and intended outcomes.

Resources

For more tips about describing evaluation plans in ATE proposals, check out the Evaluation Plan Checklist for ATE Proposals and related resources in EvaluATE’s Evaluation Plan Toolkit for ATE Proposals. To learn more about EvaluATE’s review of ATE proposal evaluation plans, view the overview of findings, the scoring rubric, or our article in the American Journal of Evaluation.