ATE Evaluation Judges Advise Alignment

In 2021, EvaluATE started the Outstanding ATE Evaluation Award program. Three exceptional STEM education evaluators served as judges: Cherie Avent, Cynthia Phillips, and Aileen Reid.

Based on their experience selecting last year’s winner, we asked them, “If you could give one piece of advice to ATE grantees and evaluators about evaluation, what would it be?”

As they reflected on their observations of the strengths and weaknesses across the nominated evaluations, a theme emerged: alignment.

Cherie said:

It is important for the intended program / project goals and the evaluation questions to be aligned. The two mutually influence the evaluation conclusions drawn and decision-making within the program.  

In other words, the questions driving an evaluation should support the project’s purpose. If a project aims to enhance teaching, the evaluation questions must go beyond asking about the number of participants and their satisfaction. That is, at least one evaluation question should ask about the project’s influence on changes in teaching practices.

Following up on Cherie’s comments, Cynthia added:

The methods selected to address the evaluation questions also need to be aligned. By “aligned” I mean fit the purpose and support valid and reliable conclusions.  

So the methods should generate data to help answer the evaluation questions and meet technical requirements for evidence-based conclusions. For example, if an evaluation question asks how the project contributes to students’ workforce readiness, a valid skill assessment is needed.

Aileen said:

Attention to stakeholders’ values―especially those least well-served by the program―is important throughout all aspects of the evaluation process.

To elaborate, the entire evaluation should reflect what people involved in the project care about. Take time to surface and include perspectives of individuals who may not have had a seat at the table during project or evaluation planning.

Evaluations are essentially a service to the clients who commission them. In the ATE context, those clients are ATE principal investigators and their project teams. In serving their clients, evaluators are professionally obligated to draw on their experience and expertise to ensure and strengthen an evaluation’s quality. That may involve guiding clients to embrace evaluation questions they hadn’t thought of; prompting them to engage people they may have overlooked; and balancing their needs and interests with requirements for technical quality.

While attending to the alignment of evaluation purpose, methods, and values, evaluators must also navigate the practical limitations of time and budget. But alignment doesn’t require extra time or money. In fact, it can enhance efficiency by focusing resources on what matters most.

Alignment shines in the 2021 award-winning evaluation. The judges praised its use of mixed methods and clear alignment of conclusions to data. The report’s organization, clear writing, graphic design, and data visualization support the evaluation’s purpose and help ensure that the client and other users clearly understand the results and how they were derived.

Submissions for the 2022 Outstanding ATE Evaluation Award are now open.