ATE Evaluation Task Validation Study

For EvaluATE to do its job well, we needed a comprehensive and accurate understanding of what ATE evaluation work really entails. To that end, this study identified the core tasks that are conducted as part typical ATE evaluations.

This study built on the work of the American Evaluation Association (AEA) and other organizations that have developed taxonomies of evaluation competencies. The AEA competencies served as the foundation for the ATE evaluation tasks. This study’s findings are helping EvaluATE translate general evaluator competencies to the specific knowledge and skills needed by ATE evaluators and project staff.

We are using the validated ATE evaluation task list to plan and prioritize EvaluATE’s training activities and resource materials. Our aim is to provide guidance for all core evaluation tasks.

Others engaged in evaluation capacity-building and research on evaluation can use our framework as a starting point for similar validation work in other evaluation contexts.

We used two methods in this study:

• Task Tracking: We asked ATE evaluators and principal investigators to track their evaluation-related tasks for one year. We used the task-tracking data to revise the task list to ensure it’s a comprehensive and accurate accounting of what’s involved in a typical ATE evaluation.
• Delphi Study: We asked people who are recognized as experts in evaluation to review and provide feedback on the tasks and their definitions. We used the experts’ feedback to finalize the task list.
• Equity, Diversity, and Inclusion (EDI) Review: Prior to finalizing the task list, our researcher partners from EvaluATE’s Measuring Equity, Diversity, and Inclusion in the ATE Program study review the task list from a DEI lens.

These methods generated evidence to answer the research question: What are the key decisions and actions involved in ATE evaluation?

The research team thanks the following evaluators and principal investigators for their participation in this study: Lola Adedokun, Ann-Claire Anderson, David Antol, Jennifer Bellville, George Chitiyo, Phillip Davis, Amy Germuth, David Hata, Sondra LoRe, Candiya Mann, Craig McAtee, Adam McKee, Megan Mullins, Matthias Pleil, Sandra Porter, David Reider, Lisa Shannon, Julie Shattuck, Pamela Silvers, Keith Sturges, Peggie Weeks, Blake Urbach, and Jill Zande. We thank the following members of the EvaluATE team for their input on draft versions of this document: Lyssa Becho, Emma Binder, Ayesha Boyce, and Tiffany Tovey.