When I assumed the PI-ship of the Scenario-Based Learning Project in 2006, I had worked closely with the prior PI as the project’s instructional designer, knew I enjoyed the ATE community, shared their vision of an innovative 21st century technician workforce, and had management and teaching experience. What more was there to know? A lot, as it turned out. It quickly became apparent to me that the world of ATE projects and centers was a different place than any I had worked in before.

When I took over as PI on the second grant proposal, the former PI suggested we use the evaluators from a previous non-ATE grant she had led. Big mistake. Those evaluators reported to my National Visiting Committee (NVC) during our initial committee meeting that they didn’t have any results to report because they did not plan to collect data until the end of the project year. My NVC was not happy. I was not happy. My stakeholders were not happy.

How did this happen? In my naivety, I didn’t even discuss the evaluation with the evaluators beyond an initial outline of a plan involving questions to be answered by the evaluation—I thought the evaluators knew what they were doing because they were evaluators. I didn’t understand the complex nature of the profession of evaluation. Since then I have joined the American Evaluation Association, attended their annual conference, and regularly attend EvaluATE’s webinars. I made a mistake, learned from it, and the project improved.

I quickly learned that some evaluators and funders are all about the summative report. The project said they would do A, here is the data to show they did or did not do A. End of report. In contrast, the ATE program is interested in how we are doing as we progress through our work. Formative reports from the evaluator serve as a check-in on where you are in your work plan and outcomes. Your evaluator needs to be a critical friend—an advisor who keeps a distance and is critical where needed yet still supportive with ideas, solutions, and contacts.

Choose an evaluator with ATE experience and expertise in collecting and analyzing the kind of data you will need. Confirm who will collect the data, how it will be collected, from whom, and when early in your discussions with your evaluator and project team. Confirm that your evaluator is willing to provide mentoring to your data collection team as needed if you decide to collect the data yourselves and have the evaluator do the analysis (saves money but requires time).

You might need interim reports/summaries from your evaluator for meetings with stakeholders, your NVC, advisory boards, the ATE annual survey, and your annual and final reports. It is a good idea to align your data collection with your reporting needs to best use your resources.

Learn more about the process of evaluation every chance you get. Choose an evaluator with the expertise appropriate to your project or center. Think of your evaluation as a resource and your evaluator as an ally to help you and your team to create the best project or center possible.

About the Authors

Jane Ostrander

Jane Ostrander box with arrow

Director, Experiential Learning Center, Truckee Meadows Community College

Dr. Jane Ostrander is Principal Investigator for the National Science Foundation Advanced Technological Education (NSF ATE) project Destination: Problem-Based Learning (PBL) (DUE#1161352), and Director of the Experiential Learning Center at Truckee Meadows Community College, Reno, NV. Ostrander serves on the Community College Liaison Panel for the ATE EvaluATE Center and as a Mentor for the Mentor-Connect for Leadership Development and Outreach Project. Her research interests include PBL, faculty professional development, online knowledge sharing in communities of practice, and social psychological interventions for transformative change. Prior to her project work Ostrander taught computer literacy, web site design, project management, and business.

Creative Commons

Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Related Blog Posts

Nation Science Foundation Logo EvaluATE is supported by the National Science Foundation under grant number 1841783. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.