Rather than thinking in terms of “how do we evaluate this project?” we recommend starting by asking what evaluative information will help the project reach its goals and achieve meaningful outcomes. For example, if the ultimate product of your work is going to require industry and/or college buy-in and input, the evaluation should attend to these matters early by comparing the plans for the product to industry standards.
Second, you should make sure you are clear about who the audience for the evaluation is—is the evaluation intended to serve project development or to convince National Science Foundation or industry stakeholders of the project’s merit—or will it have multiple purposes/audiences? If the point of the evaluation is to feed you information so that you can improve the project, the evaluation may look more at your process and provide you recommendations. If the intent is that the evaluation serve as an overall judgment of your efforts, it will focus more on your outputs and provide a summative statement about the project.
We think it is useful to overlay the evaluation question(s) onto the project’s logic model. A logic model enables you to plot the project’s inputs/resources, outputs, and goals and convey how the components link together. It’s a graphic way to plot what you hope and expect to happen in the course of your work, given what you have to work with. Our logic model template is designed to help ATE projects and centers get started with logic model development. A logic model can be elaborated by adding arrows to show how particular inputs and activities are expected to lead to certain outputs and outcomes.
To ensure the project is really evaluated and not just described, we recommend establishing success indicators for each output or outcome in your model (resources may dictate how much you can do here). Success indicators can be both a target for your work and a comparison you’re your evaluation. Evaluators often go a step further than establishing indicators by determining how data can be collected to inform the project’s progress toward each indicator. We have found it helpful to complete a indicator worksheet, detailing each indicator, its corresponding data, where the data will come from, and who will be responsible for obtaining it.
Clearly, you will have to make choices about where to focus your evaluation resources. Our suggestion is that you highlight the paths in the logic model that will answer your most pressing questions and focus on those data collection methods that will yield the most useful and credible evidence to determine the success of the program or provide information that will aid in making improvements during implementation.
Check out these other resource for guidance in designing your evaluation:
Evaluation Design Checklist (Daniel Stufflebeam)
Evaluation Plans & Operations Checklist (Daniel Stufflebeam)
Evaluation Questions (Kellogg Foundation)
Planning a Program Evaluation (University of Wisconsin-Extension)
The Evaluation Plan (Kellogg Foundation)
