As seasoned evaluators committed to utilization-focused evaluation, we partner with clients to create questions and data analysis connected to continuous improvement. We stress developmental evaluation[1] to help link implementation and outcome evaluation. Sounds good, right? Well, not so fast.

Confession time. At times a client’s attention to data wanes as a project progresses, and efforts to actively engage clients to use data for continuous improvement do not generate the desired enthusiasm. Although interest in data re-emerges as the project concludes, that enthusiasm seems more related to answering “How did we do?” rather than exploring “What did we learn?” This phenomenon, depicted in the U-shaped curve in Figure 1, suggests that when data may have great potential to impact continuous improvement (“the Messy Middle”), clients may be less curious about their data.

To address this issue, we revisit Stufflebeam’s guiding principle: the purpose of evaluation is to improve, not prove.[2] Generally, clients have good intentions to use data for improvement and are interested in such endeavors. However, as Bryk points out in his work with networked improvement communities (NIC),[3] sometimes practitioners need help learning to improve. Borrowing from NIC concepts,[4] we developed the Thought Partner Group (TPG) and incorporated it into our evaluation. This group’s purpose is to assist with data interpretation, sharing, and usage. To achieve these goals, we invite practitioners or stakeholders who are working across the project and who have a passion for the project, an interest in learning, and an eagerness to explore data. We ask this group to go beyond passive data conversations and address questions such as:

  • What issues are getting in the way of progress and what can be done to address them?
  • What data and actions are needed to support sustaining or scaling?
  • What gaps exist in the evaluation?

The TPG’s focus on improvement and data analysis breathes life into the evaluation and improvement processes. Group members are carefully selected for their deep understanding of local context and a willingness to support the transfer of knowledge gained during the evaluation. Evaluation data has a story to tell, and the TPG helps clients give a voice to their data.

Although not a silver bullet, the TPG has helped improve our clients’ use of evaluation data and has helped them get better at getting better. The TPG model supports the evaluation process and mirrors Englebart’s C-level activity[5] by helping shed light on the evaluator’s and the client’s understanding of the Messy Middle.

 

 


[1] Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.
[2] Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education.
[3] Bryk, A., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Education Publishing.
[4] Bryk A. S., Gomez, L. M., & Grunow A. (2010). Getting ideas into action: Building networked improvement communities in education. Stanford, CA: Carnegie Foundation for the Advancement of Teaching. Also see McKay, S. (2017, February 23). Quality improvement approaches: The networked improvement model. [blog].
[5] Englebart, D. C. (2003, September). Improving our ability to improve: A call for investment in a new future. IBM Co-Evolution Symposium.

About the Authors

John Cosgrove

John Cosgrove box with arrow

Senior Partner, Cosgrove & Associates

John Cosgrove is a senior partner with Cosgrove & Associates. Mr. Cosgrove has extensive evaluation experience, and Cosgrove & Associates is currently leading the evaluations for a number of external grants, including efforts with the Department of Labor, National Science Foundation, Department of Education, and Department of Agriculture. In addition, Mr. Cosgrove’s emphasis on collaborative evaluation helps breathe life into evaluation processes and helps ensure faculty and staff have the information they need to explore promising practices and use lessons-learned for continuous improvement. Specific areas of expertise include developmental and utilization-focused evaluation, institutional research and strategic planning, development of user-friendly decision-support data systems, return on investment analysis, career pathway development, and experiential and active learning instructional modes. Mr. Cosgrove is committed to social justice and efforts to enhance equity in student outcomes for all students.

Maggie Cosgrove

Maggie Cosgrove box with arrow

Senior Partner, Cosgrove & Associates

Maggie Cosgrove is a senior partner with Cosgrove & Associates. Ms. Cosgrove has deep experience in evaluation methods, policy analysis, grant management, and community college impact. Her experience includes evaluating Department of Labor, National Science Foundation, Department of Education, and Department of Agriculture grants. Ms. Cosgrove has a proven track record of providing excellent training and customer service to college partners. Specific areas of expertise include developmental and utilization-focused evaluation, developmental education redesign, development of career pathways, experiential and active learning instructional modalities, return on investment analysis, and employer and community stakeholder engagement. Ms. Cosgrove is committed to social justice and efforts to enhance equity in student outcomes for all students.

Creative Commons

Except where noted, all content on this website is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Related Blog Posts

Nation Science Foundation Logo EvaluATE is supported by the National Science Foundation under grant number 2332143. Any opinions, findings, and conclusions or recommendations expressed on this site are those of the authors and do not necessarily reflect the views of the National Science Foundation.