Using Student Survey and Placement Data in (cue drums) The Real World

VARIED ANNOUNCER VOICES: This is the true story … of two institutions picked to work together on a grant … and have their project evaluated… Find out what happens … when things stop being simple … and start getting real.

Nostalgic TV references aside, collaboration is exciting, but it challenges evaluators when multiple institutions have distinct populations and varying capacities for collecting and organizing data. Add a tight budget and staff turnover to that equation, and things can get dicey.

Evaluators can address these challenges by establishing an institutional data framework early on, coordinating demographic data collection for surveys, and effectively using data parties-collaborative sessions for analyzing data.

Build an Early Data Template

Create a standardized workbook template-an updatable, interactive dashboard-early in the process. With a template in place, institutions can align how student placement and survey data will be collected and shared.

As teams use this tool together, they can work through what data fields are needed and decide on mutually intelligible terms. The template structures the project’s data language for all parties, allowing institutions to pull comparable data for outcome evaluation purposes.

And both institutions and evaluators can monitor student placement trends and survey data on student experiences throughout the program.

Collect Demographic Data for Equity

Including demographic data is important for assessing equitable outcomes. Tracking factors like race, gender, and first-generation status can help uncover disparities, opportunities for improvement, and positive impacts. In one project, tracking demographic data in the survey for a global learning initiative revealed greater overall positive impacts for students of color.

By collecting this data, institutions can:

• Identify Disparities: Splicing student placement and survey results by demographic and academic factors allows institutions to see whether certain groups are facing unique challenges or experiencing intended program impacts differently.
• Ensure Equitable Results: By including demographic data, institutions can assess whether their programs produce fair and beneficial outcomes and uncover evidence for promising practices that improve student experiences.

Host Data Parties for Insights

Data parties bring program teams together, along with institutional research staff and evaluators, to discuss the data, ask critical questions, and identify key insights that might otherwise go unnoticed.

For example, during a recent data party, the overall data seemed to show little program impact. However, the institutional team shared that students in more rigid academic programs (or majors) had fewer choices for additional engagement. This informed how program outcome data could be examined for a more nuanced understanding.

Data parties can encourage:

• Collaborative Problem-Solving: Program teams, institutional data staff, and evaluators can identify gaps in the data or other potential data collection issues.
• Ongoing Improvement: Insights gained can inform mid-program adjustments, ensuring that program elements are targeted where they are most needed to improve outcomes for all students.

Final Thoughts

“The Real World” of program evaluation with multiple institutions and any number of other potential challenges is messier than any carefully crafted template or dashboard could account for. However, by beginning with helpful tools early on, evaluators and grant teams can start speaking a common language to problem-solve and adapt for high-quality program evaluations.