How many students participated in various ATE project activities in 2018 and 2019, as reported on the annual survey of ATE principal investigators?
ATE projects reported direct service to over 60,000 students in 2018 and 70,000 students in 2019. However, due to the structure of the questions on the annual survey of ATE principal investigators, we knew that these might not be unique counts of students. Therefore, we conducted case studies of selected ATE projects to investigate the extent of duplication in student counts.
How many unique students participated in any ATE project activities in 2019, based on a sample of ATE projects?
The nine ATE projects included in the case study sample reported serving 4,060 unique students in 2019, while these same projects’ ATE Survey responses for that year included 4,633 students. The number of unique students served as a percentage of the count reported on the ATE Survey ranged between 40% (indicating the majority of the students were counted more than once on the ATE Survey) to 121% (indicating PIs were able to identify additional students who were not counted on the ATE Survey). In total, 88% of students reported on the ATE Survey were verified as unique counts by the case study participants.
To what extent and how did the numbers of apprenticeships, internships, and other applied learning opportunities offered by employers in collaboration with ATE projects change across the lifetime of the ATE program?
Given the increased interest in applied learning and workplace-based learning, this report looked closely at the number of projects engaging in these types of activities and at the number of students such activities served. A review of project abstracts indicates that applied learning experiences were more prevalent in 2018 and 2019 than in previous years of the ATE program, with approximately 40% of funded ATE projects intending to engage in applied learning compared to a prior average of 23% of projects.
How many students who participated in ATE-funded academic programs successfully completed their educational programs in 2018 and 2019, as reported on the annual survey of ATE principal investigators?
On the ATE Survey, projects reported 710 students who completed ATE-funded academic programs in 2018 and 1,670 students in 2019.
How many students who began ATE-funded academic programs obtained a marketable credential (e.g., certificate, license, associate degree) from the program or another in a related field in 2019, based on a sample of ATE projects?
Eight of the nine ATE projects included in the case study sample reported a total of 341 unique students who obtained credentials in 2019. The majority of students obtained industry certificates (87%), followed by associate degrees (33%). The ninth project could not track student credentials.
If the ATE program wanted to report information on student participation and completion program-wide, what are the barriers they would need to consider or overcome?
Acknowledging the duplication detected in ATE Survey responses, this study closely documented barriers that would need to be considered if the ATE program wanted to collect and report unduplicated counts of student participation and completion program-wide. Structural issues within the two-year college environment, along with unclear requirements from NSF, have led to challenges in ATE projects’ collection and reporting of information regarding student participation and completion. Compounding this, many PIs interviewed in the case studies reported a lack of data literacy skills, support, or resources to track student counts or completion rates.
In addition to addressing challenges in data collection, defining a suitable measure of success for two-year colleges is important, as degree attainment alone is not an encompassing indicator of ATE project success. In reference to student success, one ATE PI stated, “If I can see a job in the future for [my students], I see that as a success.” Additional appropriate metrics might include employment rates in high-technology fields, along with rates of pay or promotion. However, the ATE Survey cannot adequately capture this information, as the survey is a point-in-time instrument used to monitor activity rather than to track long-term outcomes. Case studies may be able to investigate more indicators of student success; however, to truly capture the full effect, case studies would need to be conducted years after students’ involvement with ATE projects.
Recommendations for the NSF ATE Program
- Be explicit about the metrics ATE projects are expected to track, how indicators are defined/operationalized, and how they should be reported.
- Invest in initiatives that will support project-level data collection and reporting.
- Collaborate with other initiatives that are working to identify alternative success measures.
Recommendations for ATE Projects
- Engage evaluators during the project planning phase to ensure student tracking is built into project operations.
- Keep detailed digital records.
- Seek out technical support to help meet data collection and evaluation expectations.
Recommendations to EvaluATE about the ATE Annual Survey
- Restructure questions in the ATE Survey to capture unique counts of students where possible.
- Employ strategies to thoroughly inform PIs about the survey requirements regarding student counts.
- Continue to build capacity in the ATE program for data collection, reporting, and literacy.