Evaluation Resources

Advisory Panel

Annual Survey

Articulation

Business & Industry

  • New roles to meet industry needs: A look at the Advanced Technological Education program

    The Advanced Technological Education (ATE) program, sponsored by the National Science Foundation, is presented as a model for business and education collaboration to develop technical degree programs for producing more and better technicians. The new roles of industry and college leaders are discussed through a synthesis of literature on skill standards and workforce development. Data are presented from a national evaluation of ATE projects on collaboration, materials development, professional development, and program improvement that demonstrate substantial goal attainment. A discussion on how this process has changed and benefited the partnerships concludes the paper.

    Full Record
    http://scholar.lib.vt.edu/ejournals/JVER/v29n2/zinser.html

Collaborations

Curriculum materials development

Dissemination

Professional development

Program development and improvement

Mentoring

Student recruitment and retention

  • Underrepresented Minority Students in ATE: 2011

    Underrepresented minority students comprise almost 40 percent of all students in ATE-supported programs. By discipline, the percentage of students from underrepresented minority groups ranges from 7 percent to 54 percent.

    Resource Thumbnail
    Full Record
    http://evalu-ate.net/downloads/URM_snapshot_2012.pdf

  • How well are we serving our female students in STEM? webinar recording

    How Well are we Serving our Female Students in STEM? Webinar Recording

    Webinar description: You’d like to have more women enrolled in your technology programs such as nanotechnology and computer networking, but you aren’t sure how many you have now and what you should measure. This webinar will provide you with the evaluation tools you will need to measure enrollment and retention of women in your programs. You’ll get the answers to sticky questions such as, “Do I measure retention of women only or do I compare it to male retention?” and “Should I distinguish between advanced and introductory courses?” Regardless of whether you are in the planning phase or have already started the implementation of your project, you’ll walk away knowing how to better measure your program’s progress in recruiting and retaining female students.  As a bonus, when you sign up for the free webinar, you will receive an evaluation template that was actually used in the CalWomenTech Project.

    Presenter: Donna Milgram, Principal Investigator for the CalWomenTech Project, highlighted by NSF for demonstrating significant achievement and program effectiveness.

    Resource Thumbnail
    Full Record
    http://vimeo.com/42549275

  • How Well are we Serving our Female Students in STEM? webinar slides

    This is a PDF of the slides for the EvaluATE webinar, “How Well Are We Serving Our Female Students in STEM?”

    Webinar description: You’d like to have more women enrolled in your technology programs such as nanotechnology and computer networking, but you aren’t sure how many you have now and what you should measure. This webinar will provide you with the evaluation tools you will need to measure enrollment and retention of women in your programs. You’ll get the answers to sticky questions such as, “Do I measure retention of women only or do I compare it to male retention?” and “Should I distinguish between advanced and introductory courses?” Regardless of whether you are in the planning phase or have already started the implementation of your project, you’ll walk away knowing how to better measure your program’s progress in recruiting and retaining female students.  As a bonus, when you sign up for the free webinar, you will receive an evaluation template that was actually used in the CalWomenTech Project.

    Presenter: Donna Milgram, Principal Investigator for the CalWomenTech Project, highlighted by NSF for demonstrating significant achievement and program effectiveness.

    Resource Thumbnail
    Full Record

  • How Well Are We Serving Our Female Students in STEM? webinar handout

    This is a 1-page handout from the EvaluATE webinar, “How Well Are We Serving Our Female Students in STEM?”

    Webinar description: You’d like to have more women enrolled in your technology programs such as nanotechnology and computer networking, but you aren’t sure how many you have now and what you should measure. This webinar will provide you with the evaluation tools you will need to measure enrollment and retention of women in your programs. You’ll get the answers to sticky questions such as, “Do I measure retention of women only or do I compare it to male retention?” and “Should I distinguish between advanced and introductory courses?” Regardless of whether you are in the planning phase or have already started the implementation of your project, you’ll walk away knowing how to better measure your program’s progress in recruiting and retaining female students.  As a bonus, when you sign up for the free webinar, you will receive an evaluation template that was actually used in the CalWomenTech Project.

    Presenter: Donna Milgram, Principal Investigator for the CalWomenTech Project, highlighted by NSF for demonstrating significant achievement and program effectiveness.

    Resource Thumbnail
    Full Record

  • Female-Male Enrollment and Completion Spreadsheet

    This spreadsheet template, developed by the external evaluator for the CalTechWomen Project (Donna Milgram, PI) includes worksheets for tracking male and female enrollment and completion rates for both introductory and advanced courses.

    Full Record

  • Recruitment and retention

    This report describes findings from current literature that helped our understanding of the general background issues surrounding recruitment and retention as well as the various strategies employed to enhance both efforts. We present a model for R & R practice that contains three elements: Information, Preparation, and Support. Those elements and methods to serve them are elaborated through a series of tables. We supplemented the tabular information with five case examples. These cases show recruitment and retention from different vantage points.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/Issues_for_Consideration_Recruitment_and_Retention.pdf

Sustainability

  • Measuring ATE Sustainability

    The Foundation expects grantees to plan to sustain successful outcomes of their work after NSF funding stops.  There are many definitions of sustainability but the one used here is the capability to endure, that is, to last, or continue to exist.  Synonyms include endurability, continuation, persistence, and perhaps, survival.  The term is not used in the sense of using a resource so that it is not depleted nor permanently damaged as in the currently popular view of the word.  In the context of ATE, sustainability means that programs and activities started under Foundation support, and are determined to have value, continue in some form after the grant ends.  The purpose of this research was to describe the dimensions of sustainability and develop procedures to assess the persistence or continuation of ATE grants.

    Full Record

  • The Impact of the Advanced Technological Education Program (Research Report 2)

    The purpose of this research was to assess the impact (effect or influence) of Advanced Technological Education (ATE) grants.  Report 1 was a detailed description of the research process and the sustainability findings.  This report presents general findings on the impact of ATE grants along with additional details about the survey development and the selection of the sample.  The purpose of Report 2 is to answer the question, “What was the impact on people and institutions who received and implemented ATE grants?”

    Full Record
    http://evalu-ate.org/downloads/resources/ATE_Impact_Report_WELCH.pdf

  • The Sustainability of the Advanced Technological Education Program (Research Report 1)

    The purpose of this research is to assess the sustainability (persistence or continuation) and impact (effect or influence) of Advanced Technological Education (ATE) grants.  The general findings will be presented in two reports.  Report 1 is a description of the research process and the sustainability findings.  Report II will present general findings about the impact of ATE grants.  The purpose of Report 1 is to answer the question, “What is the level of sustainability of programs and activities implemented by institutions that received NSF/ATE grants?”

     

    Full Record

  • The Sustainability of Advanced Technological Education Supported Efforts: An Evaluation

    This report addresses the ways and extent to which Advanced Technology grantees sustain project-level efforts after funding by the National Science Foundation ended. ATE grantees whose projects were completed in 2004 or before were identified and surveyed. Responses were received from 136 of 172 grantees for a response rate of 79 percent. Five targeted aspects of productivity and two factors considered integral to the continuation of project work were addressed. These included courses, curriculum, professional development, materials development, articulation, collaborations with other entities (e.g., business and industry) and continuity of funding support. After completion of ATE funding, productivity declines were reported in two of the five targeted areas, professional development and materials development. Near “steady state” continuity was reported for the remaining three areas of course, curriculum, and articulation agreements. The number of collaborations and the amount of funding support available to the grantees declined substantially following completion of ATE funding. These findings describe a pattern that one might logically expect. That is, areas of work that can be conducted within faculty members’ job expectations and do not required additional funding tend to be continued. Those that require continuous funding tend to decline substantially when that funding ceases. The need for additional study to assess long-term impact of professional development efforts and materials development efforts was noted.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/ATESustainabilityReport.pdf

  • Sustainability: Increasing the likelihood of a long-term impact by the ATE program

    Sustainability is the ability to prolong or to supply with sustenance. This straightforward
    definition takes on a much more complex character when considered in relation to the
    Advanced Technological Education (ATE) program because of the diverse nature of this
    program (i.e., operates under several drivers [e.g., collaboration, program improvement]
    and makes awards to projects and centers). Setting these complexities aside, in a simple
    sense, sustainability for the ATE program could mean continuation of whatever activities
    had been supported by the NSF grant, including institutionalization. This is consistent
    with the definition given for sustainability by the Community College Research Center
    (CCRC) in their study of the ATE program as well. They defined sustainability as “The
    state where the major activities involved in the ATE program continue even after the
    grant expires.” Naturally, outcomes or processes that are not successful or of high quality
    should not be sustained. This places a burden on the ATE projects (i.e., projects and
    centers) and NSF to determine where efforts for sustainability should be focused.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/Issues_for_Consideration_Sustainability.pdf

  • Guide for improving sustainability

    This brochures offers suggestions for enhancing the sustainability of ATE projects and centers.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/Guide_for_Improving_Sustainability_Brochure.pdf

Targeted research

  • Developing and Analyzing a Scale to Measure the Impact of the ATE Program

    Using statements generated by team leaders and other stakeholders, I show they can be used on a Likert-type survey to create a reliable and valid scale to measure the impact of the National Science Foundation’s Advanced Technological Education program (ATE).  The process, called Peer-Generated Likert Scaling, uses these statements to solicit opinions from other ATE grantees.  They are asked to respond with the usual options of strongly agree to strongly disagree.  However, there was an option to mark, Not Applicable (NA), if the statement did not apply to their grant.  This was because the grants vary in size and duration.  It was not clear from a literature review how to score NA option on these kinds of survey.

    Full Record

  • Assessing the Sustainability of the Advanced Technological Education (ATE) Program

    This is a powerpoint presentation companion to the Research and Evaluation reports presented by Wayne Welch. This presentation provides a visual summary of the survey development and evaluation process.

    Resource Thumbnail
    Full Record

  • Peter Saflund’s Context Handout: Resource Centers and Dissemination Projects

    Full Record

  • Research to Define and Measure Effectiveness of ATE Centers/Projects

    The National Science Foundation’s (NSF) Advanced Technological Education (ATE) Program, begun in 1992, currently consists of 39 centers and about 200 projects with annual funding of $64M for new and continuing awards in FY 2010, which is expected to continue at least through FY2013. The ATE program promotes improvement in the education of science and engineering technicians at the undergraduate and secondary school levels. The purpose of this research was to develop measurable criteria of effectiveness for ATE centers/projects across the range of ATE priority areas – educational materials development, professional development and academic program improvement. Findings of the proposed research have potential to place the assessment of effectiveness for this key federally funded program on a firmer scientific basis. Results of the study are intended to allow NSF to better document the outcomes of the ATE program and to apply an objective effectiveness measurement strategy to ATE and similar programs in the future. This research could be used to demonstrate return on investment in the ATE portfolio to Congressional stakeholders.

    Full Record

  • Spring 2011 Conduit Newsletter

    Wayne Welch outlines steps toward instrument validation, we introduce methods for locating preexisting valid instrument, and Helen Sullivan and Amy Gullickson discuss use of a Project Mapping Template.

    Full Record
    http://evalu-ate.net/downloads/EvaluATE%20Spring%202011.pdf

  • Facilitating an understanding of Advanced Technological Education Targeted Research Needs

    In 2007 The Evaluation Center was funded to facilitate dialogue between researchers, two-year college educators, and other appropriate stakeholders by developing and conducting a workshop with the goal of defining research topics that would most benefit ATE project and center principal investigators (PIs) and their staff. The primary outcome for the Targeted Research Design Challenge Workshop was to be an increased understanding of ATE targeted research needs from a variety of stakeholder perspectives (e.g., researchers, ATE PIs, business/industry, and NSF personnel). That workshop was conducted in Baltimore, Maryland, on February 5-6, 2008.
    This paper builds on the Baltimore workshop. Its purpose is to enhance understanding of research needs for the ATE program and issues surrounding technician education, especially in community colleges. The paper explores various perspectives of four primary groups of stakeholders with an interest in learning more about “what works and why” with respect to technician education.1 These stakeholder groups are (1) NSF ATE program officers and the EHR directorate, (2) ATE projects and centers – as well as their faculty and administrators, (3) researchers in four-year colleges and other research settings, and (4) business and industry groups that employ the types of technicians produced in programs supported by the ATE program.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/FINAL%20ATE%20Targeted%20Research%20White%20Paper.pdf

  • NSF Expectations for ATE Projects and Centers

    This document is a distillation of the National Science Foundation’s expectations for Advanced Technological Education projects and centers, based on the 2007-09 program solicitation.

    Full Record
    http://evalu-ate.org/app/webroot/files/uploads/NSF-expectations.pdf

Teacher Preparation

  • NSF Expectations for ATE Projects and Centers

    This document is a distillation of the National Science Foundation’s expectations for Advanced Technological Education projects and centers, based on the 2007-09 program solicitation.

    Full Record
    http://evalu-ate.org/app/webroot/files/uploads/NSF-expectations.pdf

  • Evaluation Basics Webinar (SD)

    This version of the webinar is shown in standard definition.

    Looking to formalize your evaluator-client relationship with a contract? We will share materials that can help shape a clear agreement.

    Thinking about your evaluation budget? We’ll offer guidelines and tips for enhacing evaluation cost effectiveness.

    Still need an evaluator? We’ll show you how to access directories of evaluators, offer guidelines for issuing an RFP for evaluation services, and suggest questions you should ask a potential evaluator.

    This webinar is aimed at helping new grantees and their evaluators get their evaluation work off to a good start.

    Full Record
    http://vimeo.com/7706599

Workforce Needs Assessment

  • Workforce Needs Assessment

    This briefing paper focuses on how ATE projects and centers are conducting workforce needs assessments and how this information is being used in relation to project implementation. We discuss needs assessment as an evaluative activity and suggest that approaching it as such has the potential to concentrate the expertise of evaluators, increasing their capacity to make more meaningful summative evaluation statements about the impacts of ATE projects and centers. Survey findings reveal that (1) a majority of grantees do gather workforce needs assessment information in one or more ways; (2) grantees who do not gather workforce needs assessment data are in the minority; (3) centers are much more likely than projects to conduct workforce needs assessment once funding is awarded; (4) a variety of factor appear to be associated with whether or not projects and centers conduct workforce needs assessment; (5) the purposes for which workforce needs assessment information is considered most useful by grantees include developmental and formative evaluation activities; and (6) many PIs perceive that workforce needs assessment is not essential to their efforts.

    Full Record
    http://evaluation.wmich.edu/evalctr/ate/BP3NeedsAssessment.pdf

  • Webinar: Workforce Needs Assessment Handout

    What if you reached the end of your project, which experienced great successes, but the industry employers in your region said your students were off the mark? To keep closely aligned to what employers want, ATE grantees should conduct workforce needs assessments, typically in the first year of the grant.

    In this webinar, we will review the state of workforce needs assessments among ATE grantees. We will also describe solid methods that have led to successful assessments, featuring case examples from ATE experts (like Ken Bartlett). Finally, we will connect the program planning purpose of a workforce needs assessment with the uses it has for the evaluation of your grant.

    Full Record

  • Webinar: Workforce Needs Assessment Recording

    What if you reached the end of your project, which experienced great successes, but the industry employers in your region said your students were off the mark? To keep closely aligned to what employers want, ATE grantees should conduct workforce needs assessments, typically in the first year of the grant.

    In this webinar, we will review the state of workforce needs assessments among ATE grantees. We will also describe solid methods that have led to successful assessments, featuring case examples from ATE experts (like Ken Bartlett). Finally, we will connect the program planning purpose of a workforce needs assessment with the uses it has for the evaluation of your grant.

    Sign up for our next webinar at evalu-ate.org/​events

    Full Record
    http://vimeo.com/15133993

  • Webinar: Workforce Needs Assessment slides

    What if you reached the end of your project, which experienced great successes, but the industry employers in your region said your students were off the mark? To keep closely aligned to what employers want, ATE grantees should conduct workforce needs assessments, typically in the first year of the grant.

    In this webinar, we will review the state of workforce needs assessments among ATE grantees. We will also describe solid methods that have led to successful assessments, featuring case examples from ATE experts (like Ken Bartlett). Finally, we will connect the program planning purpose of a workforce needs assessment with the uses it has for the evaluation of your grant.

    Full Record
    http://evalu-ate.org/app/webroot/files/uploads/WNA_Webinar_Slides.pdf