Special Challenges Evaluating Non-Degree Credential Programs

One of my evaluations examines implementation of what project managers call incremental credentials—non-degree credentials that don’t fit a traditional associate–bachelor’s–graduate degree structure.

We have encountered particular challenges evaluating the success of these badges, certificates, certifications, and so-called micro credentials. Reflecting on those challenges might be useful for those charged with evaluating programs that result in such credentials-including technician training programs.

My hope here is to give a heads-up to other evaluators, alerting them to a few of the difficulties we have confronted when studying student success through pursuit of these credentials.

Assessing Fidelity. It’s generally necessary to document whether a program is being implemented as designed (i.e., with fidelity), as a basis for judgments about how well it works. But a key feature of many incremental credentialing approaches is flexibility, with learners able to tailor coursework to their particular needs. When a learner can choose from a menu of courses, modules, or activities (such as with stacking credentials), it may be difficult to clearly define and monitor the “treatment.” Such credentials may also change more often than old-school degree programs. Flexibility can also result in more, smaller cohorts of learners getting differing treatments, complicating outcome analysis design. This is a fundamental predicament.

Terminology. Incremental credentialing, as a rapidly evolving area of higher education practice, invokes a messy vocabulary. What exactly is a “micro credential” or “badge” in the evaluand program’s higher ed system or institution? Does everyone know the generally accepted distinctions between a “certificate” and a “certification?” The answers are central to defining what is being evaluated and testing how it works for learners. It’s also crucial that all evaluation stakeholders (funders, program designers and managers, and consumers of findings) share consensus understandings of words describing how a credential solves a problem (its theoretical basis) and the outcomes it is supposed to further. Confusion about constructs and definitions will almost certainly confound implementation and compromise the utility of an evaluation.

Identifying Comparisons. Incremental credentials are often very specialized-aligned with a particular industry or perhaps even unique to a single employer. They are typically specific to one college and may differ in important ways from legacy postsecondary offerings (e.g., requiring much less time to complete than the alternatives they replace). All of this makes finding the comparison cases required for outcome analysis very difficult. Distinctions may even be substantive enough that outcome measures typical for assessing success for comparison programs become obsolete for the cases being studied. How pertinent is “persistence” if a credential entails a few weeks and one registration, rather than 20 courses over two years?

Data Availability. Because state and institutional data systems were designed to track progress and completion for traditional degrees, outcome data for incremental credentials—even basic information like enrollment, completion, and demographics—may not be available from those sources. Data for incremental credential programs may reside in ad hoc systems, potentially even spreadsheets managed by single staff members on local computers. Results of externally administered certification exams are often not managed by (or even shared with) colleges, adding to data sharing and availability issues. And if the evaluator is tasked with measuring learning, the skills-focused orientation common among technician-training credentials makes effective independent assessment extremely difficult.

Advance planning can, however, head off these potential difficulties, albeit likely only through direct collaboration with credential designers and their partners—notably industry partners if they are involved. Regardless, I hope anticipating the above issues from an evaluation’s outset is useful.