The Success Case Method for evaluation
Brinkerhoff’s Success Case Method (SCM) is one way to evaluate the effectiveness of a training initiative by looking at specific examples of performance improvement and non-success as situated within the wider organizational context.
Brinkerhoff, R. (2005) ‘The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training’, Advances in Developing Human Resources, vol. 7, no. 1, pp. 86-101 [Online]. DOI: 10.1177/1523422304272172.
Brinkerhoff, R. (2003) The Success Case Method: Find Out Quickly What’s Working and What’s Not (excerpt), San Francisco, California, Berrett-Koehler Publisher. Excerpt available at https://www.bkconnection.com/static/The_Success_Case_Method_EXCERPT.pdf.
Brinkerhoff, R. and Dressler, D. (2003) ‘Using the Success Case Impact Evaluation Method to Enhance Training Value & Impact’, American Society for Training and Development International Conference and Exhibition 2003. San Diego, California, May 20. [Online]. Available at http://docplayer.net/23802669-Using-the-success-case-impact-evaluation-method-to-enhance-training-value-impact.html.
- A whole organization strategy for evaluation of training
- The Success Case Method (SCM)
- The structure of a SCM study
- Foundations of the SCM
- The SCM process
- An SCM case study
- The organization context
- The training context
- Preparation and the impact model
- Planning the survey
- Evaluation results
- The rest of the story
- Concluding remarks
2003 book excerpt:
- What is the SCM and how does it work?
- The basic SCM questions
- Uses for the SCM approach
- Foundations of the SCM
- Limitations of the SCM approach
2003 conference paper:
- The SCM evaluation model
- SCM steps
- Anatomy of training impact
- The logic of training impact
- Comparison of SCM approach to Kirkpatrick-based evaluation approaches
There are several notable methods for evaluating the effectiveness of a training initiative. Many evaluate the training intervention but not performance improvement, and this is a problem because the training is meant to address a performance gap. ‘Improving the quality an enjoyability of a wedding ceremony may reap some entertaining outcomes for wedding celebrants, but this may do little to create a sustained and constructive marriage’ (Brinkerhoff, 2005, p. 87).
Training is only one part of performance improvement, and we should be measuring training as it is situated within a larger performance management system, which includes things like training, but also the learner, the workplace, and the organization. Successfully applying new knowledge or skills involves the learner as well as their managers, the training leaders, and the senior leadership, because the organization and its people can facilitate or inhibit the performance improvement goals that learning interventions promote.
General questions that SCM seeks to answer
In general, using SCM should uncover answers to these types of questions:
- What is really happening?
What is happening and what is not happening? How are parts of the innovation being used, or not used? Are some groups making the most of the new skills and others not?
- What results is the program producing?
What are the “best case” outcomes? Are there any unintended outcomes?
- What is the value of those results?
Note that you could estimate unrealized value; that is, if the training has resulted in a greater value for one group, then improvement to the entire program will boost the overall value of the program.
- How could this be improved?
What changes can be made to the entire program to better mimic the outcomes found in the most successful group?
(Brinkerhoff, 2003, pp. 6-12)
When to use SCM
Use SCM when you want to:
- Quickly find out what works in a new initiative, and how to make improvements to an existing program
- Present the successes of the initiative in a compelling way
- Share stories that illustrate best practices and examples
The Success Case Method Evaluation Model (redrawn) (Brinkerhoff and Dressler, 2003, p. 4)
Step 1: Identify the training course’s goals and costs
Before you begin, identify the training course’s goals the the business needs it was meant to address. Brinkerhoff (2005) supplies an Impact Model table that can be completed to help identify this information:
|Key skills and knowledge||Critical applications||Key results||Business impact|
|(Skill taught in training course)||(Main way the skill is used in the workplace)||(Main goal when this learning objective is met)||(Way this skill affects the business)|
(His book and other publications offer different variations of this model.)
Steps 2 and 3: Identify exemplars of success and non-success
Send a brief survey to people who attended the training. Alternatively, you can identify people by looking at performance data, usage records, and reports, or by simply asking around, but surveys are best because ‘it provides the additional advantage of extrapolating results to get quantitative estimates of the proportions of people using, or not using, some new method or innovation… [and also] probability estimates of the nature and scope of success can also be determined’ (Brinkerhoff, 2003, p. 16).
In the survey, ask, ‘To what extent have you used your recent training in a way that you believe has made a significant difference to the business?’ (Brinkerhoff, 2005, p. 93). From the responses, identify a small number of people:
- Those who were very successful in applying their new skills or knowledge
- Those were not at all able to use the new skills or knowledge in a significant way
Note that you are looking for the extremes (the very successful and the very non-successful).
Steps 4 and 5: Find corroborating evidence
Conduct 20-30 minute interviews with each of the people you identified. ‘It is important that these interviews focus on gathering verifiable and documentable evidence so that success can be proven—supported by evidence that would “stand up in court”‘ (Brinkerhoff, 2005, p. 91). Start with an initial screening to verify that they were a true success or non-success, and then being to ask more probing questions to uncover documentable evidence.
For those who were very successful, find out:
- How did they applied their new skills or knowledge, and what was the business value in doing so?
- What factors supported their ability to apply the new skills or knowledge? Here you are looking at performance context factors and environmental factors such as supervisory support, feedback, etc.
For those who were not successful, find out what obstacles kept them from applying the new skills or knowledge? When you are evaluating the responses from the initial survey, remember that sometimes people who were successful in applying the new skill are more likely to respond to the survey than those who were not successful in doing so. Asking for stories from those who were not successful may reveal issues that are unrelated to the training course. For example, they may have taken the training course for various reasons (prestige, requirements, an opening in the course and an ongoing bottleneck or backlog in openings) but had no opportunity to apply the new skill or knowledge.
To support the answers given in the interviews, analyze the impact and performance support data.
Step 6: Share stories and statistics
SCM offers a mix of storytelling and research. The output of this step is a set of detailed stories that are credible and verifiable. They show what the training can do, factors that support post-training application, and factors that inhibit it. The stories are written for a wider audience, so they can be shared with many stakeholders and departments within the organization (not just the learning and development department).
However, in relaying success/non-success stories, you must also be able to back up the story with proof or data. ‘The SCM borrows a bit of methodology from journalism and judicial inquiry. In a way, we are journalists looking for a few dramatic and “newsworthy” stories of success and nonsuccess. When we find them, we seek corroborating evidence and documentation to be sure that the success story is defensible and thus reportable’ (Brinkerhoff, 2003, p. 17).