The Success Case Method for evaluation

Brinkerhoff’s Success Case Method (SCM) is one way to evaluate the effectiveness of a training initiative by looking at specific examples of performance improvement and non-success as situated within the wider organizational context.

Brinkerhoff, R. (2005) ‘The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training’, Advances in Developing Human Resources, vol. 7, no. 1, pp. 86-101 [Online]. DOI: 10.1177/1523422304272172.

Brinkerhoff, R. (2003) The Success Case Method: Find Out Quickly What’s Working and What’s Not (excerpt), San Francisco, California, Berrett-Koehler Publisher. Excerpt available at https://www.bkconnection.com/static/The_Success_Case_Method_EXCERPT.pdf.

Brinkerhoff, R. and Dressler, D. (2003) ‘Using the Success Case Impact Evaluation Method to Enhance Training Value & Impact’, American Society for Training and Development International Conference and Exhibition 2003. San Diego, California, May 20. [Online]. Available at http://docplayer.net/23802669-Using-the-success-case-impact-evaluation-method-to-enhance-training-value-impact.html.

Outline

2005 article:

  1. A whole organization strategy for evaluation of training
  2. The Success Case Method (SCM)
    1. The structure of a SCM study
    2. Foundations of the SCM
    3. The SCM process
  3. An SCM case study
    1. The organization context
    2. The training context
    3. Preparation and the impact model
    4. Planning the survey
    5. Evaluation results
    6. The rest of the story
  4. Concluding remarks

2003 book excerpt:

  1. What is the SCM and how does it work?
  2. The basic SCM questions
  3. Uses for the SCM approach
    1. Foundations of the SCM
    2. Limitations of the SCM approach

2003 conference paper:

  1. The SCM evaluation model
  2. SCM steps
  3. Anatomy of training impact
  4. The logic of training impact
  5. Examples
  6. Comparison of SCM approach to Kirkpatrick-based evaluation approaches
  7. Summary
  8. FAQs

Notes

There are several notable methods for evaluating the effectiveness of a training initiative. Many evaluate the training intervention but not performance improvement, and this is a problem because the training is meant to address a performance gap. ‘Improving the quality an enjoyability of a wedding ceremony may reap some entertaining outcomes for wedding celebrants, but this may do little to create a sustained and constructive marriage’ (Brinkerhoff, 2005, p. 87).

Training is only one part of performance improvement, and we should be measuring training as it is situated within a larger performance management system, which includes things like training, but also the learner, the workplace, and the organization. Successfully applying new knowledge or skills involves the learner as well as their managers, the training leaders, and the senior leadership, because the organization and its people can facilitate or inhibit the performance improvement goals that learning interventions promote.

General questions that SCM seeks to answer

In general, using SCM should uncover answers to these types of questions:

(Brinkerhoff, 2003, pp. 6-12)

When to use SCM

Use SCM when you want to:

SCM's five step model, as discussed below

The Success Case Method Evaluation Model (redrawn) (Brinkerhoff and Dressler, 2003, p. 4)

Step 1: Identify the training course’s goals and costs

Before you begin, identify the training course’s goals the the business needs it was meant to address. Brinkerhoff (2005) supplies an Impact Model table that can be completed to help identify this information:

Key skills and knowledgeCritical applicationsKey resultsBusiness impact
(Skill taught in training course)(Main way the skill is used in the workplace)(Main goal when this learning objective is met)(Way this skill affects the business)



(His book and other publications offer different variations of this model.)

Steps 2 and 3: Identify exemplars of success and non-success

Send a brief survey to people who attended the training. Alternatively, you can identify people by looking at performance data, usage records, and reports, or by simply asking around, but surveys are best because ‘it provides the additional advantage of extrapolating results to get quantitative estimates of the proportions of people using, or not using, some new method or innovation… [and also] probability estimates of the nature and scope of success can also be determined’ (Brinkerhoff, 2003, p. 16).

In the survey, ask, ‘To what extent have you used your recent training in a way that you believe has made a significant difference to the business?’ (Brinkerhoff, 2005, p. 93). From the responses, identify a small number of people:

Note that you are looking for the extremes (the very successful and the very non-successful).

Steps 4 and 5: Find corroborating evidence

Conduct 20-30 minute interviews with each of the people you identified. ‘It is important that these interviews focus on gathering verifiable and documentable evidence so that success can be proven—supported by evidence that would “stand up in court”‘ (Brinkerhoff, 2005, p. 91). Start with an initial screening to verify that they were a true success or non-success, and then being to ask more probing questions to uncover documentable evidence.

For those who were very successful, find out:

For those who were not successful, find out what obstacles kept them from applying the new skills or knowledge? When you are evaluating the responses from the initial survey, remember that sometimes people who were successful in applying the new skill are more likely to respond to the survey than those who were not successful in doing so. Asking for stories from those who were not successful may reveal issues that are unrelated to the training course. For example, they may have taken the training course for various reasons (prestige, requirements, an opening in the course and an ongoing bottleneck or backlog in openings) but had no opportunity to apply the new skill or knowledge.

To support the answers given in the interviews, analyze the impact and performance support data.

Step 6: Share stories and statistics

SCM offers a mix of storytelling and research. The output of this step is a set of detailed stories that are credible and verifiable. They show what the training can do, factors that support post-training application, and factors that inhibit it. The stories are written for a wider audience, so they can be shared with many stakeholders and departments within the organization (not just the learning and development department).

However, in relaying success/non-success stories, you must also be able to back up the story with proof or data. ‘The SCM borrows a bit of methodology from journalism and judicial inquiry. In a way, we are journalists looking for a few dramatic and “newsworthy” stories of success and nonsuccess. When we find them, we seek corroborating evidence and documentation to be sure that the success story is defensible and thus reportable’ (Brinkerhoff, 2003, p. 17).

Leave a reply