TiER1 Performance This article originally appeared in The Data & Measurement Issue of Performance Matters magazine. Download a digital copy of the magazine to read more on this topic.
Learning initiatives fuel strategic progress. Leaders need actionable, timely information about program impact to keep their goals on track. However, many organizations lack effective learning evaluation programs that could provide these insights. Of the 556 respondents to TiER1’s 2023 Learning Trends Survey, just 5 percent report that they generally use research-supported evaluation practices, while 64 percent feel unable to do the kinds of learning evaluation that they want to do.
This is the story of how a Canadian energy company developed both the capabilities of their learning team and a suite of powerful evaluation tools in less than nine months.
It all started when the company’s learning team approached TiER1 for help evaluating a key program intended to provide foundational training for shovel supervisors in one of their mines.
To get the most from the partnership, we adopted a hybrid model where the company’s team could learn from and work alongside Will Thalheimer, PhD, MBA, an internationally recognized learning evaluation expert and author of an award-winning book on learning evaluation. Through this hybrid model with TiER1, the team would come away with several things:
To be full partners in evaluation, the core team spent time building shared values and a common language.
We conducted research and interviews with key stakeholders, instructors, and recent learners to better understand the needs of shovel supervisors and the goals for the course. Digging in further, the team engaged in a series of six two-hour collaborative workshops to develop foundational skills in the design of metrics, studies, surveys, and scenarios and to apply these skills to our project context.
We then developed appropriate metrics and measuring tools, using the Learning-Transfer Evaluation Model (LTEM) developed by Will Thalheimer to guide our work. LTEM helps teams make decisions about the kinds of data to collect for a multidimensional picture of a learning program’s effectiveness. For instance, to measure job performance, we identified key behavioral indicators (KBIs) that would show if course objectives were really being applied on the job.
We planned a time-series study, in which we would compare the data before training and at multiple points after the training, to assess the impact the training had on the learners.
What We Wanted to Measure:
How We Measured It:
Developing the evaluation instruments was an iterative process, involving multiple stages of team discussion, review, and revision. For example, we validated scenario questions with both experts and non-experts. Listening to experts “think aloud” through their answers helped us ensure that the questions would assess decision-making based on the intended learning points. Testing with non-experts helped us identify and remove wording that could make the answers easier to guess.
The project did face challenges, but the team persevered in excavating insights and expanding their learning evaluation capability.
We ran into obstacles with implementing the time-series study as envisioned, as the number of learners entering the program dropped significantly right before our planned start for the study. Since our sample size was too small to perform meaningful statistical analysis, we ultimately derived the most reliable insights from research-inspired benchmarking based on our initial interviews and observations.
Despite this setback, we still achieved several important outcomes:
Research-based evaluation methods such as LTEM help us leverage learning and performance experiences of all kinds to uncover insights and achieve our goals. This project illustrates how collaboration and learning science can combine for a solution that delivers high-performance, develops teams, and drives business results.
Learning evaluation projects comes in many shapes and sizes. Talk to TiER1 about your goals, and we can help design the right solution for your team.
Heather Cole is a Senior Learning Consultant at TiER1 Performance. She believes in building memorable experiences that help people reach their goals and do their best work. With a background in instructional design and education, Heather brings a unique suite of skills and valuable perspective to performance challenges. Outside of work, Heather loves reading, hiking, and acrylic paint pouring.
With over 20 years of experience in corporate, public, and non-profit sectors, Alex Brierley has had the opportunity to develop, design, and deliver learning experiences for topics ranging from systems and software to web design and life skills. Curious by nature, she seeks new ways to help learners understand and apply the principles and skills they need to be successful. Outside of work, Alex loves to paddleboard, ski, hike and ride anything with two wheels from bicycles to motorcycles. She also has a weakness for pizza and electronic music.