5 key questions to ask when evaluating your online training program

You’ve completed your training audit, created your custom eLearning courses (maybe with a little external help), launched your shiny new LMS, enrolled your learners, and the Online Learning Express is leaving the station on its way to Training-ROI-land, its ultimate and eventual destination. All aboard!

“Are we there yet?”

Let me translate that to our online training world: “How do we know our training is working?” Both are very important questions — we want to know we’ll get where we want to go.

Every passenger/training manager has asked that kind of question because a journey (like a well-planned training program) can take a while to complete – and in truth, it never ends.  The Train Engineer is keeping their eyes on the track, watching for hazards, and ensuring that the engine is running at optimal performance. The Conductor checks in with the passengers to make sure they have the right ticket to take them to their destination. That would be the Chief Training Officer and your LMS Administrator(s) as Engineer and Conductor, managing the system overall and making sure the right learners are in the right courses on the right track to success.  

Before the days of computer-controlled trains, there was a person called the Track Switcher, who would operate the manual switches that controlled where the train would go – course correcting multiple times over the entire journey to bring the train safely to its destination.

How does ongoing training evaluation lead to ongoing improvement?

That’s what I want to focus on today – how your ongoing evaluation of your training program ensures you are improving employee performance. It can be far too easy to simply let your online learning program steam on down the track without making any necessary course corrections along the way. You don’t want to get to the end of the line and find out you’ve ended up in Non-improvement Valley or Unengaged Gulch.

I see the role of the Track Switcher as someone who is observant enough to make small course corrections to the journey as successes and challenges reveal themselves in the moment. It involves both a high level of oversight and a granular level of observation to find out which of the many small screwdrivers in their toolkit they might need to use to keep the learning train on track. 

Here are the five things you can monitor along the journey to ensure that your training program is improving employee performance. Some may surprise you. We’re talking about

• Engagement with training
• Learner feedback
• Performance supports
• Observed learner behavior change
• The training metrics that matter
• Key action items for you to move forward with

For each of the sections below, I’ll walk you through what you need to consider, provide you with some signals to watch for, and offer some suggestions on course correction.

Are your learners engaged?

People will log in and complete courses because that’s what they have been told to do. Completion is not a measurement of engagement. Do you hear your team talking about their courses? Are there any friendly rivalries? Has online learning become part of your corporate culture? Ever have an employee ask for additional training – either more in-depth courses or for new training topics? How many times do you have to remind your learners to take their courses and complete their assignments?

If your learners aren’t enjoying taking their courses, you may need to look at two things – your training culture and your training materials. Both will have an impact on your learners’ experience and their willingness to engage. 

Signals to watch for: mood and tone – especially around time allocated for training, listening to water-cooler conversations, metrics around log-in and time in course, and of course, time to course completions.

Course correction: making sure learners have time to learn, using points and rewards to incentivize performance until it becomes habitual (and beyond), and adequate training on “how” to use their eLearning platform to reduce friction.

Are you getting learner feedback?

When people are engaged in learning (or anything, really), they have opinions. They point out errors and omissions. They make suggestions. They tell you how much they like or dislike something. They make their voices heard. This is a subset of engagement. Bare-bones engagement (also called mandatory participation) can be enforced, but feedback is a more organically grown compound - fertilizing the learning experience. Even when the feedback is a bit “stinky,” – it’s good to know what your learners feel and think about their learning system, the course content, and the time investment you are asking them to make in online training. 

Signals to watch for: no feedback, unrealistic feedback, chronic feedback providers, and subtle signals that discourage feedback (as in complaining that someone found an error in the material rather than celebrating it.)

Course corrections: ensuring there are anonymous and visible feedback methods to meet your team in their individual comfort zones, stringent quality controls on course content to ensure minimal unforced errors in grammar, spelling, punctuation, or factual content, and regular updates to prevent state content. Celebrate “best feedback of the month” – the feedback that has the most impact to improve the experience for everyone.

Are your learners using their performance supports?

Many LMS systems offer visibility on how their knowledge base contents are accessed, so you can see what resources are getting used.  Look at what gets used and by whom. Look at what does not get used, and see if you can determine why they are not getting traction. Are they up to date? Are they correct? Are you getting feedback that could help make them better? Do you have a running list of assets in production that is visible and comment-able? Your performance supports are meant to support your team – so it makes sense that your team members are able to contribute to their production. 

Signals to watch for: unused content, out-of-date content, and over-used content (which might indicate a gap in your core training content for a complex idea).

Course corrections: regular review of all performance supports and a mandatory review when course material changes (often additional assets get forgotten at update time). Regularly ask your workers what they need – don’t just build out what “someone in management” things they need. Ensure that there are adequate performance supports to match employees at all levels of learning – material that supports an employee onboarding course for new hires will be vastly different from more in-depth resources for tenured workers.

Is learner behavior changing? 

You may have to wait until the end of the year to see if the results are captured empirically, but overall – are you seeing signs of the changes you want to accomplish? Do you even know what you should be watching for? I’m going to assume for the purposes of this article that your employees are not fully remote – that they work together in a physical space where you can observe their behavior as they go about their work.  

• Is there less wandering around in search of resources or assistance (speaks to the usefulness of performance supports and knowledge retention)
• Fewer hours or days lost to accident or injury (speaks to health and safety training)
• Accomplishing more or better work in the time allotted 
• Being able to demonstrate success in new tasks more quickly (speaks to onboarding and upskilling)
• Improving their performance in areas of specific challenge (speaks to reskilling)

You may not have “the numbers” but conversations with department managers and supervisors should reveal what they see in real time – individually and collectively. Yes, it is subtle and feels intangible. 

Signals to watch for: people doing something right, and changes in patterns of behavior that benefit the business

Course corrections: encourage your managers and supervisors to include online training in their regularly scheduled one-on-one conversations and any employee reviews. Ask how training is having an impact on the individuals and team as a whole. Your supervisors need to be vigilantly watching for signs of training uptake in the day-to-day work done by their teams. It will be subtle, but it will be there.

Are the actual training metrics you measure moving?

You want to be looking for leading indicators that things are moving in the right direction.  Are learners moving through their courses more quickly? Are their marks improving? What else are you measuring that you can assess for distant early celebration – or warning signals that your program is having the desired impact? 

Signals to watch for: small improvements in your KPIs, focusing on the early-stage metrics that matter most. Dig into individual and aggregate learner metrics looking for trends when it comes to tangible measurements – time spent, activities completed, improved grades, improved productivity, reduced absenteeism, and fewer safety incidents.

Course corrections: use the numbers to validate the human experience. As you find out where your best gains are, focus on the areas where improvement is more elusive, and make a plan to ensure you are seeing the results you want consistently across your training program.

5 key action items to evaluate your online training effectiveness

Trains just keep on rolling, taking a diverse group of passengers to any number of destinations. The trip can be accelerated to a certain point, but it takes as long as it takes – as long as the train is moving on, down the track. When it comes to online training, real growth and true change can be slow – because incorporating new ideas to change behavior can take time – but the growth will be there. 

Here’s your itinerary if you decide to embark on this journey

• Engagement with training– make sure nothing stands in the way of learner engagement – that there is time, encouragement, and properly set expectations that learners will take their training. 
• Learner feedback – ensure that there are both anonymous and transparent channels for feedback and that feedback is acted on to increase engagement and trust.
• Performance supports – see what is used and what is needed. Ask why certain resources are not used, and replace them with better tools.
• Observed learner behavior change – create a culture of ongoing learning, talk it up, promote and reward both participation and success.
• The training metrics that matter – never take your eyes off the metrics dashboard – initial improvements may be small, but should generally be trending in the right direction.

Real growth is also consistent. Don’t be misled by peaks or valleys that might measure bursts of enthusiasm or distraction if there are deadline-driven projects that might take the focus away from online training for a few days. You are looking for long trendlines, not daily micro-analysis. It’s a delicate balance, to be sure. Sometimes the train struggles to go up the hill, sometimes, you have to hit the brakes to manage the descent on the other side of that mountain – learners learn consistently at their own pace, which is affected by so many factors in their personal and professional lives. 

No journey is perfect, and that includes your training journey. There will always be room for improvement. Watching for these signals and continually evaluating your online training program – both your online training platform and eLearning content is key to ensuring that your employees are enjoying their training journey and learning as much as they can at the highest levels possible to ensure their professional success.  Their success is the true measurement of the success of your online training program. 

Pulling into the training evaluation station

So, are you ready to get on board the Evaluation Express and take the journey To Success-Ville? It’s the destination that keeps moving – because every time your Training Switch Operator makes a course correction, you and your learners get back on board and continue your journey of professional development in pursuit of training excellence.

Looking for more resources that cover training evaluation, learning analytics, or training ROI? Jump back into our learning analytics and training ROI guide in our learning hub for more great reads!