What can AI do for you vs. what should you let it do?

Soon after the release of ChatGPT I added a section on AI to the full-day workshop I teach on Learning Measurement. My first addition was using AI to write multiple choice questions. (Spoiler alert: It didn’t do a very good job back then but does a rather credible job now.) Over time, as AI has become increasingly embedded in the work of L&D, I’ve incrementally added new topics to this section of workshop, and at this point it is a full-blown module. This has led me to reflect on an important question: “As an L&D professional, what can AI do for you vs. what should you let AI do for you?” Not surprisingly, the answer isn’t binary; it’s a continuum. Let’s take a look at three examples of the can/should trade-off:

Can and Should

Prior to my current work teaching and consulting, I founded an assessment management system company (i.e. we gave tests online). The system collected mountains of performance data, but over 90% of our clients just wanted to know one thing: Who passed and who didn’t pass the test? Of our 70 or so clients I could count on one hand the number who really wanted to analyze the data and use this analysis to provide feedback on the strengths and weaknesses of their training programs. This was a shame because there was a lot of hidden gold to be mined in that data. But -- and here’s the good news -- AI is really  good at data analysis. It can analyze and synthesize actionable results for you, and it is excellent at data visualization.  I have zero talent in graphic design, but Claude produces great charts for me. Here’s an example of a graphic Claude produced showing the progress of learners through three modules of a course:

Can and Should (With Human Oversight)

Over my career I have written thousands of test questions, mostly multiple choice. Of course, multiple-choice questions have their limitations. For the most part they are used to test recall, occasionally understanding and application, but almost never more complex critical thinking. Why? It’s difficult and time consuming to write M/C questions that test at higher cognitive levels. Of course, we know that we can easily test beyond memorization with open-response questions, but we tend not to use them because we don’t have the time to score hundreds of essay-style responses. As AI has improved I have found that it is pretty good at writing both M/C and open-response questions, and at scoring open-response questions.

So, using AI to upgrade your exams from low-level M/C questions to higher-level open response questions is recommended. And for low stakes exams (e.g. quizzes, module level tests) you can even let AI score the responses (but with some quality control oversight). But for high-stakes exams (i.e. exams with job/career consequences, compliance exams) proceed with caution.  As we all know, AI is not perfect and you can’t afford a high-stakes error, so if you use AI to score responses you need to be actively involved in reviewing its output. And if you feel at all uncertain do the scoring yourself.

Shouldn’t

How do we ensure that our training has impact? In the workshop I teach on learning measurement we spend the first half of the workshop discussing creating fair, valid and reliable learning assessments. Areas where AI can help quite a bit. But, in the second half of the workshop we pivot to performance. We discuss how learning leads to capability, which leads to performance and then, ultimately, business impact. The dependency is simple: No performance change, no impact. But improved performance is a complex phenomenon, only one part of which is influenced by the training itself. Research has shown that 80% or more of performance improvement is due to the performance environment (e.g. managerial support). So, for training to succeed you need to work with others outside the strict confines of L&D.

This sort of strategic collaboration is beyond the capability of any AI model. It’s exactly where we humans excel, especially if you bring to the table years of tacit knowledge that can be leveraged to find the optimal performance solution.

 So, when contemplating where and how to use AI, consider the continuum. Use AI for the day-to-day work that used to consume so much of your time and use your newly freed-up time to make those strategic decisions that make L&D a valued business partner.

Authored by: Steven Just