AI vs. Employee-generated Learning content

Last month, we discussed how subject matter experts (SMEs) can create better learning content with ChatGPT. We explored the idea that while ChatGPT can significantly speed up the content creation process, it is meant to assist human expertise, not replace it. 

Today, we take our discussion further by presenting an experiment that highlights the contrasting outcomes of ChatGPT and an SME.

To understand ChatGPT’s capabilities, we conducted a test to see if it could provide the specific insights our anonymous client needed. Our goal was to compare the knowledge and learning content generated by ChatGPT with the expertise of an SME. As we go forward, you’ll notice the specific details, tips, and visuals shared by the SME, which are missing from ChatGPT’s content.

Comparing learning content: SME and ChatGPT

We asked an SME and ChatGPT to create a quick start guide for a customer service team on using Lava-Chat, a messenger application. This internal product guide is specific to the organization; therefore, it isn’t known by ChatGPT. 

The reason for choosing this topic is to emphasize the specificity of SME-generated content which, unlike AI-generated content, is quite generic. SMEs tend to share knowledge specific to their jobs and projects, unlike their AI counterpart, which gleans information from the web without determining its relevance to the end-users. 

Content by an SME 

This interactive guide is the learning content created by an SME. The guide is an introductory course that coaches learners on how to get started with Lava-Chat. It highlights the specific functions relevant to different team members, and provides supporting screenshots to give a clear understanding of the interface. 

The guide begins by listing all key functions of the application, which form the main learning points of the course. The SME understands which Lava-Chat functions are relevant to the customer service team; therefore, he includes only the relevant functions of the application. (In contrast, ChatGPT is not familiar with the application, nor the team’s needs and requirements.) 

Using an e-learning authoring tool, the SME added interactive screenshots that provide clarity about the features. 

Here, end-users can click “i” to understand the fields of the application interface. Interactive elements like hotspots are very effective in engaging learners—again, this is something that ChatGPT isn’t capable of incorporating into content. 

The SME also notes the prerequisites for performing certain functions. This further helps learners understand the technical requirements or tasks that need to be fulfilled. While this is a simple example, it reminds us that SMEs have a lot more context into what their learners know, the tools they have at hand, and even their technical arsenal. 

Content by ChatGPT 

Now, let’s take a look at the content generated by ChatGPT. We provided a brief and to-the-point prompt:  

“Create a product quick start guide to brief your team about a Lava-chat application. Provide screenshots where applicable.”

The result is a generic set of instructions. It is so generic that the instructions could refer to any chat application. There is no specific contextual knowledge about adopting Lava-Chat. Moreover, ChatGPT was unable to provide screenshots that would help the learners better understand how the application works. 

AI-generated vs. Employee-generated learning 

Let’s note the main differences between the learning content generated by AI and the one created by an SME.

Generic vs. specific content 

AI-generated content, by its nature, is generic content. It is the average outcome of all scraped content from the web. On the other hand, subject matter expert (SME) content is more specific and tailored to our organization. It addresses our unique challenges and provides insights into how we handle them.

Let’s take a closer look at the functional differences between SME-crafted content and content created with ChatGPT. Suppose an SME wants to create a course on email phishing and prevention. There are typically two parts to consider. The first part is a general introduction to the topic, where AI can be helpful. However, AI cannot provide organization-specific information.

Now, let’s dive deeper and examine a few parameters to compare how SMEs and ChatGPT create content.

Relevance and context 

EGL is tailored to a specific context, making the content more effective for specific knowledge cases. For instance, in the above example of the quick start guide, EGL works brilliantly. This is because employees who are already familiar with a particular chat messenger possess the knowledge of relevant activities that best suit their team’s needs.

ChatGPT, on the other hand, works well for generic topics like recruiting, sales training, and personal and professional coaching. However, users may need to adapt the information provided to their specific context for optimal application.

While ChatGPT’s learning content presents accurate summaries of existing knowledge on specific topics, it falls short in offering fresh perspectives and profound insights. It lacks the creativity and critical thinking skills necessary for crafting effective learning experiences.

Reliability of sources

ChatGPT generates content by drawing from openly available foundational information on the internet, including portions from intranet sources. While it offers a vast pool of knowledge, there is a significant potential for disinformation. The likelihood of encountering incorrect information is arguably higher compared to content generated by an SME. To ensure accuracy and correctness, the content needs to be reviewed by SMEs. 

EGL is a reliable source of information that is tailored to the context. The learning content created through EGL is typically tacit, experiential, and drawn from hands-on experience and practical know-how. It is original, fresh, and highly relevant, specifically designed to cater to individual roles, projects, or functions. However, it’s important to note that EGL is not without imperfections. We strongly advise the EGL content should be peer-reviewed by other SMEs. 

Accuracy and bias 

ChatGPT is trained to generate words based on a given input. But it can’t truly comprehend the meaning behind those words, which can result in shallow and lacking responses. 

Furthermore, the content may suffer from bias in the data and algorithms used. We call this effect “garbage in, garbage out.”

Another point is that AI cannot localize content as effectively as SMEs. AI-generated quizzes may be less effective in preparing workers for different situations, and relevance and syntactic correctness have been scored low in studies of AI-generated courses. 

Ethics and confidentiality 

EGL offers substantial advantages when it comes to upholding ethics and ensuring confidentiality. With EGL, the learning content is meticulously curated based on the firsthand experiences and insights of employees themselves. This approach enhances the reliability of the information provided and ensures that ethical considerations are addressed conscientiously.

ChatGPT lacks the understanding and contextual awareness that comes with human experience. It generates chains of words scraped from the web, and it may overlook gender and racial biases, failing to grasp the nuanced implications of certain topics. This can give rise to significant issues regarding the ethical soundness of learning materials.

Peer-to-peer connection 

Collaboration between SMEs results in diverse, multifaceted learning content. The collective expertise of SMEs contributes to a rich pool of knowledge, encompassing different perspectives, ideas, and insights. Many ideas come up from people’s impromptu discovery moments. This is crucial in providing comprehensive and well-rounded training experiences.

In contrast, AI offers a regurgitation of the information that already exists on the web, lacking the variation that arises from peer-to-peer collaboration. 

Another important aspect is the social learning factor. People like to interact with other people over machines. Likewise, people learn better from other people as well. 

Consider this: when given the choice between chatting with ChatGPT and having a conversation with someone you know, which do you find more enjoyable? Which would you prefer to invest more time in? Which phone notification would you open first: “You have a message from ChatGPT” or “You have a message from Julie”? 

These reflections highlight our innate preference for human interaction. And this preference undoubtedly affects our learning experiences.

For now, we see AI as a helpful tool for SMEs to create better learning content. It speeds up the process by generating a starting point before writing course content. It assist SMEs who struggle with effective written communication skills. It acts as an assistant, aiding in brainstorming, structuring, organizing content, and refining language and grammar. 

But on its own, it’s not enough. AI is not a complete solution in itself. The true potential lies in the collaborative partnership between humans and AI, as both contribute their unique strengths to achieve optimal outcomes.