New
European Journal of Information Systems (2025) AI Processed Human Approved
Tell me more, tell me more:the impact of explanations on learning from feedback provided by Artificial Intelligence
Drawing on Feedback Theory, this study investigates whether, how, and when explanations provided alongside AI feedback enhance human learning and task performance. The research model was evaluated using a randomized between-subjects online experiment with 573 participants completing a image-location matching task, supplemented by focus group discussions with AI experts and users.
Problem
Although supporting human learning is a primary objective of Explainable Artificial Intelligence (XAI), there is limited understanding of how explanations actually facilitate learning from AI feedback. Specifically, research lacks clarity regarding the underlying theoretical mechanisms and how a user's prior knowledge influences learning and performance outcomes.
Outcome
- Explanations positively impact learning outcomes for users with less prior knowledge, an effect fully mediated by perceived informativeness.
- Explanations improve task performance directly for users with high prior knowledge.
- For users with low prior knowledge, the positive impact of explanations on task performance is fully mediated by informativeness.
- Explanations improve task performance directly for users with high prior knowledge.
- For users with low prior knowledge, the positive impact of explanations on task performance is fully mediated by informativeness.
What it means for you
- CIO / IT Executive: On Monday morning, initiate a cross-functional working group involving IT, product development, and user experience teams to prioritize the integration of AI-driven feedback systems that include clear, context-specific explanations, focusing initially on applications with less experienced user bases.
- IT Manager: On Monday morning, review the current AI feedback mechanisms within your team's systems and identify 1-2 pilot opportunities to implement or enhance explanations, specifically targeting areas where user error or confusion is frequently observed, and gather baseline performance data.
- Business Strategist: On Monday morning, task your team with identifying key business processes or customer-facing interactions where AI feedback is being used or considered, and assess the potential impact of explanation-enhanced AI on user adoption, efficiency, and overall customer satisfaction, particularly for novice users.
- Researcher: On Monday morning, design a pilot study protocol to specifically investigate how the *type* and *level of detail* of explanations provided by AI feedback influence perceived informativeness and learning outcomes for users with varying levels of domain expertise in a controlled environment.
- Policymaker: On Monday morning, begin drafting guidelines or a framework for the ethical deployment of AI systems that mandate, where feasible, the inclusion of explanations for AI-generated feedback, prioritizing sectors with high user impact and potential for knowledge gaps, such as education and public services.
Transcript
Host: Welcome to A.I.S. Insights — powered by Living Knowledge. I'm your host, Anna Ivy Summers. Today, we're diving into a fascinating new study titled, "Tell me more, tell me more: the impact of explanations on learning from feedback provided by Artificial Intelligence." Joining me to break it down is our expert analyst, Alex Ian Sutherland. Alex, welcome!
Expert: Thanks, Anna. It is great to be here.
Host: To kick things off, Alex, this study looks at how AI explanations impact how humans actually learn and perform when getting feedback from an AI system. Can you set the stage for us on why this topic is so critical right now?
Expert: Absolutely. As organizations rapidly deploy AI copilots and decision support tools, supporting human learning is often touted as a primary goal of Explainable AI. We tend to assume that whenever an AI gives a user feedback along with an explanation—explaining why it flagged a transaction or made a specific prediction—the user will naturally learn better.
Host: But in reality, it is not that simple, right?
Expert: Exactly. Until this study, we actually had very limited understanding of whether, how, and when these explanations facilitate real human learning. Specifically, researchers lacked clarity on the underlying mechanisms and how a worker's existing prior knowledge influences their learning and task performance.
Host: That makes a lot of sense. A beginner taking advice from an AI is in a completely different boat than a veteran employee with years of domain experience. So, how did the researchers investigate this problem?
Expert: Grounding their work in Feedback Theory, the researchers conducted a randomized between-subjects online experiment with 573 participants. The task was a non-routine inference challenge: matching Google Street View pictures to their city of origin.
Host: That sounds like a tough task! How was the experiment structured?
Expert: Participants were given AI feedback on their choices, either with or without detailed explanations. The researchers measured their prior knowledge, immediate task performance, and long-term learning outcomes. They also conducted follow-up focus groups with AI experts and users to really understand the human thought processes behind the numbers.
Host: So, what were the key findings from the study?
Expert: The study revealed that AI explanations do not work the same way for everyone. For users with low prior knowledge, explanations positively impacted both their learning outcomes and their task performance, but only through perceived informativeness.
Host: So for a novice, the explanation only helps if they find it genuinely informative and easy to understand?
Expert: Spot on. If an explanation is confusing or lacks informative value, less experienced users get no boost in learning or performance. On the flip side, for users with high prior knowledge, explanations directly improved their immediate task performance, regardless of whether they actively rated the feedback as informative.
Host: That is a critical distinction. Highly knowledgeable users can immediately process and leverage explanations to execute better decisions on the spot, whereas novices need explanations that actively build their understanding over time.
Expert: Precisely. Experienced users use the explanation as a rapid validation tool, while novices need it as a teaching tool.
Host: This brings us to why this study matters so much for business leaders and technology teams. What are the key takeaways for organizations integrating AI into their workflows?
Expert: The biggest takeaway is that AI explanations must be tailored to the user's expertise level. A one-size-fits-all approach to Explainable AI simply does not work. If you are building AI tools to upskill junior staff, you must ensure the explanations are designed specifically for high informativeness—helping them build mental models of the domain.
Host: And for senior, highly skilled workers?
Expert: For senior staff, AI explanations should focus on speed and actionable detail. They do not need foundational teaching; they need quick, contextual insights that help them validate non-routine decisions faster.
Host: That is such a powerful insight for designing better AI workflows and driving real return on investment in workforce training.
Expert: It really is. By aligning AI explanation design with user prior knowledge, companies can ensure their AI investments actually boost both employee learning and operational performance.
Host: Alex, thank you so much for breaking down this impactful study for us today.
Expert: It was my pleasure, Anna.
Host: And thank you to our listeners for tuning into A.I.S. Insights — powered by Living Knowledge. Until next time, keep exploring and keep learning!
Expert: Thanks, Anna. It is great to be here.
Host: To kick things off, Alex, this study looks at how AI explanations impact how humans actually learn and perform when getting feedback from an AI system. Can you set the stage for us on why this topic is so critical right now?
Expert: Absolutely. As organizations rapidly deploy AI copilots and decision support tools, supporting human learning is often touted as a primary goal of Explainable AI. We tend to assume that whenever an AI gives a user feedback along with an explanation—explaining why it flagged a transaction or made a specific prediction—the user will naturally learn better.
Host: But in reality, it is not that simple, right?
Expert: Exactly. Until this study, we actually had very limited understanding of whether, how, and when these explanations facilitate real human learning. Specifically, researchers lacked clarity on the underlying mechanisms and how a worker's existing prior knowledge influences their learning and task performance.
Host: That makes a lot of sense. A beginner taking advice from an AI is in a completely different boat than a veteran employee with years of domain experience. So, how did the researchers investigate this problem?
Expert: Grounding their work in Feedback Theory, the researchers conducted a randomized between-subjects online experiment with 573 participants. The task was a non-routine inference challenge: matching Google Street View pictures to their city of origin.
Host: That sounds like a tough task! How was the experiment structured?
Expert: Participants were given AI feedback on their choices, either with or without detailed explanations. The researchers measured their prior knowledge, immediate task performance, and long-term learning outcomes. They also conducted follow-up focus groups with AI experts and users to really understand the human thought processes behind the numbers.
Host: So, what were the key findings from the study?
Expert: The study revealed that AI explanations do not work the same way for everyone. For users with low prior knowledge, explanations positively impacted both their learning outcomes and their task performance, but only through perceived informativeness.
Host: So for a novice, the explanation only helps if they find it genuinely informative and easy to understand?
Expert: Spot on. If an explanation is confusing or lacks informative value, less experienced users get no boost in learning or performance. On the flip side, for users with high prior knowledge, explanations directly improved their immediate task performance, regardless of whether they actively rated the feedback as informative.
Host: That is a critical distinction. Highly knowledgeable users can immediately process and leverage explanations to execute better decisions on the spot, whereas novices need explanations that actively build their understanding over time.
Expert: Precisely. Experienced users use the explanation as a rapid validation tool, while novices need it as a teaching tool.
Host: This brings us to why this study matters so much for business leaders and technology teams. What are the key takeaways for organizations integrating AI into their workflows?
Expert: The biggest takeaway is that AI explanations must be tailored to the user's expertise level. A one-size-fits-all approach to Explainable AI simply does not work. If you are building AI tools to upskill junior staff, you must ensure the explanations are designed specifically for high informativeness—helping them build mental models of the domain.
Host: And for senior, highly skilled workers?
Expert: For senior staff, AI explanations should focus on speed and actionable detail. They do not need foundational teaching; they need quick, contextual insights that help them validate non-routine decisions faster.
Host: That is such a powerful insight for designing better AI workflows and driving real return on investment in workforce training.
Expert: It really is. By aligning AI explanation design with user prior knowledge, companies can ensure their AI investments actually boost both employee learning and operational performance.
Host: Alex, thank you so much for breaking down this impactful study for us today.
Expert: It was my pleasure, Anna.
Host: And thank you to our listeners for tuning into A.I.S. Insights — powered by Living Knowledge. Until next time, keep exploring and keep learning!