What it takes to control Al by design: human learning
Dov Te'eni, Inbal Yahav, David Schwartz
This study proposes a robust framework, based on systems theory, for maintaining meaningful human control over complex human-AI systems. The framework emphasizes the importance of continual human learning to parallel advancements in machine learning, operating through two distinct modes: a stable mode for efficient operation and an adaptive mode for learning. The authors demonstrate this concept with a method called reciprocal human-machine learning applied to a critical text classification system.
Problem
Traditional methods for control and oversight are insufficient for the complexity of modern AI technologies, creating a gap in ensuring that critical AI systems remain aligned with human values and goals. As AI becomes more autonomous and operates in volatile environments, there is an urgent need for a new approach to design systems that allow humans to effectively stay in control and adapt to changing circumstances.
Outcome
- The study introduces a framework for human control over AI that operates at multiple levels and in two modes: stable and adaptive. - Effective control requires continual human learning to match the pace of machine learning, ensuring humans can stay 'in the loop' and 'in control'. - A method called 'reciprocal human-machine learning' is presented, where humans and AI learn from each other's feedback in an adaptive mode. - This approach results in high-performance AI systems that are unbiased and aligned with human values. - The framework provides a model for designing control in critical AI systems that operate in dynamic environments.
Host: Welcome to A.I.S. Insights, the podcast at the intersection of business and technology, powered by Living Knowledge. I'm your host, Anna Ivy Summers. Host: Today, we’re diving into a critical question for any organization using artificial intelligence: How do we actually stay in control? We'll be discussing a fascinating study titled, "What it takes to control AI by design: human learning." Host: It proposes a new framework for maintaining meaningful human control over complex AI systems, emphasizing that for AI to learn, humans must learn right alongside it. Here to break it all down for us is our analyst, Alex Ian Sutherland. Welcome, Alex. Expert: Thanks for having me, Anna. It’s a crucial topic. Host: Absolutely. So, Alex, let's start with the big picture. What is the real-world problem this study is trying to solve? Expert: The problem is that AI is evolving much faster than our methods for managing it. Think about critical systems in finance, cybersecurity, or logistics. We use AI to make high-stakes decisions at incredible speed. Expert: But our traditional methods of oversight, where a person just checks the final output, are no longer enough. As the study points out, AI can alter its behavior or generate unexpected results when it encounters new situations, creating a huge risk that it no longer aligns with our original goals. Host: So there's a growing gap between the AI's capability and our ability to control it. How did the researchers approach this challenge? Expert: They took a step back and used systems theory. Instead of seeing the human and the AI as separate, they designed a single, integrated system that operates in two distinct modes. Expert: First, there's the 'stable mode'. This is when the AI is working efficiently on its own, handling routine tasks based on what it already knows. Think of it as the AI on a well-defined autopilot. Expert: But when the environment changes or the AI's confidence drops, the system shifts into an 'adaptive mode'. This is a collaborative learning session, where the human expert and the AI work together to make sense of the new situation. Host: That’s a really clear way to put it. What were the main findings that came out of this two-mode approach? Expert: The first key finding is that this dual-mode structure is essential. You get the efficiency of automation in the stable mode, but you have a built-in, structured way to adapt and learn when faced with uncertainty. Host: And I imagine the human is central to that adaptive mode. Expert: Exactly. And that’s the second major finding: for this to work, human learning must keep pace with machine learning. To stay in control, the human expert can't be a passive observer. They must be actively learning and updating their own understanding of the environment. Host: That turns the typical human-in-the-loop idea on its head a bit. Expert: It does. Which leads to the third and most interesting finding, a method they call 'reciprocal human-machine learning'. In the adaptive mode, it’s not just the human teaching the machine. The AI provides specific feedback to the human expert, pointing out patterns or inconsistencies they might have missed. Expert: So, the human and the AI are actively learning from each other. This reciprocal feedback loop ensures the entire system gets smarter, performs better, and stays aligned with human values, preventing things like algorithmic bias from creeping in. Host: A true partnership. This is where it gets really interesting for our listeners. Alex, why does this matter for business? What are the practical takeaways? Expert: This framework is a roadmap for de-risking advanced AI applications. For any business using AI in critical functions, this is a way to ensure safety, accountability, and alignment with company ethics. It's about moving from a "black box" to a controllable, transparent system. Expert: Second, it's about building institutional knowledge. By keeping humans actively engaged in the learning process, you're not just improving the AI; you're upskilling your employees. They develop a deeper expertise that makes your entire operation more resilient and adaptable. Expert: And finally, that adaptability is a huge competitive advantage. A business with a human-AI system that can learn and respond to market shifts, new cyber threats, or supply chain disruptions will outperform one with a rigid, static AI every time. Host: So to recap: traditional AI oversight is failing. This study presents a powerful framework where a human-AI system operates in a stable mode for efficiency and an adaptive mode for learning. Host: The key is that this learning must be reciprocal—a two-way street where both human and machine get smarter together, ensuring the AI remains a powerful, controllable, and trusted tool for the business. Host: Alex, thank you so much for these valuable insights. Expert: My pleasure, Anna. Host: And thank you to our audience for tuning into A.I.S. Insights. Join us next time as we continue to explore the ideas shaping our world.
Balancing fear and confidence: A strategic approach to mitigating human risk in cybersecurity
Dennis F. Galletta, Gregory D. Moody, Paul Benjamin Lowry, Robert Willison, Scott Boss, Yan Chen, Xin “Robert” Luo, Daniel Pienta, Peter Polak, Sebastian Schuetze, and Jason Thatcher
This study explores how to improve cybersecurity by focusing on the human element. Based on interviews with C-level executives and prior experimental research, the paper proposes a strategy for communicating cyber threats that balances making employees aware of the dangers (fear) with building their confidence (efficacy) to handle those threats effectively.
Problem
Despite advanced security technology, costly data breaches continue to rise because human error remains the weakest link. Traditional cybersecurity training and policies have proven ineffective, indicating a need for a new strategic approach to manage human risk.
Outcome
- Human behavior is the primary vulnerability in cybersecurity, and conventional training programs are often insufficient to address this risk. - Managers must strike a careful balance in their security communications: instilling a healthy awareness of threats ('survival fear') without causing excessive panic or anxiety, which can be counterproductive. - Building employees' confidence ('efficacy') in their ability to identify and respond to threats is just as crucial as making them aware of the dangers. - Effective tools for changing behavior include interactive methods like phishing simulations that provide immediate feedback, gamification, and fostering a culture where security is a shared responsibility. - The most effective approach is to empower users by providing them with clear, simple tools and the knowledge to act, rather than simply punishing mistakes or overwhelming them with fear.
Host: Welcome to A.I.S. Insights — powered by Living Knowledge. I’m your host, Anna Ivy Summers. Today, we’re looking at a critical issue that costs businesses billions: cybersecurity. But we're not talking about firewalls and encryption; we’re talking about people. Host: We're diving into a fascinating new study titled "Balancing fear and confidence: A strategic approach to mitigating human risk in cybersecurity." It proposes a new strategy for communicating cyber threats, one that balances making employees aware of dangers with building their confidence to handle them. Host: Here to break it down for us is our analyst, Alex Ian Sutherland. Alex, welcome. Expert: Great to be here, Anna. Host: So, Alex, let's start with the big picture. We invest so much in security technology, yet we keep hearing about massive, costly data breaches. What's the core problem this study addresses? Expert: The core problem is that despite all our advanced tech, the human element remains the weakest link. The study highlights that data breaches are not only increasing, they’re getting more expensive, averaging nearly 9.5 million dollars per incident in 2023. Host: Nine and a half million dollars. That’s staggering. Expert: It is. And the research points out that about 90% of all data breaches result from internal causes like simple employee error or negligence. So, the traditional approach—annual training videos and dense policy documents—clearly isn't working. We need a strategic shift. Host: So how did the researchers approach this? It sounds like a complex human problem. Expert: It is, and they took a very practical approach. They combined findings from their own prior experiments on how people react to threats with a series of in-depth interviews. They spoke directly with ten C-level executives—CISOs and CIOs—from major companies in healthcare, retail, and manufacturing. Host: So, this isn't just theory. They went looking for a reality check from leaders on the front lines. Expert: Exactly. They wanted to know what actually works in the real world when it comes to motivating employees to be more secure. Host: Let’s get to their findings. What was the most significant discovery? Expert: The biggest takeaway is the need for a delicate balance. Managers need to instill what the study calls a healthy 'survival fear'—an awareness of real threats—without causing panic or anxiety, which just makes people shut down. Host: 'Survival fear' is an interesting term. Can you explain that a bit more? Expert: Think of it like teaching a child not to touch a hot stove. You want them to have a healthy respect for the danger, not to be terrified of the kitchen. One executive described it as an "inverted U" relationship: too little fear leads to complacency, but too much leads to paralysis where employees are too scared to do their jobs. Host: So you make them aware of the threat, but then what? You can’t just leave them feeling anxious. Expert: And that’s the other half of the equation: building their confidence, or what the study calls 'efficacy.' It’s just as crucial to empower employees with the belief that they can actually identify and respond to a threat. Fear gets their attention, but confidence is what drives the right action. Host: What did the study find were the most effective tools for building that confidence? Expert: The executives universally praised interactive methods over passive ones. The most effective tool by far was phishing simulations. These are fake phishing emails sent to employees. When someone clicks, they get immediate, private feedback explaining what they missed. It's a safe way to learn from mistakes. Host: It sounds much more engaging than a PowerPoint presentation. Expert: Absolutely. Gamification, like leaderboards for spotting threats, also works well. The key is moving away from a culture of punishment and toward a culture of shared responsibility, where reporting a suspicious email is seen as a positive, helpful action. Host: This is the critical part for our listeners. Alex, what are the practical takeaways for a business leader who wants to strengthen their company's human firewall? Expert: There are three key actions. First, reframe your communication. Stop leading with fear and punishment. Instead, focus on empowerment. The goal is to instill that healthy ‘survival fear’ about the consequences, but immediately follow it with simple, clear actions employees can take to protect themselves and the company. Host: So, it's not "don't do this," but "here's how you can be a hero." Expert: Precisely. The second takeaway is to make security easy. The executives pointed to the success of simple tools, like a "report this email" button that takes just one click. If security is inconvenient, people will find ways around it. Remove the friction from doing the right thing. Host: And the third action? Expert: Make your training relevant and continuous. Ditch the generic, annual "check-the-box" training that employees just play in the background. Use those phishing simulations, create short, engaging content, and tailor it to different teams. The threats are constantly evolving, so your training has to as well. Host: So, to summarize, it seems the old model of just telling employees the rules is broken. Host: The new approach is a delicate balance: make people aware of the risks, but immediately empower them with the confidence and the simple tools they need to become an active line of defense. It's about culture, not just controls. Host: Alex, this has been incredibly insightful. Thank you for making this complex topic so clear. Expert: My pleasure, Anna. Host: And thanks to all of you for tuning in to A.I.S. Insights — powered by Living Knowledge. Join us next time as we translate another key piece of research into actionable business strategy.
Cybersecurity, Human Risk, Fear Appeals, Security Awareness, User Actions, Management Interventions, Data Breaches
Communications of the Association for Information Systems (2024)
Design Knowledge for Virtual Learning Companions from a Value-centered Perspective
Ricarda Schlimbach, Bijan Khosrawi-Rad, Tim C. Lange, Timo Strohmann, Susanne Robra-Bissantz
This study develops design principles for Virtual Learning Companions (VLCs), which are AI-powered chatbots designed to help students with motivation and time management. Using a design science research approach, the authors conducted interviews, workshops, and built and tested several prototypes with students. The research aims to create a framework for designing VLCs that not only provide functional support but also build a supportive, companion-like relationship with the learner.
Problem
Working students in higher education often struggle to balance their studies with their jobs, leading to challenges with motivation and time management. While conversational AI like ChatGPT is becoming common, these tools often lack the element of companionship and a holistic approach to learning support. This research addresses the gap in how to design AI learning tools that effectively integrate motivation, time management, and relationship-building from a user-value-centered perspective.
Outcome
- The study produced a comprehensive framework for designing Virtual Learning Companions (VLCs), resulting in 9 design principles, 28 meta-requirements, and 33 design features. - The findings are structured around a “value-in-interaction” model, which proposes that a VLC's value is created across three interconnected layers: the Relationship Layer, the Matching Layer, and the Service Layer. - Key design principles include creating a human-like and adaptive companion, enabling proactive and reactive behavior, building a trustworthy relationship, providing supportive content, and fostering a motivational and ethical learning environment. - Evaluation of a coded prototype revealed that different student groups have different preferences, emphasizing that VLCs must be adaptable to their specific educational context and user needs to be effective.
Host: Welcome to A.I.S. Insights, the podcast where we connect academic research to real-world business strategy, powered by Living Knowledge. I’m your host, Anna Ivy Summers.
Host: Today, we’re exploring a topic that’s becoming increasingly relevant in our AI-driven world: how to make our digital tools not just smarter, but more supportive. We’re diving into a study titled "Design Knowledge for Virtual Learning Companions from a Value-centered Perspective".
Host: In simple terms, it's about creating AI-powered chatbots that act as true companions, helping students with the very human challenges of motivation and time management. Here to break it all down for us is our expert analyst, Alex Ian Sutherland. Welcome, Alex.
Expert: Thanks for having me, Anna. It’s a fascinating study with huge implications.
Host: Let's start with the big picture. What is the real-world problem that this study is trying to solve?
Expert: Well, think about anyone trying to learn something new while juggling a job and a personal life. It could be a university student working part-time or an employee trying to upskill. The biggest hurdles often aren't the course materials themselves, but staying motivated and managing time effectively.
Host: That’s a struggle many of our listeners can probably relate to.
Expert: Exactly. And while we have powerful AI tools like ChatGPT that can answer questions, they function like a know-it-all tutor. They provide information, but they don't provide companionship. They don't check in on you, encourage you when you're struggling, or help you plan your week. This study addresses that gap.
Host: So it's about making AI more of a partner than just a tool. How did the researchers go about figuring out how to build something like that?
Expert: They used a very hands-on approach called design science research. Instead of just theorizing, they went through multiple cycles of building and testing. They started by conducting in-depth interviews with working students to understand their real needs. Then, they held workshops, designed a couple of conceptual prototypes, and eventually built and coded a fully functional AI companion that they tested with different student groups.
Host: So it’s a methodology that’s really grounded in user feedback. What were the key findings? What did they learn from all this?
Expert: The main outcome is a powerful framework for designing these Virtual Learning Companions, or VLCs. The big idea is that the companion's value is created through the interaction itself, which they break down into three distinct but connected layers.
Host: Three layers. Can you walk us through them?
Expert: Of course. First is the Relationship Layer. This is all about creating a human-like, trustworthy companion. The AI should be able to show empathy, maybe use a bit of humor, and build a sense of connection with the user over time. It’s the foundation.
Host: Okay, so it’s about the personality and the bond. What's next?
Expert: The second is the Matching Layer. This is about adaptation and personalization. The study found that a one-size-fits-all approach fails. The VLC needs to adapt to the user's individual learning style, their personality, and even their current mood or context.
Host: And the third layer?
Expert: That's the Service Layer. This is where the more functional support comes in. It includes features for time management, like creating to-do lists and setting reminders, as well as providing supportive learning content and creating a motivational environment, perhaps with gentle nudges or rewards.
Host: This all sounds great in theory, but did they see it work in practice?
Expert: They did, and they also uncovered a critical insight. When they tested their prototype, they found that full-time university students thought the AI’s language was too informal and colloquial. But a group of working professionals in a continuing education program found the exact same AI to be too formal!
Host: Wow, that’s a direct confirmation of what you said about the Matching Layer. The companion has to be adaptable.
Expert: Precisely. It proves that to be effective, these tools must be tailored to their specific audience and context.
Host: Alex, this is the crucial part for our audience. Why does this matter for business? What are the practical takeaways?
Expert: The implications are huge, Anna, and they go way beyond the classroom. Think about corporate training and HR. Imagine a new employee getting an AI companion that doesn't just teach them software systems, but helps them manage the stress of their first month and checks in on their progress and motivation. That could have a massive impact on engagement and retention.
Host: I can see that. It’s a much more holistic approach to onboarding. Where else?
Expert: For any EdTech company, this framework is a blueprint for building more effective and engaging products. It's about moving from simple content delivery to creating a supportive learning ecosystem. But you can also apply these principles to customer-facing bots. An AI that can build a relationship and adapt to a customer's technical skill or frustration level will provide far better service and build long-term loyalty.
Host: So the key business takeaway is to shift our thinking.
Expert: Exactly. The value of AI in these roles isn't just in the functional task it completes, but in the supportive, adaptive relationship it builds with the user. It’s the difference between an automated tool and a true digital partner.
Host: A fantastic insight. So, to summarize: today's professionals face real challenges with motivation and time management. This study gives us a three-layer framework—Relationship, Matching, and Service—to build AI companions that truly help. For businesses, this opens up new possibilities in corporate training, EdTech, and even customer relations.
Host: Alex, thank you so much for translating this complex study into such clear, actionable insights.
Expert: My pleasure, Anna.
Host: And thank you to our audience for tuning in. This has been A.I.S. Insights — powered by Living Knowledge. Join us next time as we uncover more valuable knowledge for your business.
Conversational Agent, Education, Virtual Learning Companion, Design Knowledge, Value
