Back to the library

Key question New

AI-processed · Author-approved

Why do professionals follow AI recommendations despite doubts about their practical value, and how does this impact downstream decisions?

Ethical Uncertainty in Human-AI Collaborative Decision-Making: A Study of AI-Assisted Radiology
Yao Ma, Ela Klecun · International Conference on Information Systems (ICIS) · 2026
Illustration for: Why do professionals follow AI recommendations despite doubts about their practical value, and how does this impact downstream decisions?

Loads Spotify's player; Spotify may set cookies. This device remembers your choice.

Key findings

  • While professionals comfortably override obvious AI errors, they hesitate to dismiss clinically insignificant AI findings due to moral conflicts between patient disclosure and avoiding unnecessary patient anxiety.
  • Situational pressures, such as patient disputes over inconsistent reports and the auditability of AI-flagged records in malpractice litigation, drive professionals toward defensive adherence to AI outputs.
  • Professionals exhibit precautionary adherence by reporting all AI-flagged findings to shield themselves legally, which cascades decision-making burdens and unnecessary testing onto downstream physicians and patients.
  • Professionals also utilize signalling adherence by adopting subtle hedging language in reports to informally alert subsequent decision-makers about their diagnostic reservations.
  • The findings suggest system designers and regulators should introduce interactive buffer mechanisms and non-binary reporting options rather than forcing rigid accept-or-override workflows.

What it means for you

CIO / IT Executive

Initiate a procurement review to mandate that all AI vendor contracts include non-binary reporting options and interactive buffer mechanisms, moving away from rigid accept-or-override workflows.

IT Manager

Configure the radiology reporting software interface to remove binary forced-choice prompts for AI-flagged nodules and add a standardized 'hedging/clinical reservation' tagging option.

Business Strategist

Partner with hospital legal counsel and risk management on Monday morning to draft institutional liability guidelines that protect clinicians from malpractice penalties when legitimately dismissing clinically insignificant AI findings.

Researcher

Design a qualitative study to track the downstream cascading burden and patient anxiety caused by precautionary adherence to AI-flagged incidentalomas in tertiary care settings.

Policymaker

Draft regulatory guidance clarifying that failure to follow unvalidated, clinically insignificant AI screening flags does not constitute negligence in medical malpractice litigation.

Summary

This qualitative study investigates how healthcare professionals perceive and exercise discretion when collaborating with AI under conditions of ethical uncertainty. The authors conducted field observations, archival analysis, and 39 interviews across three tertiary hospitals in China implementing AI-assisted lung nodule CT screening.

Problem

Prior research on human-AI collaboration predominantly focuses on knowledge uncertainty and opacity, presuming clear standards for what constitutes a correct outcome. However, in professional settings, workers often encounter ethical uncertainty where competing ethical obligations and legal liability make it ambiguous which course of action is right.

Transcript
Host: Welcome to A.I.S. Insights — powered by Living Knowledge. I’m Anna Ivy Summers. Today we are looking at a study titled "Ethical Uncertainty in Human-AI Collaborative Decision-Making: A Study of AI-Assisted Radiology," presented at the International Conference on Information Systems in 2026 by researchers Yao Ma and Ela Klecun. Alex, this study examines how professionals exercise discretion when collaborating with AI tools.

Expert: That's right, Anna. In many corporate and professional settings, the assumption is that AI augmentation is simply a knowledge problem: the AI makes a prediction, and the expert corrects it if the model lacks context or makes a factual error. But this study shows that decisions often involve ethical uncertainty, where multiple choices are defensible, yet each involves competing ethical principles and trade-offs.

Host: What specific problem did the authors investigate to understand this dynamic?

Expert: They examined chest CT screening for lung nodules across three major hospitals. The AI deployed there is highly sensitive—detecting tiny micro-nodules down to fractions of a millimeter. While radiologists could easily spot and discard obvious system errors like blood vessels, they faced an ethical dilemma with tiny, clinically insignificant nodules. Overriding the AI could risk missing a rare early malignancy, while reporting it causes patient anxiety and triggers costly, potentially harmful follow-up procedures.

Host: How did the researchers gather their data?

Expert: They conducted a qualitative field study over two years across three Tertiary Class-A hospitals in China. This included over forty hours of direct observation in radiology departments, thirty-nine formal interviews with clinicians and technical specialists, and an analysis of institutional guidelines and regulatory documents.

Host: And what did they find about how these professionals handle AI outputs under pressure?

Expert: Rather than using their clinical discretion to filter out insignificant nodules, professionals felt compelled to adhere to the AI’s suggestions. This was driven by two situational pressures: patient complaints about discrepancies between hospitals, and legal liability concerns. Because AI outputs are logged in hospital databases, omitting a flagged nodule creates an audit trail that could be used against a physician in malpractice disputes.

Host: How did that adherence manifest in practice?

Expert: The study identified two distinct patterns. The first is precautionary adherence. Here, radiologists include every detected nodule in their report to protect themselves institutionally, despite knowing most carry minimal diagnostic value. This cascades the ethical burden downstream to oncologists and surgeons, who then face dilemmas over ordering unnecessary follow-up scans or invasive biopsies.

Host: And what was the second pattern?

Expert: The second is signalling adherence. In this case, radiologists still report the AI-flagged finding, but they deliberately use hedging terms like "shadow" or "opacity" instead of "nodule." This subtle wording informally signals their skepticism to downstream physicians without creating legal exposure on the formal record.

Host: What are the key takeaways for business leaders and technology designers from this study?

Expert: First, organizations need to recognize that deploying highly sensitive AI often generates new categories of ambiguous tasks. Second, binary workflows—forcing a user simply to accept or reject an AI recommendation—fail to accommodate professional nuances. The study suggests designing digital buffer mechanisms where professionals can document reasoning and contextual caveats for subsequent decision-makers. Finally, regulators and developers should explore continuous feedback loops, allowing systems to learn from situated human interpretations rather than locking them into rigid, frozen models.

Host: A clear reminder that human-in-the-loop systems require thoughtful organizational and workflow design, not just algorithmic precision. Thank you for breaking down this study, Alex, and thank you to our listeners for tuning into A.I.S. Insights — powered by Living Knowledge.
Artificial Intelligence & Automation Information Systems & Management Human-Computer Interaction Data Analytics & Big Data Digital Transformation & Strategy Cybersecurity & Privacy Cloud & Infrastructure E-Commerce & Digital Platforms Operations & Supply Chain Sustainability & Green IS

Ethical uncertainty, human-AI collaboration, professional discretion, artificial intelligence, clinical decision-making, radiology