Back to the library

Key question New

AI-processed · Author-approved

How can leaders measure whether an AI model's decisions match their organization's specific practices and reasoning?

Measuring Contextualized AI Alignment in Organizational Decision-Making
Niklas Weller, Ivo Blohm · International Conference on Information Systems (ICIS) · 2026
Illustration for: How can leaders measure whether an AI model's decisions match their organization's specific practices and reasoning?

Loads Spotify's player; Spotify may set cookies. This device remembers your choice.

Key findings

  • Baseline policy alignment varied widely across the ten evaluated models, ranging from negative alignment (-0.211) to strong positive alignment (0.844), with general benchmark scores and pricing failing to predict contextual fit.
  • Providing explicit summaries of organizational cue weighting in prompts significantly improved decision policy alignment for models that started with poor baseline alignment.
  • Policy alignment strongly correlated with outcome accuracy across conditions (r = 0.85, p < 0.001), with decision errors coinciding with severe policy divergences.
  • Highly aligned models showed diminishing returns under prompt interventions, and one model significantly regressed due to cue overadjustment, indicating that prompt adjustments do not guarantee uniform improvement.
  • Residual policy gaps after prompt interventions offer a practical upper-bound estimate of the tacit knowledge that textual prompts cannot convey, identifying where human-in-the-loop oversight remains necessary.

What it means for you

CIO / IT Executive

Stop selecting LLMs based solely on generic benchmark scores or pricing tiers; on Monday morning, mandate that any AI vendor selection process must include an empirical cue-weighting audit to measure baseline alignment with your organization's specific historical decision policies.

IT Manager

Pull a sample of 50 recent historical organizational decisions, extract the primary weighting cues, and test your current production LLM against them to establish its baseline CALM score before deploying any new prompt-based interventions.

Business Strategist

Draft an explicit, written summary of your organization's unwritten decision rules and weighting policies to incorporate into system prompts, while carving out a mandatory human-in-the-loop review workflow for high-stakes decisions where textual prompts hit their residual tacit knowledge limit.

Researcher

Design a controlled evaluation protocol that measures cue-weighting policy divergence rather than relying solely on final outcome accuracy, specifically testing for model regression caused by prompt-induced cue overadjustment.

Policymaker

Draft procurement and deployment standards requiring high-risk automated decision systems to publish their cue-weighting alignment metrics against established organizational baselines rather than relying on unverified general capability claims.

Summary

This study introduces the Contextualized Alignment Lens Model (CALM), a method that compares the cue-weighting decision policies of large language models against historical organizational decisions. The authors evaluated the approach across ten large language models generating 30,000 legal decisions on 1,000 European Court of Human Rights cases. The evaluation tested baseline model policies and measured the impact of two prompt-based interventions designed to externalize organizational decision rules.

Problem

Generic AI alignment benchmarks evaluate broad human values but fail to measure whether models follow the specific, often unwritten rules and practices of a particular organization. Furthermore, relying only on final decision accuracy can mask shortcut reasoning, making it difficult to detect when an AI relies on irrelevant or biased features.

Transcript
Host: Welcome to A.I.S. Insights — powered by Living Knowledge. I'm Anna Ivy Summers.

Expert: And I'm Alex Ian Sutherland.

Host: Today we're examining a study titled "Measuring Contextualized AI Alignment in Organizational Decision-Making," authored by Niklas Weller and Ivo Blohm, presented at the International Conference on Information Systems in 2026. This study addresses a critical operational question: how do we know if a large language model's decision-making actually aligns with the specific, often unwritten practices of an organization?

Expert: That's right, Anna. Most existing alignment research evaluates whether an AI system conforms to broad human values like general helpfulness or safety. But organizations operate within distinct sociotechnical environments governed by specialized routines, precedents, and tacit knowledge. When an LLM is deployed for tasks like hiring, loan approvals, or legal assessment, it usually relies only on the text provided in the prompt. That leaves a knowledge gap between the model and the organization's actual operational practice.

Host: And looking only at whether the AI gets the final answer right isn't sufficient, is it?

Expert: Exactly. In organizational decision-making, an AI can achieve high outcome accuracy through shortcut learning—relying on irrelevant or biased cues that happen to correlate with the outcome in a specific dataset. If the context shifts, that decision policy can fail. Evaluating policy alignment means checking whether the model relies on the same decision-relevant features that an informed human decision-maker would use.

Host: To measure this, the study introduces CALM, the Contextualized Alignment Lens Model. How does that work?

Expert: CALM adapts the Brunswik lens model from psychology. It treats both the organization and the LLM as judges evaluating the same set of observable cues. By regressing past organizational decisions on those cues, the authors derive an organizational policy vector, alpha. They then regress the LLM's decisions on the same cues to get a model policy vector, beta. Policy alignment is calculated as the cosine similarity between those two vectors, ranging from minus one to plus one.

Host: The authors evaluated CALM across ten language models making 30,000 decisions on 1,000 European Court of Human Rights Article 6 cases, using 45 coded factual cues. What did the baseline results show?

Expert: Baseline alignment varied substantially, ranging from minus 0.211 for GPT-5.4-nano to positive 0.844 for GPT-5.4-mini. Four of the ten models exhibited baseline alignment that was statistically indistinguishable from zero. Importantly, general benchmark rankings and pricing tiers did not predict contextual fit. For instance, Claude Haiku 4.5 was roughly five times more expensive per token than Grok 4.1 Fast, yet Haiku recorded a baseline policy alignment of minus 0.057, whereas Grok scored positive 0.842.

Host: The study also tested two prompt interventions to provide externalized decision rules to the models. What happened there?

Expert: Providing explicit cue-weighting guidance produced significant improvements for models starting with poor baseline alignment. Claude Haiku 4.5 shifted from minus 0.057 to positive 0.625, and GPT-5.4-nano improved from minus 0.211 to positive 0.695. However, models with already high baseline alignment showed modest gains due to ceiling effects. Additionally, prompt interventions did not work uniformly: Grok 4.1 Fast regressed significantly under introspective guidance, dropping to 0.496 because it overadjusted on targeted cues and shifted weights on non-targeted cues.

Host: That indicates prompt engineering requires diagnostic verification rather than assuming prompts will always improve behavior. How did policy alignment relate to overall accuracy?

Expert: Policy alignment strongly correlated with outcome accuracy across all thirty model-condition combinations, with a Pearson r of 0.85. A deeper analysis revealed that on cases where models reached the correct outcome, their average policy alignment was positive 0.811, but on incorrect cases, it dropped to minus 0.303. Decision errors directly coincided with policy divergence.

Host: What are the main practical takeaways for business leaders deploying LLMs?

Expert: First, evaluate off-the-shelf models on baseline contextual fit and intervention responsiveness rather than general public leaderboards. Second, CALM provides a bounded estimate for automation readiness. The residual policy gap remaining after prompt interventions indicates the upper limit of tacit knowledge that text prompts cannot convey. Where that residual gap remains wide, human-in-the-loop oversight is empirically justified.

Host: That provides a concrete method for auditing decision policies before deployment. Thank you, Alex, and thank you to our listeners. Join us next time on A.I.S. Insights — powered by Living Knowledge.
Artificial Intelligence & Automation Information Systems & Management Human-Computer Interaction Data Analytics & Big Data Digital Transformation & Strategy Cybersecurity & Privacy Cloud & Infrastructure E-Commerce & Digital Platforms Operations & Supply Chain Sustainability & Green IS

AI alignment, large language models, decision-making, lens model, organizational knowledge, explainable AI