Key question New
AI-processed · Author-approved
How can managers measure whether employees are critically and effectively working with generative AI tools?
Loads Spotify's player; Spotify may set cookies. This device remembers your choice.
Key findings
- Conceptualized critical engagement with generative AI as a reflective construct containing two sub-dimensions: Thoughtful Structuring of Input and Skeptical Review of Output.
- Reduced an initial 34-item pool to 18 items through formal content validity testing with 34 evaluators, establishing correspondence and distinctiveness.
- Demonstrated support for the two-factor structure through exploratory factor analysis with 153 respondents (KMO = 0.860), establishing a refined 15-item measurement model.
- Found that skepticism toward AI outputs directly prompts more controlled, step-by-step input behaviors and guardrails among knowledge workers.
What it means for you
CIO / IT Executive
Allocate Q3 budget to transition enterprise AI training away from basic prompt-writing tutorials and fund programs focused on cognitive oversight, specifically teaching staff how to conduct systematic skeptical reviews of AI-generated code and text.
IT Manager
Update your team's code-review and document-drafting checklists to explicitly require a documented 'skeptical review' step, ensuring engineers do not blindly copy-paste probabilistic AI outputs without testing edge cases.
Business Strategist
Audit your department's current AI-driven workflows to identify where deterministic assumptions are being made, and mandate a step-by-step 'thoughtful structuring of input' protocol for any high-stakes market analysis or forecasting tasks.
Researcher
Use the validated 15-item measurement scale from this study to design a survey measuring baseline critical engagement levels across a cohort of enterprise knowledge workers before rolling out a new generative AI tool.
Policymaker
Draft internal corporate governance guidelines specifying that high-impact AI use cases (such as compliance or HR) require human-in-the-loop verification steps to counter non-deterministic hallucinations and probabilistic errors.
Summary
This study develops and validates a measurement scale to evaluate how knowledge workers critically engage with generative artificial intelligence tools. Grounded in critical thinking and problem-solving literature, the authors conducted focus groups with experienced professionals, evaluated content validity with 34 judges, and performed exploratory factor analysis using data from 153 respondents.
Problem
Generative AI produces probabilistic, non-deterministic outputs that can introduce subtle errors and hallucinations, demanding cognitive oversight rather than superficial prompt-writing skills. Prior information systems research focuses primarily on deterministic tools and observable behaviors, leaving a gap in measuring the underlying cognitive capabilities needed to skillfully interact with generative AI.
Transcript
Expert: And I am Alex Ian Sutherland. Today, we examine a study titled "Critical Engagement with Generative AI: Development and Validation of a Scale." The study was conducted by Anjali Tripathi, Soumyakanti Chakraborty, Abhipsa Pal, Rajiv Kumar, and Tejaswini Herath, and presented at the International Conference on Information Systems in 2026.
Host: This study addresses how knowledge workers interact with generative AI. Alex, many organizations have adopted tools like ChatGPT, but measuring how effectively employees actually work with them has been a challenge. Why does generative AI require a dedicated measurement approach?
Expert: Most existing frameworks in information systems were developed for deterministic software, such as enterprise databases or electronic health records. In those tools, system rules are fixed, and inputs yield predictable outputs. Generative AI is fundamentally different. It is probabilistic, non-deterministic, and conversational. It can produce highly convincing, well-phrased responses that nonetheless contain factual errors, embedded bias, or hallucinations. Because these tools present uncertain claims with apparent confidence, knowledge workers face the risk of overreliance and automation bias.
Host: Many companies have focused primarily on teaching employees prompt engineering. Does the study view prompting skills as sufficient?
Expert: The authors point out that writing prompts is merely an observable, technical behavior. As interfaces evolve, technical prompt formulation becomes more accessible. What matters more is the worker's underlying cognitive capability to frame problems, direct the system, and question the results. The goal of this study was to conceptualize and measure that specific capability, termed critical engagement.
Host: How did the research team design and validate this new scale?
Expert: They followed established scale development procedures. First, they grounded the construct in literature covering critical thinking, problem-solving, and professional skepticism from accounting and auditing. They combined this theory with insights from two rounds of focus groups involving experienced professionals who regularly use generative AI. From this, they drafted an initial pool of 34 items. Next, they evaluated content validity using a panel of 34 judges to test how well the items corresponded to their intended concepts and remained distinct from related ones. This step reduced the pool to 18 items.
Host: And then they tested the scale statistically with users?
Expert: Yes. They conducted an exploratory factor analysis using survey responses from 153 knowledge workers. The statistical checks showed strong sampling adequacy, with a Kaiser-Meyer-Olkin value of 0.860. The analysis confirmed two distinct dimensions and refined the instrument to a final set of 15 items.
Host: What are those two dimensions of critical engagement?
Expert: The first is Thoughtful Structuring of Input. This captures how users deliberately decompose complex tasks into smaller parts, provide background context, and share clear instructions to guide the model. The second is Skeptical Review of Output. This captures how users systematically verify claims, compare outputs against expectations, cross-check across sources, and ask the model to identify its own potential flaws.
Host: Did the data reveal anything notable about how these two dimensions interact?
Expert: An interesting pattern emerged. Certain items originally drafted around task structuring, such as guiding the tool step by step or providing specific source lists, loaded alongside skeptical review items. Focus group data supported this connection: when users are skeptical of the model's reliability, they actively establish guardrails at the input stage to constrain errors before they occur.
Host: Turning to the business application, how can organizational leaders put this 15-item scale to work?
Expert: Organizations can use the scale as a diagnostic instrument to assess how workforce teams interact with generative AI. Instead of offering generic prompting workshops, learning and development leaders can design training programs around the two verified capabilities: decomposing complex problems for input, and critically interrogating outputs. Furthermore, this scale gives managers a metric to monitor whether teams are developing the oversight habits needed to prevent costly hallucinations and unverified assumptions in knowledge-intensive work.
Host: In summary, critical engagement with generative AI depends not just on technical prompts, but on structuring inputs deliberately and evaluating outputs with disciplined skepticism. Thank you for walking us through the findings, Alex.
Expert: My pleasure, Anna.
Host: And thank you for listening to A.I.S. Insights — powered by Living Knowledge. Join us next time as we examine practical research shaping technology and business.
Generative AI, Critical Engagement, Scale Development, Human-AI Interaction, Knowledge Work, Large Language Models