ECIS 2026 (2026) AI Processed
DESIGNING FOR IMPERFECT PROMPTS: A SIMULATION-BASED EVALUATION OF AN LLM-BASED ENTERPRISE DATA ASSISTANT USING KNOWLEDGE GRAPHS
This study designs and evaluates a semantic mediation layer to automatically resolve imperfect natural-language queries from business users. The authors compare two architectural variants: schema-based mediation utilizing semantic similarity and knowledge graph (KG)-based mediation constructed from database schemas. The evaluation is performed through a simulation using real-world interaction logs and validated using human annotators and LLM-as-a-Judge frameworks.
Problem
Business users frequently submit brief, incomplete, and structurally deficient prompts containing domain-specific terminology rather than executable database instructions. Standalone large language models fail to align these imperfect inputs with underlying database schemas, leading to inaccurate queries and incorrect data retrieval. Prior work relying on manual prompt engineering and few-shot examples is highly labor-intensive and fails to scale effectively.
Outcome
- The KG-based mediation statistically outperformed the schema-based mediation across Precision, Recall, and F1-Score metrics.\n- For GPT-4.1, the KG-based approach achieved a Precision of 90.67% (compared to 69.05% for schema-based) and an F1-Score of 89.47% (compared to 71.02% for schema-based).\n- For Claude Sonnet 4, the KG-based approach achieved a Precision of 87.23% (compared to 75.17% for schema-based) and an F1-Score of 90.29% (compared to 82.17% for schema-based), while Recall remained statistically comparable (93.52% vs 92.56%).\n- The study is limited by its focus on a single-enterprise setting within a CFO unit, a finite dataset of 158 interactions from 29 users, and reliance on a simulation-based evaluation instead of live field deployments.
What it means for you
- CIO / IT Executive: Issue a directive to transition the enterprise's LLM-based data assistant architecture away from simple schema-based semantic similarity matching and onto a Knowledge Graph-based mediation layer constructed from database schemas to capture the demonstrated 12% to 21% precision improvements.
- IT Manager: Instruct your database and engineering teams to begin constructing a Knowledge Graph from your current database schemas to serve as a semantic mediation layer for your LLM assistant, replacing manual prompt engineering.
- Business Strategist: Launch a pilot program within the CFO business unit to deploy the Knowledge Graph-based data assistant, leveraging its verified ability to resolve incomplete, domain-specific prompts into accurate database queries for business users.
- Researcher: Draft a research proposal to test the Knowledge Graph-based mediation layer in a live, real-world field deployment to gather empirical performance data beyond the current study's simulation of 158 interactions across 29 users.
- Policymaker: Draft an internal data governance policy requiring that all natural-language LLM interfaces querying corporate databases must use a schema-derived Knowledge Graph mediation layer to ensure structural query alignment and prevent inaccurate data retrieval.
Transcript
Host: Welcome to A.I.S. Insights - Turning IS Research into Business Action. I'm your host, Aaron Ivan Sterling. Today, we are exploring a study presented at the 2026 European Conference on Information Systems, or ECIS. The study is titled "DESIGNING FOR IMPERFECT PROMPTS: A SIMULATION-BASED EVALUATION OF AN LLM-BASED ENTERPRISE DATA ASSISTANT USING KNOWLEDGE GRAPHS." Joining me to break down this research is our expert analyst, Ava Irene Solis. Welcome, Ava.
Expert: Thank you, Aaron. It is great to be here to discuss this study, which focuses on a very practical challenge: how to help business users get accurate data from enterprise databases without needing to know SQL or complex database structures.
Host: This is indeed a major issue for organizations trying to implement AI-driven data assistants. What is the core problem the study addresses?
Expert: The core issue is what the authors call "imperfect prompts." When business users query a database using natural language, they rarely provide technically perfect instructions. Instead, they write brief, incomplete prompts using domain-specific jargon or shorthand. For instance, in the study's real-world setting within the CFO unit of a multinational energy corporation, prompts averaged just ten words and never referenced actual database schema elements.
Host: And I imagine standard Large Language Models struggle with that.
Expert: Exactly. Standalone LLMs often fail to map these brief inputs to the actual database schemas. They might guess the wrong tables, anchor on similar-sounding but incorrect columns, or hallucinate join relationships. Organizations have tried to fix this through manual prompt engineering and setting up few-shot examples, but that process is highly labor-intensive and simply does not scale effectively as databases grow and change.
Host: So, how did the researchers behind this study approach this challenge?
Expert: They designed and evaluated a semantic mediation layer. This is an intermediate unit that sits between the user and the database assistant to automatically enrich and resolve those imperfect prompts. Specifically, they compared two architectural variants for this layer. The first is schema-based mediation, which uses semantic similarity to find relevant tables. The second is knowledge graph-based mediation, where a lightweight knowledge graph is automatically constructed from the database schema to capture tables, columns, and their structural relationships, like join paths.
Host: That sounds like a highly structured approach. What did the evaluation show?
Expert: The study evaluated these approaches using a simulation based on 158 real-world interaction logs from 29 users in that CFO unit, validated by both human annotators and LLM-as-a-Judge frameworks. The results showed that the knowledge graph-based mediation statistically outperformed the schema-based mediation across Precision, Recall, and F1-Score metrics.
Host: Do we have specific numbers on that performance difference?
Expert: Yes, we do. For example, when using GPT-4.1, the knowledge graph-based approach achieved a Precision of 90.67% compared to 69.05% for the schema-based approach. Its F1-Score was 89.47% compared to 71.02% for the schema-based variant.
Host: And what about other models? Did they test more than one?
Expert: They also tested Claude Sonnet 4. With that model, the knowledge graph-based approach achieved a Precision of 87.23% compared to 75.17% for the schema-based variant, and an F1-Score of 90.29% compared to 82.17%. Interestingly, the Recall rates between the two mediation variants were statistically comparable when using Claude, at 93.52% for the knowledge graph and 92.56% for schema-based.
Host: Those are clear metrics, but we should also mention any boundaries or limitations of these findings, right?
Expert: Absolutely. It is important to note that the study was conducted within a single-enterprise setting, specifically a CFO unit, and used a finite dataset of 158 interactions from 29 users. Also, the evaluation was simulation-based rather than a live field deployment, so future real-world applications under different network structures might vary.
Host: Given those findings, what can business and technology practitioners actually do with this knowledge?
Expert: The practical value here is that instead of spending countless hours manually writing prompts or curating few-shot examples, developers can automate the creation of a lightweight knowledge graph directly from their database schemas. By implementing this semantic mediation layer, organizations can improve the accuracy of their internal data assistants, reduce downstream query failures, and make data retrieval much more efficient, even when users write very brief or vague queries.
Host: So, it essentially bridges the "semantic gap" between what the user actually means and how the database is structured.
Expert: Exactly. It meets the users where they are, using their own vocabulary, without requiring them to learn database structures or SQL, while keeping the background maintenance of the AI system manageable.
Host: That is a very clear and actionable takeaway. Thank you, Ava, for sharing these insights with us.
Expert: My pleasure, Aaron.
Host: And thank you to our listeners for tuning in to A.I.S. Insights. We hope this research helps you optimize your enterprise data strategies. See you next time.
Expert: Thank you, Aaron. It is great to be here to discuss this study, which focuses on a very practical challenge: how to help business users get accurate data from enterprise databases without needing to know SQL or complex database structures.
Host: This is indeed a major issue for organizations trying to implement AI-driven data assistants. What is the core problem the study addresses?
Expert: The core issue is what the authors call "imperfect prompts." When business users query a database using natural language, they rarely provide technically perfect instructions. Instead, they write brief, incomplete prompts using domain-specific jargon or shorthand. For instance, in the study's real-world setting within the CFO unit of a multinational energy corporation, prompts averaged just ten words and never referenced actual database schema elements.
Host: And I imagine standard Large Language Models struggle with that.
Expert: Exactly. Standalone LLMs often fail to map these brief inputs to the actual database schemas. They might guess the wrong tables, anchor on similar-sounding but incorrect columns, or hallucinate join relationships. Organizations have tried to fix this through manual prompt engineering and setting up few-shot examples, but that process is highly labor-intensive and simply does not scale effectively as databases grow and change.
Host: So, how did the researchers behind this study approach this challenge?
Expert: They designed and evaluated a semantic mediation layer. This is an intermediate unit that sits between the user and the database assistant to automatically enrich and resolve those imperfect prompts. Specifically, they compared two architectural variants for this layer. The first is schema-based mediation, which uses semantic similarity to find relevant tables. The second is knowledge graph-based mediation, where a lightweight knowledge graph is automatically constructed from the database schema to capture tables, columns, and their structural relationships, like join paths.
Host: That sounds like a highly structured approach. What did the evaluation show?
Expert: The study evaluated these approaches using a simulation based on 158 real-world interaction logs from 29 users in that CFO unit, validated by both human annotators and LLM-as-a-Judge frameworks. The results showed that the knowledge graph-based mediation statistically outperformed the schema-based mediation across Precision, Recall, and F1-Score metrics.
Host: Do we have specific numbers on that performance difference?
Expert: Yes, we do. For example, when using GPT-4.1, the knowledge graph-based approach achieved a Precision of 90.67% compared to 69.05% for the schema-based approach. Its F1-Score was 89.47% compared to 71.02% for the schema-based variant.
Host: And what about other models? Did they test more than one?
Expert: They also tested Claude Sonnet 4. With that model, the knowledge graph-based approach achieved a Precision of 87.23% compared to 75.17% for the schema-based variant, and an F1-Score of 90.29% compared to 82.17%. Interestingly, the Recall rates between the two mediation variants were statistically comparable when using Claude, at 93.52% for the knowledge graph and 92.56% for schema-based.
Host: Those are clear metrics, but we should also mention any boundaries or limitations of these findings, right?
Expert: Absolutely. It is important to note that the study was conducted within a single-enterprise setting, specifically a CFO unit, and used a finite dataset of 158 interactions from 29 users. Also, the evaluation was simulation-based rather than a live field deployment, so future real-world applications under different network structures might vary.
Host: Given those findings, what can business and technology practitioners actually do with this knowledge?
Expert: The practical value here is that instead of spending countless hours manually writing prompts or curating few-shot examples, developers can automate the creation of a lightweight knowledge graph directly from their database schemas. By implementing this semantic mediation layer, organizations can improve the accuracy of their internal data assistants, reduce downstream query failures, and make data retrieval much more efficient, even when users write very brief or vague queries.
Host: So, it essentially bridges the "semantic gap" between what the user actually means and how the database is structured.
Expert: Exactly. It meets the users where they are, using their own vocabulary, without requiring them to learn database structures or SQL, while keeping the background maintenance of the AI system manageable.
Host: That is a very clear and actionable takeaway. Thank you, Ava, for sharing these insights with us.
Expert: My pleasure, Aaron.
Host: And thank you to our listeners for tuning in to A.I.S. Insights. We hope this research helps you optimize your enterprise data strategies. See you next time.