New
Communications of the Association for Information Systems (2025) AI Processed
What Does ChatGPT Know About Information Systems?
This study evaluates the depth and quality of domain-specific knowledge in ChatGPT (GPT-3.5) within the field of Information Systems (IS). Using an established IS competency framework, the researchers analyzed ChatGPT's performance on over 3,000 questions gathered from university IS courses and professional certification exams.
Problem
While Generative AI tools demonstrate impressive general academic abilities, their specific knowledge accuracy across specialized fields like Information Systems remains uncertain. Educators and industry professionals lack clear benchmarks regarding whether ChatGPT can reliably produce accurate IS content, pass professional certifications, or answer complex technical queries without hallucinating.
Outcome
- ChatGPT answered 65% to 85% of Information Systems queries correctly across all core framework areas, performing roughly at the level of an average student.
- The model performed significantly better on university course exams than on professional certification exams, failing to reach passing scores for the PMP and Microsoft Azure Administrator tests.
- ChatGPT achieved higher relative accuracy against class averages on closed-book exams compared to open-book or online exam environments where students had external resource access.
- While ChatGPT displayed broad coverage across diverse IS topics, its responses favored listing structured information over demonstrating expert-level critical thinking or complex problem-solving.
- The model performed significantly better on university course exams than on professional certification exams, failing to reach passing scores for the PMP and Microsoft Azure Administrator tests.
- ChatGPT achieved higher relative accuracy against class averages on closed-book exams compared to open-book or online exam environments where students had external resource access.
- While ChatGPT displayed broad coverage across diverse IS topics, its responses favored listing structured information over demonstrating expert-level critical thinking or complex problem-solving.
What it means for you
- CIO / IT Executive: On Monday morning, task your IT training lead to pilot a small internal workshop for a select group of IT professionals focusing on the PMP and Microsoft Azure Administrator certifications, using ChatGPT as a supplementary learning tool to identify specific knowledge gaps and assess its effectiveness in preparing for these professional exams.
- IT Manager: On Monday morning, instruct your team members responsible for generating internal documentation or knowledge base articles to compare the accuracy and completeness of ChatGPT's initial drafts on at least two core Information Systems topics (e.g., database management, network security) against your existing, verified internal resources.
- Business Strategist: On Monday morning, task a market analyst to explore potential use cases for ChatGPT in rapidly generating initial drafts of business process documentation or competitive analysis reports within your IS domain, noting its tendency to provide structured information over deep critical thinking.
- Researcher: On Monday morning, design a controlled experiment to compare ChatGPT's performance on retrieving specific, factual information about IS concepts in a closed-book scenario versus an open-book scenario using publicly available academic papers, to validate the finding about its performance in different exam environments.
- Policymaker: On Monday morning, request a briefing from your education and technology policy advisors on the implications of AI tools like ChatGPT achieving 'average student' performance in IS education, and explore potential guidelines for incorporating such tools responsibly in curriculum development and assessment.