I’ve been researching what happens when AI systems retrieve technical documentation and use it to answer customer questions. Much of that investigation concerns explicitness, particularly around docs that contain the right facts but leaves the relationships between them unstated.
Example: A procedure can be technically correct while omitting conditions that experienced technicians understand and an AI system must infer.
During my research, I came across studies asking whether language models can learn to recognize when they don’t have enough information to answer a question. Some researchers are teaching models to admit uncertainty and decline to answer rather than guess. Depending on what’s missing, a model might also ask the user for more details.
That reminded me of an argument I’ve been having with software developers for decades.
An Old Argument With Software Developers
I’ve spent a big part of my career fixing tech doc issues by getting companies to use structured content like DITA XML. I’ve definitely had software devs question why we even bother. A lot of them would rather write custom software to fix problems we're already solving with good content strategy and structured writing.
I understand their reasoning. Developers are accustomed to solving problems with software. When custom code appears capable of producing the desired result, changing how an organization creates and manages information can seem like an unnecessary complication.
The problem with relying on code alone is that software can assemble docs from reusable content components without determining whether those components contain the explicit information a reader must have to successfully complete a task. For example, an AI-powered search engine can find product instructions without establishing whether they apply to a particular configuration. Content standards give software systems structured information to process. Deciding which facts belong in that information and which relationships must be explicit requires knowledge of the product and how people use it.
I see a related challenge in AI development. Language models can generate plausible answers from retrieved information that is incomplete or doesn’t establish which instructions apply, and researchers are investigating how to make models recognize when they lack the evidence to answer reliably.
Why AI Systems Are Rewarded For Guessing
AI systems are rewarded for guessing because most of the tests used to grade them give them no credit for saying “I don’t know.” A guess might earn points. Admitting uncertainty doesn’t.
It works like a multiple-choice classroom exam that gives no credit for blank answers.
Of course, AI models aren’t human and doesn’t care about scores, but the people who build them do. Devs use scores to decide which versions of a model to release, and many training methods adjust a model toward the kinds of answers that score well. Repeated over many rounds of training and selection, guessing becomes the default behavior.
That’s the argument researchers from OpenAI and Georgia Tech made in a 2026 paper in Nature. They propose scoring that penalizes wrong answers and gives credit for admitting uncertainty, while rewarding correct answers so models aren’t encouraged to refuse questions they can answer.
What The Model Learns Vs. What It’s Given
AI systems can fail to know something in two different ways. The first involves what the model learned during training.
A 2024 study tested a method called Contextualized Knowledge Graph Embedding (CoKE) that teaches a model to recognize when it’s unsure, based on how confident it is in its own predictions. In the researchers’ tests, models trained this way got better at answering questions they knew and declining questions they didn’t.
The second way AI systems can fail to know something involves what the model is given when someone asks a question.
AI assistants that answer product information questions from tech docs don’t rely on training alone. They retrieve documentation content and pass it to the model along with the user’s question. A model can know a lot about industrial sensors in general and still need the product docs to answer questions about a specific model or configuration, including details it never saw in training.
That second kind most directly affects tech writers. The question for us is whether the retrieved docs contain sufficient information to answer what the user asked.
Measuring Whether AI Knows When To Stop
Researchers are building tests to check whether a model says "I don't know" when it should, without refusing to provide trustworthy answers to questions it could answer.
One approach tests models with deliberately unanswerable questions, including questions based on false assumptions or missing information. AbstentionBench, published in June 2025, evaluated 20 frontier models across 20 datasets. Even the largest models struggled to tell when they should say “I don’t know.” And in the researchers’ tests, models given extra training to reason through problems step by step were, on average, 24% worse at recognizing when they shouldn’t answer.
Other approaches examine internal prediction probabilities, compare responses to repeated questions, or use another language model to judge whether an answer is supported by the supplied evidence. Each method has limitations. A model can confidently produce an incorrect answer, repeat the same error consistently, or be evaluated incorrectly by an automated judge.
These evaluations examine model behavior. Documentation teams also need to determine whether the retrieval system supplied enough information for the model to answer.
