A customer types “resize jpg in Acrobat.” Another asks, “Unable to delete PDF content need help.” Neither resembles the carefully named task topics we tend to write in technical documentation. Both came from real customer queries.
The gap between the language customers use to ask questions about products and the language documentation teams use sits at the center of a 2024 study from researchers at the University of Massachusetts Amherst and Adobe Research.
Their paper, KaPQA: Knowledge-Augmented Product Question-Answering, examined whether adding product-specific knowledge to a retrieval-augmented generation system could improve answers to questions about Adobe Acrobat and Photoshop.
The Gist
Customer questions rarely match the way documentation is written. Before the AI can search for an answer, it has to guess what the customer is asking. To help with that guessing, the researchers built a simple knowledge base of relationships pulled from the documentation itself — things like rotation handle → rotates → text box — and used it to rewrite vague customer questions into clearer ones before searching.
It worked, but barely. And in some cases, the "help" made things worse.
When Extra Help Backfired
One customer in the study wanted to split a large PDF into smaller files. The system pulled in product knowledge about reducing file size — a related but different task — and rewrote the question into “How can I reduce the size of a PDF file?,” sending the search in the wrong direction.
The docs had the answer that the system needed. The rewriting step introduced the confusion, by pulling in information that was topically close to the question but not what the customer actually meant.
When The Correct Answer Ranks Too Low, The AI Gets It Wrong
Even when the system retrieved the correct documentation, that wasn’t the end of the hallucination story. If the correct source showed up lower in the list of results handed to the AI, it was more likely to generate a wrong answer.
And for questions that were already clear and well-formed, the same “helpful” generative AI rewriting step made answers worse, not better. Often, it nudged good questions in the wrong direction.
What This Means For Documentation Teams
This isn’t proof that structured authoring, taxonomies, or knowledge graphs fix AI retrieval. The study didn’t test that. What it does show is that getting an AI to connect a customer’s words with the right product knowledge, while ignoring nearby-but-wrong knowledge, is difficult.
Tech writers already make the calls that shape how clearly a procedure names its object and action, whether two similar tasks are distinguished by name, and the extent to which the relevant conditions and outcomes are spelled out. Those choices shape what’s available to the people reading the docs; and, it turns out, to a retrieval system trying to make sense of them.
What’s Still Unanswered
The researchers didn’t test whether making those distinctions clearer in the source docs would have prevented the mix-up. That’s the open question this study leaves for us to investigate on our own.
We’ve got work to do. Get busy. 🤠