Jurgen Gravestein, who writes the newsletter Teaching Computers How to Talk, recently covered a research project that borrows an idea from food packaging: give people a standardized way to compare AI models before choosing one.
What the Researchers Built
MIT Media Lab and researchers at other institutions created ImpactBench, an open benchmark for evaluating whether AI models support or undermine users’ wellbeing. It covers psychological, physical, and social outcomes through 809 metrics drawn from 26 benchmarks. Researchers evaluated ten models in 48,540 simulated conversations.
Across those evaluations, the models promoted behaviors the benchmark defines as desirable 68.7% of the time and avoided identified harms 53.1% of the time. Those are aggregate results, not scores for any single model.
The research paper describes the evaluation method and its limits.
Why Gravestein Says It Matters
Gravestein points out that regular chatbot users often don't realize how easily these tools can make things up or just tell people what they want to hear. Being upfront about how different AIs act can help people choose the right one to use, he says.
Claude Sonnet 5: What One Label Shows
ImpactBench presents complex evaluation results in a familiar format: a compact nutrition-style label with an overall grade and separate categories beneath it.
I found the label for Claude Sonnet 5. The Anthropic-produced model received an overall grade of C+. Fourteen category grades were grouped by whether they avoided negative impact or promoted positive impact. Arrows beside the grades invite readers to do further learning. There’s also an option to access the “full evaluation.”
Tech writers use this progressive disclosure-type approach to content design. Offer up a useful summary, then make the detail available to readers who need it. Explaining how an AI feature behaves, including its known limitations, belongs in that same documentation practice.
📝 Note: The category grades displayed in the AI nutrition label are more useful than the overall impact score on its own. Overall, Claude Sonnet 5 received a C+ score, with an A- for avoiding factual hallucination and a D- for promoting learning and skill development.
What Tech Writers Can Do With It
AI documentation usually tell you what a tool can do, but rarely does it explain what happens when it makes mistakes or hits its limits. For teams writing this content, the ImpactBench label is a great example of how to show this information clearly so users can quickly compare different tools.
Keep in mind that this test has its own limits. It only measures how the AI handles fake conversations about wellbeing. Good scores don’t guarantee the AI will correctly answer questions about your company’s products. You still must test it on your own specific tasks and data.
Finally, any AI label must include a version and a date (this one doesn’t). AI behavior changes when the system is updated. If we are going to grade AI tools, we need these dates to track and compare those updates over time.🤠