XYBench: Helping with the problem behind the question

Back to Articles XYBench: Helping with the problem behind the question Enterprise Article Published November 13, 2026 Upvote - Kyle Wiggers Ai2Comms Follow allenai 📄 Paper | 🤗 Data | 💻 Code Truly useful AI assistants need to be able to help users who ask misguided questions—not just users who ask the right question with the right amount of detail.

We’re releasing XYBench, a benchmark of 8K+ questions that tests how AI assistants respond to requests based on misconceptions, such as asking how to thicken frosting that keeps sliding off cupcakes when the real problem is that the cupcakes are still too warm to frost.

XYBench won the Best Resource Paper Award at EMNLP 2026.A mismatch between the approach someone asks about and the goal they want to achieve is known as the XY problem: a person wants to accomplish X, assumes a particular approach Y will work, and asks for help with Y.

In the cupcake example, the person is so committed to making the frosting stick that they fail to consider the warmth of the cupcake as the true problem to be addressed.

Developed with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench exposes a critical gap in LLMs' ability to troubleshoot problems: recognizing when a user's request is based on an implicit misconception and appropriately redirecting them toward a solution that addresses their actual goal.

Drawing on 8,115 real-world questions spanning programming and everyday activities such as cooking and home repair, we introduce a benchmark that measures whether models identify these misconceptions and are able to prioritize effective solutions in their responses.

The models we evaluated on XYBench – Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro – often suggested a solution to the user’s problem but kept their answers focused on the original, mistaken approach.

Unlike human reference answers, all five models were less likely to steer users away from that approach and toward a solution suited to their goal.People use AI systems with different levels of knowledge and may not always know how best to approach a problem.

We hope XYBench helps researchers measure whether model improvements make assistants better at recognizing mistaken assumptions, explaining why a proposed approach might not work, and suggesting more effective alternatives—even when users don't know the right questions to ask.

Building XYBench We built XYBench from four sources: 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides.

For each question, we identified the stated request (Y), what the person explicitly asks for, and the intended request (X)—the underlying goal that can be inferred from context.

For the public forum questions, we used the three highest-scored human answers and the accepted answer, where available, to establish the intended request and possible solutions.We filtered out cases where answering the stated request was judged sufficient.For WikiHow, we used Sonnet 4.

5 to generate questions around plausible misconceptions in procedural guides, then refined them to include enough context for an expert to infer the underlying goal without stating it outright.We removed examples where answering the literal request would largely resolve the problem.

Evaluating how an answer helps A model might spend several paragraphs recommending a more powerful vacuum cleaner, then briefly suggest checking for a clogged hose.If a clog is causing the weak suction, clearing it could solve the problem.

A more helpful answer would recognize that the user wants to restore suction and focus on diagnosing the cause.This draws on pragmatics, the branch of linguistics concerned with how context and communicative intent shape meaning.

To test whether models use this understanding in their answers, XYBench measures three aspects of each response: Pragmatic solution—records whether a model’s response includes solutions to the stated request (Y) and the intended request (X), checking each separately.

Pragmatic emphasis—measures whether a model’s answer focuses on the stated request (Y), focuses on the intended request (X), or balances the two.

Misconception identification—measures whether a model’s response steers the user toward the underlying goal (X) while discouraging or avoiding the mistaken approach (Y).In XYBench, Claude Sonnet 4.5 serves as the automated judge.

It classifies individual suggestions within each model’s response and assesses the answer’s pragmatic emphasis.To validate Sonnet 4.5’s judgments, two human reviewers independently assessed 500 model responses to XYBench questions.

Agreement with the judge, averaged across the reviewers, ranged from 97.5–100% on whether a solution appeared, 84.6–93.3% on misconception identification, and 74.2–79.8% on pragmatic emphasis.What the experiments show We evaluated Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.

5-9B, and DeepSeek-V4 Pro on XYBench in a single-turn setting, with one response per question.We evaluated human reference answers separately using the same metrics.Across all four sources, every model was more likely to include a solution to the stated request (Y) than to the underlying goal (X).

On Stack Overflow, stated-request solutions appeared in 75–86% of responses, compared with 59–71% for underlying-goal solutions.A passing mention of a solution does not make it the focus of an answer.

In the vacuum example, a model could briefly suggest unclogging the hose but focus most of its response on buying a more powerful vacuum.We would count that response as containing a solution to the underlying goal, even though the model emphasized the stated request.

We measured overall emphasis separately: on Stack Overflow, models focused primarily on the underlying goal in only 35–41% of their responses, and on the stated request in 47–57%.In the remaining responses, models gave similar emphasis to the stated request and the underlying goal.

We also tested whether making the user’s goal explicit would help.In an oracle experiment, we supplied both the stated request (Y) and the intended request (X) alongside each question.On Stack Overflow, Sonnet 4.

6’s rate of including a solution to the underlying goal rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%.Across the sources with human reference answers, every model in this experiment still trailed humans on misconception identification.

Supplying intent helped, but did not consistently close the gap with human responses.This suggests that detecting intent is only part of the challenge: models also need to act on it when it diverges from the stated request.

A separate experiment tested whether models could identify useful advice when judging responses instead of writing them.We gave the same five models, plus GPT-4.

1, each question with two possible answers: one focused on the literal request and one explaining the misconception and addressing the underlying goal.On Stack Overflow, all six chose the goal-focused answer in 71–84% of cases.

By comparison, only 35–41% of the answers written by the five models in our main evaluation primarily focused on the underlying goal.The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.

What this means for evaluation Researchers can use XYBench to test whether changes to prompting or training help models identify misconceptions, offer solutions to users’ underlying problems, and make those solutions the focus of their answers.

Measuring each behavior individually lets researchers see whether progress in one also improves the others.Future work could test which training methods help models act on inferred intent—AKA what users need and not just what they literally asked for.

The XYBench currently covers English-language queries and mainly procedural problems; extending it to other languages, cultures, and conversational settings remains an open direction.

We hope XYBench enables researchers to build models that offer useful advice even when a user’s first question rests on a mistaken assumption.

Datasets mentioned in this article 1 More from this author ToyBlocks: A controlled world for robot learning and evaluation 26 November 6, 2026 StateBench: Testing how language models track state changes over long documents 17 November 3, 2026 Community EditPreview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images Comment · Sign up or log in to comment Upvote - Datasets mentioned in this article 1