ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Back to Articles ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model Enterprise Article Published October 20, 2026 Upvote 2 Ashwath Vaithinathan Aravindan ashwath-vaithina Follow ibm-research Vatche Isahagian Vatche Follow ibm-research Vinod Muthusamy vinodmut Follow ibm-research Jayaram Radhakrishnan jayaramkr Follow ibm-research Evelyn Duesterwald evduester Follow ibm-research Gaodan Fang gaodan-fang Follow ibm-research Punleuk Oum illeatmyhat Follow ibm-research G Thomas gsthomasx Follow ibm-research Michał Ulewicz Michal Follow ibm-research Merve Unuvar mrvnvr Follow ibm-research Ayhan Sebin ayhansebin Follow ibm-research ALTK-Evolve learns and applies a playbook using GPT-4.

1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.Learned memory takes a basic ReAct agent with GPT-4.1 to the top Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.

5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks.It scores above every retail entry in our October 12, 2026 leaderboard snapshot.It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt.

ALTK-Evolve builds that playbook using GPT-4.1.It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return.

Those guidelines are then added to the agent's prompt for subsequent conversations.The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.Our previous work demonstrated this approach on coding agents.

Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions.

τ²-bench tests policy-bound customer service τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints.

Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow.

The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward.We use its two customer-service domains, retail and airline.

We run τ²-bench's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task.

The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench.

Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list: You are a customer service agent that helps users according to the policy below.Always follow the policy strictly.Use the provided tools when needed.

# Retail agent policy As a retail agent, you can help users: - cancel or modify pending orders - return or exchange delivered orders - modify their default user address … ## Guidelines learned from past task attempts (follow these carefully): - Authenticate the user before taking any order-related action.

If email is unavailable, verify identity via full name and zip code using the finduseridbynamezip API.- Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action.

- When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask.- Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying.

- Before looking a user up, parse and clean their name — e.g.split a combined handle like daikijohnson_9523 into first/last name — and after a couple of failed lookups, state the exact format you need.… We extract guidelines from the agent's trajectories on τ²-bench train tasks using gpt-4.

1, the same model that powers the agent, then apply them unchanged on the held-out test set.Guidelines improve capability and reliability In a prior post, we talked about how different models / tasks require different dosage of guidance.

Inline with the idea, we also evaluate different amounts of guidelines provided to the model.Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off.

We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e.support ≥ 1).Table 1.Guideline configurations and retained guideline counts by domain.

Variant Guidelines included in context Retail Airline baseline no guidelines 0 0 sup3 support n ≥ 3 (high-support core) 31 14 lossless all guidelines 292 101 The metrics we track.

Every task is attempted 4 times, and we read those trials two ways: Mean Accuracy — the average success rate across the 4 trials.This is the capability axis: how often does the agent get this right?Pass^4 — the percentage of tasks that passed across all 4 trials.

This measures along the reliability axis and the metric the τ²-bench leaderboard reports.It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent every time, not just on average.Figure 1.

Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using gpt-4.1 across four trials per task.Agent model: gpt-4.1 (user simulator: gpt-5.5).Table 2.Held-out agent performance using gpt-4.1 across four trials per task.Parentheses show percentage-point changes from baseline.

Domain (test tasks) Config Mean Accuracy Pass^4 Retail (40) baseline 78.1% 57.5% sup3 78.8% (+0.7) 60.0% (+2.5) lossless 82.5% (+4.4) 67.5% (+10.0) Airline (20) baseline 57.5% 35.0% sup3 60.0% (+2.5) 45.0% (+10.0) lossless 65.0% (+7.5) 40.0% (+5.0) Mean Accuracy climbs in both domains — retail +4.

4 pts (78.1 → 82.5) and airline +7.5 pts (57.5 → 65.0).Reliability moves further: the strict Pass^4 rises +10 pts in retail (57.5 → 67.5) and +10 pts in airline (35.0 → 45.0).

And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in Better performance costs more tokens, not more turns below.The best amount of guidance depends on the domain The two domains disagree about how much guidance pays off.

In retail, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the full lossless set for the real jump.

In airline the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact sup3 core (14 guidelines) is the more reliable one, taking Pass^4 to 45% against lossless's 40%.

Echoing the earlier finding that delivery is a dial, not a constant, support count is a knob worth tuning per domain, not a value to hard-code.

Better performance costs more tokens, not more turns Guidelines aren't free, but the cost lands where you'd expect — in prompt input tokens, not in a longer, more meandering conversation: Figure 2.Total LLM cost per conversation by domain and guideline configuration on the full base split.Table 3.

Total LLM cost per conversation (agent and user simulator) on the full base split across four trials.Lower is better.Domain Config Cost / conversation vs.baseline Retail baseline $0.084 — sup3 $0.095 +12% lossless $0.139 +65% Airline baseline $0.081 — sup3 $0.097 +19% lossless $0.

107 +32% Does the agent get there by talking longer?To answer that cleanly we compare like for like: the task runs solved under both baseline and the guideline variant — identical scenario and seed, a true "same task, done right" set — and count assistant turns on each side.Table 4.

Assistant turns per conversation for matched runs solved under both settings on the test split, pooling four trials.Domain Shared set Solved under both (n) Baseline turns +Guidelines Δ Retail baseline ∩ sup3 108 10.40 10.35 −0.05 baseline ∩ lossless 111 10.59 10.16 −0.

43 Airline baseline ∩ sup3 38 8.97 8.53 −0.45 baseline ∩ lossless 40 9.15 8.82 −0.33 The agent isn't talking longer — on the shared set, guidelines make conversations slightly shorter: lossless trims ~0.3–0.4 turns in both domains (retail 10.59 → 10.16, airline 9.15 → 8.

82), and no variant lengthens them.That matters beyond cost: each turn is a sequential LLM round-trip, so fewer turns means lower end-to-end latency — a faster time-to-completion for the customer, not just a cheaper one.

The only real price is on the input side — the larger system prompt riding on every LLM call: retail's 292-guideline prompt pushes cost from ~$0.084 to ~$0.139 per conversation, while airline's smaller playbook is cheaper.

And in production even that is largely reclaimable with prompt caching, since the guideline block is identical across every call.

So the trade is a favorable one: for a slightly higher input cost (much of it cached away in practice) the agent lands higher accuracy, higher reliability, and lower time-to-completion at once — not the usual quality-for-latency tradeoff, but all three moving the right way together.

Learned guidelines also improve agents using frontier models So far, we have used an older, non-reasoning model — gpt-4.1.A fair objection is that learned guidelines might have less to add to an agent using a frontier model: maybe they only patch the gaps of a weaker model.

So we tested it directly — we reran airline using gpt-5.5 for both the agent and guideline extraction, on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent's own runs.

One detail changes with the model: none of the guidelines extracted in this setup reached support ≥ 3 (they top out at support 2), so the high-support core here is sup2 rather than sup3.Table 5.Held-out airline performance for the gpt-5.5 agent across four trials per task.Airline · gpt-5.

5 (20 held-out test tasks) Accuracy Pass^4 baseline (0 guidelines) 72.5% 55.0% sup2 (103 guidelines) 80.0% (+7.5) 70.0% (+15.0) lossless (146 guidelines) 82.5% (+10.0) 70.0% (+15.0) The learned guidelines improve this setup too.

On the same held-out test tasks, appending them to the prompt lifts accuracy +10 pts (72.5 → 82.5) and the strict Pass^4 by +15 pts (55.0 → 70.0) — a larger reliability gain than we observed with gpt-4.1 on these very tasks (+10 pts), and from a much higher floor.

Learning guidelines around the model can improve an agent even when it starts from a stronger base.And where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly — the gain it buys is dependable, not a lucky draw.

ALTK-Evolve tops the leaderboard comparison Submissions are scored on the full base split (train ∪ test), as τ²-bench requires, and reported at Pass^4.Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers reported above.

This is where the practical upshot shows up — prompt-level memory carries agents using each base model into, or past, current frontier company: Retail: #1.On the base split the agent with learned guidelines reaches Pass^4 = 67.5%, ahead of every frontier model and ahead of RAFT-30B-A3B (61.

4%), a model fine-tuned on synthetic τ²-bench ret