Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

Back to Articles Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

Enterprise Article Published November 3, 2026 Upvote 6 Sami Marreed samimarreed Follow ibm-research Anupama Murthi anupamamurthi Follow ibm-research Vinod Muthusamy vinodmut Follow ibm-research Ankita Naik ankita-naik Follow ibm-research Segev Shlomov segevshlomov Follow ibm-research Jayaram Radhakrishnan jayaramkr Follow ibm-research Aminat A aadebiyi Follow ibm-research Ayhan Sebin ayhansebin Follow ibm-research Hamid Adebayo harmedox Follow ibm-research Erez Bilgory erezbi Follow ibm-research Asaf Adi AsafAdi Follow ibm-research Jim Laredo laredoj Follow ibm-research Merve Unuvar mrvnvr Follow ibm-research There are nearly 200 agent harnesses to choose from.

We gave three of them the same open-weight model and 90 AppWorld tasks, and changing the harness moved task completion from 30% to 81%.For enterprise use, CUGA adds what this benchmark does not test: rules that the harness enforces and deployment inside your own boundary.

⭐ Star CUGA on GitHub 🛡️ Try the Policy Playground → Community lists such as awesome-agent-harness and best-of-Agent-Harnesses track nearly 200 of them, and they make one point clearly: the harness matters, not only the model.In one analysis that best-of-Agent-Harnesses cites, the same GLM-5.

2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.So the right question is not "which harness is best?" but "which harness fits my workload, with my model?

" Hermes from Nous Research is built for always-on personal assistants, and Deep Agents from LangChain for long, multi-step tasks such as research.Both are excellent at those jobs.Enterprise work is a different job.

Agents act on business systems, such as CRM and payments, and they must follow rules that hold whatever the model decides.

CUGA (ConfigUrable Generalist Agent) is IBM's open-source harness built for that job, with open-weight or hosted models, inside your own environment, including sovereign clouds and air-gapped deployments.To measure what the harness adds, we kept the model the same and changed only the harness.

A controlled experiment: same model, different harness We ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, such as email, payments, and file storage.

The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs, such as logging in, reading API docs, paging through results, and changing state across several apps.Every harness used the same open-weight model, GLM-5.

3-flash, on the same 90 tasks in 30 scenarios, with one attempt per task.All runs used the Harbor evaluation framework, and our setup is public in harbor-eval, our Harbor fork.

Harness How it ran CUGA CUGA's AppWorld configuration: AppWorld instructions, task decomposition, verification before execution, and automatic reflection Hermes Harbor's default Hermes setup, AppWorld tools over MCP Deep Agents AppWorld tools over MCP, plus LangChain's built-in middleware, configured for the 469 tools: tool selection, context summarization, a to-do list, retries, and call limits.

We also added two small fixes so that malformed GLM replies did not end runs The results AppWorld, 90 tasks in 30 scenarios, GLM-5.3-flash TGC: tasks passed SGC: scenarios with all 3 variants passed CUGA 81.1 (73/90) 66.7 (20/30) Hermes 46.7 (42/90) 26.7 (8/30) Deep Agents 30.0 (27/90) 6.

7 (2/30) Figure 1.AppWorld, 90 tasks in 30 scenarios, GLM-5.3-flash for every harness.TGC: tasks passed.SGC: scenarios where all three variants passed.How to read this.This is not a verdict on Hermes or Deep Agents.

Neither was built for multi-app API work like AppWorld, and that is the point: the harness has to fit the job.The most useful number is SGC.A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run.

Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents.Three limits apply.This is a 90-task sample on one benchmark, so the results may not generalize to other tasks.The setups were not identical (see the setup table above).

And the runs did not use policies, so they do not measure policy enforcement.Where the points come from In the 90-task comparison, the model was the same in every run, so the differences come from the harness.Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat.

The same loop scales from one CugaAgent with its own tools to a CugaSupervisor coordinating several agents, local or remote over A2A.

These are the CUGA features that ran in the AppWorld setup, grouped by what they do: 🧭 Plan the work Task decomposition: splits a task into steps, one app at a time Focused tools: each step sees the tools of its app and a few shared ones, not all 469 Tool shortlister: find_tools searches for any other tool a step needs 🐍 Act in code CodeAct: the model writes Python that calls the app APIs Sandbox: the code runs in a restricted environment, with a time limit 🔍 Check before acting Verification before execution: a check reviews each code block that changes data before it runs 🔁 Check after acting Automatic reflection: reviews the results and lets the agent correct its plan Pagination audit: flags a list call that returned a full page when the next page was never requested With these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.

Every step the harness owns is an enforcement point Each step that the harness controls, like those in the list above, is also a place where it can check a rule.Most teams write business rules, such as payment approval or keeping customer PII inside the organization, into the system prompt.

A prompt is an instruction, not enforcement.The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.CUGA policies move these rules out of the prompt and into the harness.

We describe this policy layer in our ACM CAIS 2026 paper, Governance by Construction for Generalist Agents (arXiv).

The policies act at four points in every run: Point in the run Policy type How it acts Example User intent Intent Guard Blocks or answers matching requests before the agent acts Block a request to delete all customer records Agent guidance Playbook Gives the agent a standard procedure Follow the refund procedure step by step Tool execution Tool Approval Hard stop: the run pauses for a person's approval and shows the proposed code Ask a person before a payment tool runs Tool execution Tool Guide with ToolGuard Hard check: guard code runs before the tool.

A blocked call returns a policy violation, and the tool is never called Block a payment over $1,000 Final response Output Formatter Shapes the answer to an approved template Use the approved reply template Guidance steers the model, but approvals and ToolGuards are enforced by the runtime.

Put rules such as payment approval and PII handling in the second group, where they hold whatever the model decides.You can see the difference yourself in the Policy Playground, which sends the same message to two live agents, one without policies and one with them: Figure 2.

For "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections.The agent with policies held the payment for approval, then paid only the $10 that was due.Why pick CUGA Choose a harness as carefully as the model.

In our comparison, the same model completed between 30% and 81% of AppWorld tasks, depending on the harness.For enterprise work, CUGA gives you: Strong results, now with open-weight models.

With hosted models, CUGA held #1 on WebArena (February to September 2025) and on AppWorld (July 2025 to February 2026).Rules that the runtime enforces.Approvals and tool guards stop a call in the harness, whatever the model decides.Deployment inside your sovereign boundary.CUGA is Apache 2.

0, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in your private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles.IBM Sovereign Core builds on CUGA to run agents fully inside the customer's boundary.

Measurable cost.Budgets limit spending, and run receipts show tokens, tool calls, and time for every run.Configuration, not code.Set tools, budgets, and policies in the UI or SDK, test a draft, and publish a version.

When to pick something else CUGA is not the right harness for every job: For a terminal coding agent, use opencode, Codex, or Claude Code.For a personal, always-on assistant that lives in your chat apps, use OpenClaw or Hermes.For a blank-canvas graph framework, use LangGraph directly.

CUGA is built on it.CUGA is for automating business work across APIs, apps, and web UIs, under rules you can enforce.

Try CUGA 🛡️ Policy Playground · policies guide 🧪 Reproduce the comparison ⭐ Star CUGA on GitHub 🚀 Try CUGA: quick install The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not.

If you can configure Hermes or Deep Agents to do better, send a configuration to harbor-eval.We will rerun it on the same tasks and publish the results, whichever harness wins.

Spaces mentioned in this article 1 More from this author Ensemble forecasting with IBM Granite time-series models 28 October 22, 2026 ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model 27 October 20, 2026 Community EditPreview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images Comment · Sign up or log in to comment Upvote 6 Spaces mentioned in this article 1