How AI agents score on
real business software.

1. Highest success

   Claude Opus 5.5

   Success

   92.9%86.6–96.4%

   Per episode

   $17.79

   Per success

   $19.15lower bound

   Ran in Claude Code on all 113 tasks. In rtrvr with your own Anthropic key.

2. 4.9× cheaper per success

   GPT-6.1 Sol

   Success

   90.3%83.4–94.5%

   Per episode

   $3.54

   Per success

   $3.93

   Ran in Codex CLI on all 113 tasks.

3. 8.6× cheaper per success

   DeepSeek Flash

   Max reasoning

   Success

   62.4%55.9–68.4%

   Per episode

   $1.38

   Per success

   $2.22

   Ran in rtrvr's production agent on all 113 tasks. In rtrvr as a managed model or with your own DeepSeek key.

4. Lowest cost per success

   GLM 5.3 Flash

   High reasoning

   Success

   33.9%29.1–39.1%

   Per episode

   $0.58

   Per success

   $1.70

   Ran in rtrvr's production agent on all 113 tasks. In rtrvr as a managed model.

• Separates models, leaves headroom

  On the 6 rows that ran all 113 tasks, success spans 9.1% (GLM 5.3 Flash, low reasoning) to 92.9% (Claude Opus 5.5). The best open-weight model, DeepSeek Flash, max reasoning, trails Claude Opus 5.5 by 30.5 points: room to train into.

• Partial credit on every episode

  Rewards run from 0 to 1. Mean reward is 0.58 to 0.99 where success is 9.1% to 92.9%, so failed episodes still carry signal.

• Long horizons

  Successful episodes in rtrvr's agent take a median of 111 to 130 steps; 67 of 113 tasks are tier 4 (150–300 actions); reference solutions to the public tasks take 64 to 219 actions.

• Known rollout cost

  Per episode at list API prices: $0.58 to $1.38 for open-weight models and $3.54 to $17.79 for Anthropic and OpenAI models, measured over 1,356 episodes.

• Hard cases built in

  66 tasks inject production failures and 33 script a user correction. A scripted mid-task correction and tier-4 length lowered success for every model setting measured by kind of task.

Plug in your agent

harbor run -p ./rtrvr-sim/<task-id>-s0 -a claude-code -m <provider>/<model>
harbor run -p ./rtrvr-sim/<task-id>-s0 -a oracle   # reference solution

[[environment.mcp_servers]]
name = "rtrvr-sim"
transport = "streamable-http"
url = "http://sim:8000/mcp"

POST /reset {"task_id": "<task-id>", "seed": 0, "tool_profile": "sim-core-v1"}
  -> {episode_id, observation, tools, task}
POST /step  {"episode_id": "<id>", "action": {"tool_name": "click_element", "args": {"tab_id": 0, "element_id": 412}}}
  -> {observation, reward, done, info}

rtrvr-sim run --policy https://<endpoint>/v1 --model <model-id> --tasks <task-ids>

from trl import GRPOTrainer
from rtrvr_sim_trl import RtrvrSimEnv  # one typed method per tool

trainer = GRPOTrainer(
    model=model,
    train_dataset=tasks,          # rows carry task_id and seed
    reward_funcs=episode_reward,  # reads the verifier's reward at episode end
    environment_factory=RtrvrSimEnv,
)

Each task's verifier writes reward, success, weighted score, steps, invalid actions, no-op actions that left the page unchanged, gates passed and whether the agent answered.

12 unbranded replicas

4 of 12 environments were built from anonymized production page structure; 8 were authored from product documentation. Each aggregated environment lists how closely its replica matches the structure it was built from.

5 public tasks

5 of the 113 admitted tasks are public. The other 108 stay private, and the 86 held-out tasks run only as hosted evaluation. Of 113 admitted tasks, 54 were derived from anonymized production agent tasks, 8 were informed by anonymized usage statistics and 51 were authored from product documentation.

1. 01engineering manager

   Contractor access and team role audit with sign-off

   Length

   Tier 3 · 60–150 actions

   Apps

   Code Host, Groupware

   Reference solution

   64 actions

   Step budget

   96

   Provenance

   Authored from product documentation

   • Approval pause

   Summary, instruction and verifier checks

   An engineering manager audits outside collaborators and team roles on a code host against a contractor roster spreadsheet and a collaborator policy document. Contractors whose contract ended, or who aren't on the roster, lose all access except keep-list rows; active contractors lose access outside their contract scope and are lowered to their contract role without ever being raised, and a contract ending today still counts. Suspended people leave every team and only listed leads may stay team maintainers. Removing an Admin grant needs the requester's sign-off on the full list, and the requester holds one back until a vendor handover finishes.

   The instruction

   Please audit outside collaborators and team roles in Code Host against our Collaborator policy and the Contractor roster (Groupware › Files › Engineering › Vendors). Today is 2026-09-10. Ask me before you remove anything that has Admin. When you're done, list the access you removed as "repository: @login" (the repository's short name, without the organization) and tell me how many team membership changes you made.

   The verifier checks

   • At least one access entry or team role was changed
   • warehouse-loader: @aingersoll-ext lowered to Write (above the contract role)
   • mobile-app: @aingersoll-ext removed (outside the contract scope)
   • design-system: @astridthorvaldsen removed (contract ended)
   • developer-docs: @astridthorvaldsen removed (contract ended)
   • developer-docs: @estanhope-ext removed (outside the contract scope)
   • admin-console: @fdunmore-ext removed (contract ended)
   • developer-docs: @fokafor-ext removed (contract ended)
   • warehouse-loader: @pkowalczyk-ext lowered to Read (above the contract role)
   • developer-docs: @pkowalczyk-ext removed (outside the contract scope)
   • mobile-app: @rkowalczyk-ext removed (outside the contract scope)
   • developer-docs: @rkowalczyk-ext lowered to Write (above the contract role)

   Watch the reference run

2. 02RevOps analyst

   Turn call recaps into next steps and follow-ups

   Length

   Tier 4 · 150–300 actions

   Apps

   CRM, Groupware

   Reference solution

   171 actions

   Step budget

   360

   Provenance

   Authored from product documentation

   • Server error (500) on the first save
   • Confirm dialog before submitting
   • Required field in a collapsed section
   • 1 scripted correction

   Summary, instruction and verifier checks

   A sales manager's team emails a recap after every customer call. The RevOps analyst applies each recap of an open deal currently owned by the team: the deal's next step becomes the recap's text and one follow-up task is created on the deal with a standard subject, the recap's follow-up date (a date the rep corrects in a later reply wins, and a missing date means a week after the call), a priority that depends on the deal stage, and the deal owner as assignee. Recaps for closed deals, deal names that are not in the CRM (even when a similarly named deal exists) and deals owned by another team (including one reassigned since the call) are skipped and counted by reason, and a forwarded copy of a recap must not create a second task. The first task save fails with a server error, every task save asks for confirmation and the next-step field starts collapsed. Partway through, the manager announces that one rep is going on leave and that follow-ups for that rep's deals go to the manager instead, including any already created.

   The instruction

   Kasimir Farrow's Central team sent their call recaps to my inbox in Groupware. Please process every call recap there (whoever sent it) in the CRM as described in "Call recap follow-up standard" (Groupware, Files > Sales Operations > Playbooks). Answer with how many deals you updated, how many follow-up tasks you created, and how many recaps you skipped under each code (CLOSED, NOT_FOUND, OTHER_TEAM; give all three, 0 when none).

   The verifier checks

   • At least one follow-up task was created
   • Ravensloe Hospitality – Reporting suite FY26: Next step "Rollout plan walkthrough"
   • Ravensloe Hospitality – Reporting suite FY26: one open follow-up task "Follow up: Rollout plan walkthrough" due 2026-06-03, normal, owner 4
   • Juniperra Veterinary Clinic – Data migration FY26: Next step "Implementation timeline review"
   • Juniperra Veterinary Clinic – Data migration FY26: one open follow-up task "Follow up: Implementation timeline review" due 2026-06-07, high, owner 12
   • Caldovine Realty – Training package FY26: Next step "Second demo for the warehouse leads"
   • Caldovine Realty – Training package FY26: one open follow-up task "Follow up: Second demo for the warehouse leads" due 2026-06-04, normal, owner 4
   • Nevrassa Consulting – Security bundle FY26: Next step "Usage forecast for the second year"
   • Nevrassa Consulting – Security bundle FY26: one open follow-up task "Follow up: Usage forecast for the second year" due 2026-06-12, normal, owner 10
   • Marrowby Realty – Partner portal FY26: Next step "Procurement intake form submitted"
   • Marrowby Realty – Partner portal FY26: one open follow-up task "Follow up: Procurement intake form submitted" due 2026-06-21 (date corrected in a reply), high, owner 4
   • Carrowmere Pharmacy – Mobile access FY26: Next step "Security questionnaire back from IT"

   Watch the reference run

3. 03cloud security administrator

   Least-privilege migration wave with blocked rows

   Length

   Tier 4 · 150–300 actions

   Apps

   Access Admin, Groupware

   Reference solution

   196 actions

   Step budget

   294

   Provenance

   Authored from product documentation

   • Includes an infeasible item

   Summary, instruction and verifier checks

   A cloud security administrator applies a least-privilege migration plan across four non-production projects in a cloud access console: for each plan row, replace the principal's Editor role with exactly the listed roles in one change (keeping other roles, never granting a role twice), resolve rows that name permissions to the single smallest role that covers them, and record each row's result in the plan sheet. Three rows can't be applied (no single role covers the permissions, a listed custom role is disabled, the Editor role is only inherited from a folder): the administrator must leave them untouched, record a reason code and report the wave as partial.

   The instruction

   Please run wave 3 of our least-privilege migration in Access Admin today (Tuesday, September 8, 2026). The rows are in the "Least-privilege migration plan - wave 3" sheet in Groupware (Files, Platform › Least privilege), and the Least-privilege migration standard in the same folder says how each row is applied, which rows can't be applied, and the reason codes to use. Fill in the plan's Result column for every row as you go. When you're done, answer with status "partial" if any row could not be applied (otherwise "completed"), how many rows you applied and, for each reason code (NO_COVERING_ROLE, ROLE_DISABLED, INHERITED_ONLY), the emails of the principals whose rows were blocked with it.

   The verifier checks

   • The answer status is "partial" (some rows could not be applied)
   • CHG-2597: vikram.sandoval@pellmore-hospitality.example.test on partner-api-staging-7415 has exactly database-client, secrets-reader (Editor replaced, other roles kept)
   • CHG-2597: the Editor removal carries the change ID as its reason
   • CHG-2597: the plan's Result says Done (a short note after it is fine)
   • CHG-2598: freya.adeyemi@pellmore-hospitality.example.test on partner-api-staging-7415 has exactly artifacts-writer, functions-developer (Editor replaced, other roles kept)
   • CHG-2598: the Editor removal carries the change ID as its reason
   • CHG-2598: the plan's Result says Done (a short note after it is fine)
   • CHG-2599: default-runtime@partner-api-staging-7415.svc.example.test on partner-api-staging-7415 has exactly functions-developer, logging-viewer (Editor replaced, other roles kept)
   • CHG-2599: the Editor removal carries the change ID as its reason
   • CHG-2599: the plan's Result says Done (a short note after it is fine)
   • CHG-2600: freya.adeyemi@pellmore-hospitality.example.test on partner-api-dev-7238 has exactly keys-user, secrets-viewer (Editor replaced, other roles kept)
   • CHG-2600: the Editor removal carries the change ID as its reason

   Watch the reference run

4. 04payments operations analyst

   Work the support refund queue under the refund policy

   Length

   Tier 4 · 150–300 actions

   Apps

   Payments, Support Desk, Groupware

   Reference solution

   219 actions

   Step budget

   330

   Provenance

   Authored from product documentation

   • Server error (500) on the first save
   • Required field in a collapsed section

   Summary, instruction and verifier checks

   A payments operations analyst works the refund requests that the support desk routes to finance. For each ticket the analyst finds the store payment in the payments dashboard, applies the written refund policy in order (bank disputes, requests already refunded or raised twice, an approval limit that sends large refunds to the controller, double checkouts, damaged deliveries within 60 days, change-of-mind returns within 30 days with a restocking fee, and late deliveries that earn the shipping fee back) and refunds exactly what the policy allows with a ticket-number note. The first refund save fails with a server error and must be retried without refunding twice. Each ticket then gets an outcome tag, the refunded amount, a reply quoting the refund reference and the right status.

   The instruction

   Support Desk has routed this week's refund requests to finance (Billing group tickets tagged refund_request). Work every unsolved one under the "Customer Refund Policy" in Groupware Files (Finance > Policies): decide each request in Payments, refund there exactly what the policy allows, then tag, fill in and answer the ticket in Support Desk as the policy says. Create at most one refund per payment (a failed save must not end up refunded twice), and don't change anything else in either app. When you're done, answer with the IDs of the refunds you created, the total amount refunded and the numbers of the tickets you escalated.

   The verifier checks

   • At least half of the refund-request tickets carry an outcome tag
   • #4323 (Wrenhollow Mechanical): tagged refund_denied (and no other outcome), product area Hardware & orders, order ORD-41458, refund amount empty
   • #4323 (Wrenhollow Mechanical): public reply
   • #4323 (Wrenhollow Mechanical): Solved
   • #4324 (Sabrevel Tree Care): PAY-WLL6GVE3 refunded once, $118.00, reason Customer request, note "Ticket #4324"
   • #4324 (Sabrevel Tree Care): tagged refund_full (and no other outcome), product area Hardware & orders, order ORD-41461, refund amount $118.00
   • #4324 (Sabrevel Tree Care): public reply with the refund ID
   • #4324 (Sabrevel Tree Care): Solved
   • #4325 (Norvessa Garage Doors): PAY-PSM6YWKE refunded once, $186.15, reason Customer request, note "Ticket #4325"
   • #4325 (Norvessa Garage Doors): tagged refund_partial (and no other outcome), product area Hardware & orders, order ORD-41464, refund amount $186.15
   • #4325 (Norvessa Garage Doors): public reply with the refund ID
   • #4325 (Norvessa Garage Doors): Solved

   Watch the reference run

5. 05operations analyst

   Stand up a customer onboarding tracker and load the hand-off data

   Length

   Tier 3 · 60–150 actions

   Apps

   Tables

   Reference solution

   151 actions

   Step budget

   227

   Provenance

   Authored from product documentation

   Summary, instruction and verifier checks

   An operations analyst sets up a new page with a customer onboarding database from a written spec: the starter database is renamed and reshaped (renamed title and long-text properties, the starter stage options renamed and extended), eight properties are added, and five views are built: a renamed sorted table, an any-of filter with a nested all-of group, a two-level grouped table, a board with chosen card properties, and an intake form with custom labels, a required question, a custom button and a confirmation message. The pasted sales hand-off list must be cleaned on the way in (owner initials mapped to full names, coded plan, phase and health values, year/month/day dates the importer cannot read, a test account and a customer entered twice), and finance's ARR corrections must update only the matching onboardings without adding the customer that is not onboarding.

   The instruction

   We're moving customer onboarding out of the sales hand-off spreadsheet into Tables. Please create a new page called "Customer Onboarding" with a database on it (start from the starter database) and shape that database into our tracker: DATABASE "Onboardings" (rename the starter database; keep its properties, renamed): - Account: the title property (the starter's Name), required. - Blockers: the starter's Details (Long text). - Stage: keep the starter's Stage property but its options must be, in order: Kickoff, Data migration, Training, Live, Stalled. - CSM: Select with options, in this order: Ximena Farrow, Keziah Kavanagh, Dario Alvarado. - Plan: Select, required, options in this order: Starter, Growth, Enterprise. - Kickoff date and Go-live target: Date properties. - ARR: Currency in whole dollars. - Seats: Number (whole numbers). - Health: Select with options, in this order: Green, Yellow, Red. - Sponsor email: Email. Nothing else (we don't keep the CRM reference). VIEWS: - "All onboardings": the default Table view, renamed, sorted by Go-live target, earliest first. - "At risk": a Table view showing onboardings that match ANY of: Health is Red; Stage is Stalled; Health is Yellow and the Go-live target is before today (both of those together). - "By CSM": a Table view grouped by CSM and then by Stage, sorted by ARR, highest first. - "Onboarding board": a Board with a column per Stage; cards show CSM, Health and Go-live target only. - "Onboarding intake": a form whose questions are, in this order: Account labelled "Customer name"; Plan; Seats; Sponsor email labelled "Executive sponsor's email" and required on the form. Button text "Start onboarding"; confirmation message "Thanks! Your CSM will reach out within one business day." DATA: here's the sales hand-off export. Owners are written as initial + surname (they're our CSMs above), plan tiers are starter / growth / ent, phase "on hold" means Stalled, RAG is G / Y / R for Green / Yellow / Red, and the dates are year/month/day. Leave out test accounts. One customer was entered twice by sales; the later row is the correct one. Don't bring in the CRM ref column. Customer,Owner,Plan tier,Phase,Kickoff,Target go-live,Annual value,Licenses,RAG,Sponsor,CRM ref,Blockers Torvalen Consulting,K. Kavanagh,ent,kickoff,2026/08/25,2026/09/05,"$154,000",625,G,teodor.pennington@torvalen-consulting.example.test,CRM-83123,SSO setup waiting on customer IT Ondrevia Insurance Agency,X. Farrow,starter,training,2026/07/20,2026/09/09,"$81,000",820,Y,gaia.thorvaldsen@ondrevia-insurance-agency.example.test,CRM-36795,Sponsor on leave until next month Serravin Motors,K. Kavanagh,ent,live,2026/08/28,2026/11/07,"$18,000",728,G,mireille.garside@serravin-motors.example.test,CRM-94087,Needs custom SLA review Dunvessa Veterinary Clinic,D. Alvarado,starter,live,2026/08/25,2026/12/02,"$147,000",338,G,kasimir.lindqvist@dunvessa-veterinary-clinic.example.test,CRM-58064,SSO setup waiting on customer IT Serravin Consulting,X. Farrow,growth,kickoff,2026/08/15,2026/10/17,"$51,000",638,G,leilani.montoya@serravin-consulting.example.test,CRM-26577,Needs custom SLA review Torvalen Consulting,K. Kavanagh,ent,live,2026/08/25,2026/09/26,"$164,000",625,G,teodor.pennington@torvalen-consulting.example.test,CRM-83123,SSO setup waiting on customer IT Westerlyn Freight,D. Alvarado,ent,kickoff,2026/08/24,2026/12/04,"$133,000",776,Y,priya.stanhope@westerlyn-freight.example.test,CRM-26750,Sponsor on leave until next month Graymantle Outfitters,X. Farrow,ent,live,2026/08/26,2026/10/29,"$80,000",162,G,jordan.pellegrini@graymantle-outfitters.example.test,CRM-46963, TEST - Wrenhollow sandbox,X. Farrow,starter,kickoff,2026/07/20,2026/10/24,"$1,000",820,Y,qa@example.test,CRM-00000, Elmstrand Realty,K. Kavanagh,ent,migration,2026/09/03,2026/10/11,"$103,000",742,G,casey.esposito@elmstrand-realty.example.test,CRM-70690,Needs custom SLA review Meridane Engineering,D. Alvarado,starter,on hold,2026/07/19,2026/09/04,"$169,000",237,G,wren.brightwell@meridane-engineering.example.test,CRM-47729, Norvessa Academy,X. Farrow,ent,training,2026/09/01,2026/11/19,"$235,000",302,R,yara.barrientos@norvessa-academy.example.test,CRM-21272,Legacy data export is late Thistlecombe Energy,K. Kavanagh,starter,live,2026/08/05,2026/10/09,"$201,000",393,G,emil.ferreira@thistlecombe-energy.example.test,CRM-83843, Nevrassa Robotics,D. Alvarado,growth,training,2026/08/27,2026/12/13,"$232,000",816,G,farida.macready@nevrassa-robotics.example.test,CRM-27412,SSO setup waiting on customer IT Ostrevel Apparel,X. Farrow,starter,migration,2026/09/10,2026/11/01,"$58,000",39,R,esperanza.jablonski@ostrevel-apparel.example.test,CRM-67152, Then apply finance's ARR corrections below to the matching onboardings. Finance's list covers all customers, so ignore anyone who isn't in the tracker; don't add them. Customer,Corrected ARR Halvessen Media,"$117,000" Dunvessa Veterinary Clinic,"$141,000" Nevrassa Robotics,"$240,000" Graymantle Outfitters,"$82,000" When you're done, tell me how many onboardings are in the tracker, which customers from either list you deliberately didn't add, and how many onboardings the At risk view shows.

   The verifier checks

   • Exactly one "Customer Onboarding" page with an "Onboardings" database
   • Hand-off rows were brought in
   • Account is the title property (renamed from Name), required
   • Details is renamed Blockers (Long text)
   • Stage options are Kickoff, Data migration, Training, Live, Stalled
   • CSM: Select with the three CSMs' full names
   • Plan: required Select with Starter, Growth, Enterprise
   • Kickoff date is a Date property
   • Go-live target is a Date property
   • ARR: Currency $ in whole dollars
   • Seats is a Number in whole numbers
   • Health: Select with Green, Yellow, Red

   Watch the reference run

1. 01

   Structure from anonymized production pages

   For 4 of 12 environments, the interface structure is aggregated from anonymized production page data: roles, layout, action hints and short generic labels that recur across many users. No page text, values or links are kept. The other 8 were authored from product documentation.

2. 02

   An unbranded, live app

   Each replica is a working web app with its own database, seeded with generated people, companies and records. Every tool call changes real application state.

3. 03

   rtrvr's production observation and tools

   Agents read each page as the accessibility-tree text rendered by the pinned rtrvr extension (build 528f571), and act with rtrvr's production act tools: 37 tools in profile sim-core-v1.

4. 04

   A deterministic verifier

   SQL, event-log and answer checks grade the final state, with no LLM judges. Faults, user corrections and approval pauses are scripted per task, so every run is repeatable.

[h1] Data Analyst
[textbox][editable][actions=click→type:focus-first] Name * [id=7802] [name=fullName] [required] [autocomplete=name]
[textbox][editable][actions=click→type:focus-first] Email * [id=7811] [type=email] [name=email] [required] [autocomplete=email]
[combobox][autocomplete=list] Location * [id=7829] [name=location] [placeholder="Start typing a city"]
[textNode] Resume *
[occluded][textbox][read-only][actions=click→type:focus-first][format=accepts .pdf,.doc,.docx] Upload file [id=7839] [type=file]
[group] Are you legally authorized to work in the United States?
  [below][button] Yes [id=7870]
  [below][button] No [id=7872]
[below][button] Next [id=7874]