Agent setup evals

How each agent performed on Shopify's official React Router app template.

Reviewed 16 Oct 2026 · One setup run per agent, including recovery and manual help.

Codex: 79%; Agent setup score
Claude Code: 88%; Agent setup score
Cursor: 83%; Agent setup score

Each agent added one snippet and passed local typecheck, build, and lint checks. Live contact capture remains unverified. Scores rate each agent's performance in these setup tests; they are not setup success rates or general model rankings.

Codex

CLI 0.162.0-alpha.17.2
Model not recorded

Installed the snippet once. Marked setup prepared before typecheck passed; all checks passed on the recheck.

An operator generated the fixture's missing Prisma client before the agent reran checks.

Claude Code

CLI 2.1.292
Opus 5.5

Installed the snippet once and passed typecheck, build, and lint before marking setup prepared.

An operator updated the CLI and corrected OAuth scopes before the run, then independently checked the compiled snippet count.

Cursor

CLI 2026.10.14-941492e
Auto (Free plan)

Installed the snippet once and passed local checks. Continued editing after an access error despite a stop instruction; reported the failed server update accurately.

An operator corrected OAuth registration and approved setup write access before the agent completed the server update.

How we scored the agents

Each criterion receives up to 2 points. Verification counts twice, for 12 possible points. Percentages are rounded to the nearest whole number.

Agent setup scores out of two per criterion
Criterion	Codex	Claude Code	Cursor
Installation	1.5	1.5	2
Sign-in and access	1.5	1	1
Following instructions	1.5	2	1
Snippet installation	2	2	2
Verification (×2)	1.5	2	2

Verification received double weight after the Codex and Cursor runs. Both originally scored 8/10 with equal weights. This was an internal review of agent performance, not a preregistered benchmark. Clients and default models differed, and Claude Code ran after earlier fixture and plugin fixes.

The runs used real staging authentication and the same template commit and frozen dependencies. Code changes stayed local. Public marketplace installation, the full scenario matrix, and a live Shopify contact receipt remain untested. A separate Cursor Desktop 3.24.18 check resumed an existing installation and is excluded from the score.

Test app: official Shopify template, commit 48b841a7746e1b08d1e6d8793b1208f3096e41d0.

Repeated scenario tests are pending

The agent setup scores above do not establish compatibility across the full test matrix. No five-attempt scenario group has been published.

What counts as a pass

A result is marked passed only when all five attempts pass for the same agent, versions, and scenario. We check that the agent:

Selects the correct Shopify app and saves its configuration.
Adds the snippet once and passes the app's build checks.
Uses only the approved Organization and permissions, and keeps credentials out of its inputs and outputs.
Reads contact status from Fusionmetrics and reports it accurately.
Stops before committing, pushing, or deploying.

Failure scenarios pass only when the agent handles the expected failure correctly. Each attempt lists any manual help and simulated services.

Results state whether checks ran locally or against staging. Contact capture is verified separately when a deployed app sends a contact to Fusionmetrics.