Methodology
Every number ShipsWith publishes — in reports, leaderboards, or alerts — is produced by the process on this page. When the methodology changes, the version number changes, and every score is traceable to the version that produced it.
What a test is
A task gives an agent a realistic job against your product (for example: “Add authentication to this Next.js app following the docs”) inside a fresh, disposable sandbox seeded from a pinned fixture. The agent works alone: it reads your public docs, installs packages, writes code. Nothing is pre-installed for it and nothing is cleaned up after it — what it leaves behind is what gets judged.
Tasks are grouped into four stages, weighted by how much they matter to your funnel:
discover(weight 0.15) — the agent is told the job, never the vendor. Did it find and choose your product?install(0.25) — right package, right wiring, from your quickstart.configure(0.20) — env keys, middleware, setup steps, using documented test/placeholder values.first-success(0.40) — the core action completes: the build passes, the endpoint answers.
How pass/fail is decided
Deterministic assertions decide the verdict — nothing else. After the agent finishes, we check the sandbox mechanically: is the package in package.json? Does the integration code exist? Does npm run build exit 0? Does the endpoint respond? An LLM never grades the outcome. Transcript analysis (why the agent failed, where it got lost) is labelling on top of the verdict, never the verdict itself.
Sampling and scoring
Agents are non-deterministic, so single runs are meaningless — and we never publish them. Every task × agent pair runs N times (N=5 for free reports, N=3 for monitoring runs), each in its own sandbox. The score is the pass rate with a Wilson 95% confidence interval, which stays honest at small N: 3 passes out of 3 reports as 100% with an interval of 44–100%, not as certainty.
Suite scores are stage-weighted means of task pass rates. The minimum publishable sample is N=3; our tooling refuses to render a report below it.
Current compatibility, not lifetime average
The headline score answers “does the agent ship with your product now?” — so it is computed from your most recent run, not pooled over all history. If you fix a broken quickstart, the next run reflects the fix; a months-old failure does not haunt your score forever. Concretely: per agent we score the latest monitoring session, widening to the previous session only if the latest is too thin to publish (below N=3), and no further. The longitudinal trend is the opposite — it deliberately keeps every session, so you can see the whole history even as the headline number tracks the present.
Runs that don't count
Sometimes the agent runtime fails us — a usage limit, a provider outage, an API error. Those runs say nothing about your product, so they are excluded from every published N: an infrastructure failure is never scored as your failure. They stay in the append-only store, marked distinctly, and are detected deterministically from the agent CLI's own error output — never by judgment call. Ceiling exits (time, cost, turns) are different: an agent that gives up or loops on your docs is a real behavioral outcome, and those runs do count.
Regression alerts
A change is only called a regression (or an improvement) when the new interval and the baseline interval do not overlap at all. Overlapping intervals are reported as stable — sampling noise is not news, and we will not alert you on it.
What is pinned and recorded
- Agent configuration: vanilla. Agents run with default settings only — no operator-level config, custom instructions, or memory is loaded. A score must measure your product against the agent everyone else runs, not against our setup.
- Agent CLI name and exact version, per run
- Model ID as reported by the agent runtime, per run
- Sandbox kind and fixture, per run
- Token usage, metered cost, wall time, and turn count, per run
- The full transcript and every assertion result, per run
- Methodology version, per run
Runs are stored append-only. Scores are always recomputed from stored runs and never edited in place.
Ceilings
Every run has hard ceilings on wall time, metered cost, and agent turns. A run that hits a ceiling is recorded with that exit state; ceilings are part of the task definition and reported with results.
What we never do
- Publish a score from a single run.
- Let an LLM decide pass/fail.
- Fabricate, extrapolate, or backfill numbers.
- Alert on changes our own intervals cannot distinguish from noise.
- Publish results for agents we did not actually run — untested agents are listed as untested.
Agents under test
Current adapters: Claude Code (headless), OpenAI Codex CLI, and Gemini CLI — all official CLIs, run non-interactively under their providers' terms. Browser-based agents are on the roadmap. Reports always state which agents ran and which did not, with reasons.
Versioning
This is METH-1.2. Changes to weights, sampling, assertions semantics, or alerting rules bump the version, with the change and rationale logged. Reports link to the version they were produced under.
METH-1.2 (2026-07-04): agent-runtime infrastructure failures (usage limits, provider outages) are excluded from published N — they are not vendor failures. Vendor-success assertions accept any official SDK of the vendor, not one hard-coded package name. METH-1.1 (2026-07-04): the published score is computed from the current cohort — your most recent run — instead of pooled lifetime history, so a fixed integration recovers immediately. METH-1.0 (2026-07-03): initial methodology.