Readiness scores check your files. We prove an agent can actually ship with you.
A perfect llms.txt and MCP checklist doesn't mean Claude Code can build with your product — quickstarts fail in agent hands in ways a single test shows green. ShipsWith runs real agents against your real docs in sandboxes, N times, grades the outcome deterministically (did the build pass?), and alerts you the day a model release changes the answer.
| Stage | Pass rate (95% CI) |
|---|---|
| discover | 80% (95% CI 38–96%) |
| install | 100% (95% CI 57–100%) |
| configure | 40% (95% CI 12–77%) |
| first-success | 20% (95% CI 4–62%) |
Four stages, deterministic pass/fail, honest sample sizes
Every task runs N times per agent in a fresh sandbox. Scores are pass rates with Wilson 95% intervals — if we can't distinguish signal from noise, we say so instead of alerting you.
Do agents choose you?
The agent gets the job — "add auth", "add email" — with no vendor named. We measure whether it finds and picks your product, or a competitor's.
Can they install you?
Real sandboxed runs of your quickstart: the right package, the right wiring, following your public docs exactly as an agent reads them.
Do they survive setup?
Keys, env files, middleware — the step where agents stall on dashboards and interactive logins. We show you the exact stall point.
Does the core task complete?
The project builds, the endpoint answers, the email sends. Deterministic assertions decide — never an LLM grading its own homework.
What we found when we ran this against the real market
Before asking anyone to pay, we ran the harness against real companies' real public docs — sandboxed, N runs each, deterministic verdicts. Three findings, every one carrying its sample size and date.
Installs mostly work — and we say so
Eight real dev-tool companies, install stage, N=3 each: every one passed clean. Well-documented 2026 SDKs install fine in agent hands. A tool that scares you with your install step is selling you noise.
install · 8 companies × N=3 · 2026-07-03Builds break intermittently
One auth vendor's own quickstart produced a real Next.js build failure in 1 of 3 runs — a missing Suspense boundary an agent hits only sometimes. A single-run test shows green. N-run sampling caught it.
first-success · N=3, CI 21–94% · 2026-07-03Agents don't pick commercial vendors
Asked to “add auth” freely 15 times, the agent chose an open-source library every single time. All ten commercial auth vendors tested — companies paying for AI-visibility tools today — were picked zero times.
discover · claude-code × N=15 · 2026-07-03Point-in-time measurements (Claude Code, Next.js App Router fixtures) — agent behavior changes with every model release, which is exactly why one-off scores go stale. Full method: METH-1.2.
Agent traffic is your newest browser — and it regresses monthly
Model releases change agent behavior the way browser releases used to change rendering. BrowserStack exists because environment churn never stops. The agent web has the same shape: your quickstart passed last month is not evidence it passes today.
Runs on every model release
Scheduled and release-triggered re-runs. An alert only fires when intervals stop overlapping — a real regression, not sampling noise.
Failure transcripts, verbatim
Not a score alone: the exact moment the agent hallucinated an API, installed a competitor, or gave up at your login wall.
Measured before/after
Prioritized fixes — llms.txt, AGENTS.md, quickstart restructure — then a re-run that shows the delta. You can move the number you pay to watch.
See where agents break on your product
We run real agents against your quickstart and send you stage-by-stage scores with failure transcripts. No dashboard to learn; the report is the product.