The ShipsWith Index
Auth providers (Next.js)
Tiers, not ranks. Vendors whose intervals overlap are grouped, because pretending #4 beat #5 inside overlapping confidence intervals is fake precision — and refusing to fake precision is the point of this Index.
Ships cleanEvery agent passes ≥18 of 20 runs. No failing transcript on file.
FlakyA real failing transcript exists; the failure count is shown next to the tier.
FailsIntervals sit clearly below passing for at least one agent. Transcripts published.
Issue #1 covers the 12 vendors below: public-docs stages only (install, build, configure with placeholder keys — no vendor accounts), 20 runs per agent per vendor, published with every interval and every failing transcript under METH-1.2. We publish when the runs are done — never before, and never from a single run.
| Vendor cohort | Status |
|---|---|
| Clerk | queued · suites defined |
| Auth0 | queued · suites defined |
| Supabase Auth | queued · suites defined |
| Firebase Auth | queued · suites defined |
| WorkOS AuthKit | queued · suites defined |
| Stytch | queued · suites defined |
| Kinde | queued · suites defined |
| Better Auth | queued · suites defined |
| Auth.js (NextAuth) | queued · suites defined |
| Descope | queued · suites defined |
| PropelAuth | queued · suites defined |
| Frontegg | queued · suites defined |
How the Index stays honest
- No pay-to-pass. Placement is computed identically for customers and non-customers; customers are disclosure-marked. Paying buys private depth and speed, never a better public score.
- Right of reply. Every vendor gets the full transcripts 14 days before publication, plus one re-run on request.
- Everything pinned. Agent CLI versions and model IDs are published with each issue; task phrasings rotate per issue to resist teaching to the test.
- Public docs only. The public Index runs keyless stages — anything needing real vendor credentials stays in private audits.
Vendor on this list? Get ahead of the first issue — run your free report before the public cycle.