A tale by claude-code · run run_ckt_9f2a · 10 Jul 2026

Track down the checkout slowdown

succeeded

Checkout p95 latency had tripled to 2.9s since the v4.2 deploy. The cause was an N+1: each cart line fired its own address lookup, so a 40-item cart ran 41 queries. Batching the lookups into a single query cut checkout to 340ms and dropped queries per checkout from 41 to 2. No public API changed.

merged to main · 2026-07-10T09:41:00Z

agent claude-codedate 10 Jul 2026took 39m 0smodel claude-opus-4-8steps 34trigger sentry · alert: checkout p95 > 2s for 30m

pr#2184 Batch address lookups in checkoutfileapi/checkout/pricing.ts

143,000tokens

$2.14cost

2files changed

1 ✓tests

The tale

09:02

Confirmed the regression

Pulled 14 days of checkout p95 from Grafana. Latency was flat around 850ms until the morning of the 8th, then tripled and stayed there. The step lines up exactly with the v4.2 deploy — nothing else shipped that day.

citationGrafana · checkout latency dashboard#

“p95 request duration for POST /checkout rose from 0.85s to 2.90s on 2026-07-08.”

open source

09:11

Bisected to the deploy

Only checkout regressed — catalogue and cart p95 were untouched, so this was code, not the database or a noisy neighbour. Narrowed v4.2 to the three commits that touched the checkout path and ranked them by how much query work they could add.

Commit	Author	What it changed	Suspicion
a1c3f9	dependabot	bump stripe-node 14→15	low
7be220	m.okafor	per-line delivery estimates	high
4d0e11	j.tran	rename order fields	low

✓

Profile the checkout query path before reverting chosen

Reverting all of v4.2 would also drop the delivery-estimate feature product wanted. Cheaper to confirm the exact cause first, then fix only that.

Revert the whole deploy set aside

Blunt: it backs out three unrelated changes and we'd relive this the next time the feature ships.

09:24

Found the N+1

Commit 7be220 added a delivery estimate per cart line, and each estimate calls addressFor(lineId) — one query per line. Turned on query logging for a single 40-item checkout: 41 queries, 40 of them the same address lookup with a different id.

✓

Batch the 40 lookups into one query chosen

The line ids are all known up front, so a single `WHERE line_id = ANY($1)` returns every address in one round trip. No schema change, no cache to invalidate.

Add a per-address cache set aside

Solves the symptom, not the shape. A cache adds an invalidation bug surface for a query that batching removes outright.

$LOG_QUERIES=1 bun run replay fixtures/checkout-40-items.json | grep -c SELECT#

41

  SELECT * FROM addresses WHERE line_id = $1   -- ×40 (one per cart line)
  SELECT * FROM carts WHERE id = $1              -- ×1

exit 0

09:33

Batched the lookups

Replaced the per-line addressFor call with one addressesFor(lineIds) that fetches every address in a single query, then maps them back by line id in memory. The public checkout response is byte-for-byte identical — only the query count changed.

diff2 files, +18 −7#

api/checkout/pricing.ts+12−7api/checkout/addresses.ts+6−0

view full diff

09:39

Proved it on real traffic

Replayed the same 40-item checkout, then a 500-request sample captured from production. Query count per checkout dropped from 41 to 2, and p95 came back down to 340ms — below where it sat before the regression.

testcheckout replay · 500-request production samplepassed#

$ bun run replay fixtures/prod-sample-500.json --assert-p95=500ms

500 requests · p95 340ms · p99 410ms · 0 errors
assert p95 < 500ms: OK

$LOG_QUERIES=1 bun run replay fixtures/checkout-40-items.json | grep -c SELECT#

exit 0

What it read

Grafana14 days of checkout p95, deploy markers220ms
Sentrythe alert and the affected transaction
gitblame + the v4.2 commit range on the checkout path3 spans

Provenance

run withclaude-code 2.1 · AI-assisted
published bythe taleseal team — a hand-written example
reviewed byPriya Menon, staff engineer · Read the diff and the load-test output, re-ran the replay locally, merged. (2026-07-10)
full sessionraw transcript