Webhook Durability & Replay

Every provider you take money through assumes you built a durable webhook ingestion pipeline, and none of them ship one — Shopify retries 8 times over 4 hours, then drops the event permanently, offers no replay API, and tells you in its own docs to build reconciliation jobs instead. This pack builds the layer underneath your handlers for Stripe, Shopify, Apple, Google Play and Resend: two distinct dedupe keys doing two different jobs, a per-resource ordering watermark, a real dead-letter, and backfill for the drops each provider documents — with the acceptance suite and a chaos simulator already written. Status: draft. There is no published receipt yet, so the only proof currently on offer is one you run yourself: verify.sh drops, duplicates, reorders and delays a multi-provider event history against a real Postgres, then recovers from a simulated three-day outage, and asserts the final state matches a clean-delivery oracle exactly.

Example use cases

• An indie SaaS on Stripe that has been double-crediting accounts and can't reproduce it. Stripe documents that it doesn't guarantee event order and that endpoints may receive the same event more than once. The pack's effect ledger makes the second delivery free and the reordered one harmless, and the chaos test is where you find out whether your version actually does that.
• A Shopify app developer who has been quietly losing orders. Shopify's retry budget is 4 hours and there is no replay API; developers have publicly reported missing orders/create webhooks and 20-minute-plus orders/paid delays. This pack ships the reconciliation job Shopify's own best-practices page tells you to write, keyed so it can never double-count an order you already received.
• A mobile team shipping on both stores. Apple App Store Server Notifications and Google Play RTDN have completely different failure shapes — Apple retries 5 times over ~3 days and never retries at all in the sandbox; Play arrives through Pub/Sub, unordered, carrying a signal rather than state. One pipeline, one ledger, one dead-letter for both, with the per-provider differences pinned in a table instead of discovered in production.
• A team that already bought one of the other packs in this catalogue. The billing, email and App Store packs each ship a correct handler for their own domain. This is the ingestion layer under all of them — see "How it composes" below, including what it deliberately does not do for you.
• Anyone who has ever written "TODO: make this idempotent" above a webhook route. That TODO is four separate mechanisms, and the pack's whole argument is that collapsing them into one is the bug.

Why this pack

No "always" claims — agents are stochastic, and every ingestion design trades something. Here is what the pack pre-solves, what it costs, and what you give up.

What a bare agent typically ships that this pack pre-solves — specific bug classes, not a claim that every agent hits every one:

• It writes a handler, not a pipeline. No dead-letter, so a poisoned event retries forever or vanishes into a log line. No replay, so recovery means a manual SQL session. No backfill, so a provider's documented permanent drop is simply data you never had. Shopify is the case that makes this non-theoretical: 8 retries, 4 hours, no replay API, and Shopify's own guidance is "your app shouldn't rely on receiving data from Shopify webhooks."
• It dedupes on the wrong key. This catalogue's own email-deliverability pack documents exactly this bug: Resend's data.email_id identifies the email, so it is shared across that message's delivered, bounced and complained events, and deduping on it silently collapses three distinct events into one. Resend's docs name the svix-id header as the dedupe key. Shopify ships the same trap in a different shape — X-Shopify-Event-Id is documented as "shared across all deliveries produced by the same merchant action", while X-Shopify-Webhook-Id is the per-delivery key. Picking the one with "event" in the name is the natural mistake.
• It uses one key for two jobs. Even a correct delivery key can't stop a double-application that arrives by a different path — a reconciliation job re-importing an order you already processed. The effect key has to be derived from the business fact, not the HTTP delivery, or backfill becomes the thing that corrupts your data.
• It wraps the whole transaction in an idempotency catch. The saas-billing-tax pack documents this trap on the billing side: a transaction-wide catch swallows an unrelated constraint violation and reports "processed" while the database diverges. In an ingestion pipeline it is strictly worse — burning the idempotency key means the retry is deduped away and the effect never happens at all. And in raw Postgres it needs a SAVEPOINT, without which the transaction is already aborted by the time you catch anything.
• It skips stale events uniformly. An ordering watermark is correct for snapshot events and catastrophic for accumulating ones: refuse a late invoice as "stale" and that payment is never counted, ever. The two need different treatment and the difference has to be declared, not inferred.

Scale envelope

Derived from the pinned stack (Next.js 16 route handlers, pg 8.22, Postgres 17.10, a SKIP LOCKED worker), not from a benchmark. Nothing here is load-tested; the acceptance suite proves correctness, not throughput, and it says so in ACCEPTANCE.md. Treat every figure as an order-of-magnitude estimate.

• Ingest is deliberately cheap: one HMAC or JWS verification plus one INSERT … ON CONFLICT DO NOTHING. On a small managed instance that is comfortably inside Shopify's 5-second total / 1-second connect budget — the tightest published response limit of the five providers, and the one the design targets. The realistic bound at this tier is the web service's request concurrency, not Postgres.
• Processing scales by adding worker processes, with no queue broker, no lease table and no Redis: FOR UPDATE SKIP LOCKED is the entire scheduler. Workers need no coordination because snapshot handlers are watermark-guarded and accumulate handlers are commutative — that is a property of the design, not a tuning parameter.
• The enrich step is the per-event cost that matters. Google Play notifications require a purchases.subscriptionsv2.get call before any key is derived, so Play throughput is bounded by that API, not by your database. The other four providers need no outbound call at all.
• First ceiling: table growth, not throughput. webhook_deliveries is append-only and the pack ships no retention policy on purpose — the right retention is "at least as long as your longest recovery lookback", which depends on which providers you use (Apple's history goes back 180 days, Stripe's 30, Pub/Sub's 7 by default). At roughly 1–2 KB of JSONB per row, a million deliveries a month is on the order of a gigabyte or two a month. Deciding your own delete from webhook_deliveries where status = 'done' and received_at < now() - interval '<N> days' is a one-line cron, and it is a decision the pack refuses to make for you rather than one it forgot.
• Second ceiling: a single hot resource. Effects on one resource_key serialize on that row's lock. Thousands of events per second spread across many resources is a different problem from thousands against one — the second is the one that needs a redesign, and the redesign is partitioning the resource, not replacing the pipeline.
• Upgrade path: more workers first (free, no code change); then a retention cron; then Postgres vertical scale; and only after all three does replacing the SKIP LOCKED loop with a broker become the right move. The worker loop is the only thing you'd replace — the keys, the ledger and the ordering rules are unaffected.

What it comfortably covers as shipped: a single product taking webhooks from one to five of these providers at the volume a small-to-mid SaaS, storefront or mobile app generates — thousands to low millions of events a month. What it isn't sized for out of the box: multi-tenant ingestion where each tenant has its own provider credentials (the config is per-app, not per-tenant), or a single resource under sustained high-frequency contention.

What it costs to skip the pack (modelled, not measured): building this layer the way most people actually work — one agent on a top-tier model, no cheap-tier routing, no parallel subagents, no prompt-caching discipline — models out at roughly $100–400 in API tokens, driven mostly by an exploration factor: five signature schemes and five sets of retry semantics are a lot of documentation to read badly. That range is this pack's own estimated build volume repriced at Opus-class rates, and it will be replaced with a measured figure when the receipt lands. The honest reading: buy this for the bug classes, not the token arithmetic. The bugs here are the silent, off-by-one, discovered-in-a-month kind.

Verified against

• claude-opus-5
• claude-sonnet-5
• claude-haiku-4-5

Supported stacks

• nextjs-postgres-worker

  node	24.18.1
  next	16.2.12
  react	19.2.8
  react-dom	19.2.8
  pg	8.22.0
  @types/pg	8.20.3
  jose	6.2.7
  @apple/app-store-server-library	3.1.0
  svix	1.99.1
  typescript	6.0.3
  vitest	4.1.10
  tailwindcss	4.3.3
  @tailwindcss/postcss	4.3.3
  @types/node	24.3.0
  @types/react	19.2.2
  @types/react-dom	19.2.2
  postgres	17.10
  stripe-api-version	2026-06-24.dahlia
  shopify-admin-api-version	2026-07
  apple-notifications-version	2.0
  play-rtdn-version	1.0

Estimated buyer token cost

1,750,000 tokens for one build pass (estimate, not a guarantee).