Build Webhook Durability & Replay on nextjs-postgres-worker

Every provider you take money through assumes you built a durable webhook ingestion pipeline, and none of them ship one — Shopify retries 8 times over 4 hours, then drops the event permanently, offers no replay API, and tells you in its own docs to build reconciliation jobs instead. This pack builds the layer underneath your handlers for Stripe, Shopify, Apple, Google Play and Resend: two distinct dedupe keys doing two different jobs, a per-resource ordering watermark, a real dead-letter, and backfill for the drops each provider documents — with the acceptance suite and a chaos simulator already written. Status: draft. There is no published receipt yet, so the only proof currently on offer is one you run yourself: verify.sh drops, duplicates, reorders and delays a multi-provider event history against a real Postgres, then recovers from a simulated three-day outage, and asserts the final state matches a clean-delivery oracle exactly.

Example use cases

• An indie SaaS on Stripe that has been double-crediting accounts and can't reproduce it. Stripe documents that it doesn't guarantee event order and that endpoints may receive the same event more than once. The pack's effect ledger makes the second delivery free and the reordered one harmless, and the chaos test is where you find out whether your version actually does that.
• A Shopify app developer who has been quietly losing orders. Shopify's retry budget is 4 hours and there is no replay API; developers have publicly reported missing orders/create webhooks and 20-minute-plus orders/paid delays. This pack ships the reconciliation job Shopify's own best-practices page tells you to write, keyed so it can never double-count an order you already received.
• A mobile team shipping on both stores. Apple App Store Server Notifications and Google Play RTDN have completely different failure shapes — Apple retries 5 times over ~3 days and never retries at all in the sandbox; Play arrives through Pub/Sub, unordered, carrying a signal rather than state. One pipeline, one ledger, one dead-letter for both, with the per-provider differences pinned in a table instead of discovered in production.
• A team that already bought one of the other packs in this catalogue. The billing, email and App Store packs each ship a correct handler for their own domain. This is the ingestion layer under all of them — see "How it composes" below, including what it deliberately does not do for you.
• Anyone who has ever written "TODO: make this idempotent" above a webhook route. That TODO is four separate mechanisms, and the pack's whole argument is that collapsing them into one is the bug.

Scale envelope

Derived from the pinned stack (Next.js 16 route handlers, pg 8.22, Postgres 17.10, a SKIP LOCKED worker), not from a benchmark. Nothing here is load-tested; the acceptance suite proves correctness, not throughput, and it says so in ACCEPTANCE.md. Treat every figure as an order-of-magnitude estimate.

• Ingest is deliberately cheap: one HMAC or JWS verification plus one INSERT … ON CONFLICT DO NOTHING. On a small managed instance that is comfortably inside Shopify's 5-second total / 1-second connect budget — the tightest published response limit of the five providers, and the one the design targets. The realistic bound at this tier is the web service's request concurrency, not Postgres.
• Processing scales by adding worker processes, with no queue broker, no lease table and no Redis: FOR UPDATE SKIP LOCKED is the entire scheduler. Workers need no coordination because snapshot handlers are watermark-guarded and accumulate handlers are commutative — that is a property of the design, not a tuning parameter.
• The enrich step is the per-event cost that matters. Google Play notifications require a purchases.subscriptionsv2.get call before any key is derived, so Play throughput is bounded by that API, not by your database. The other four providers need no outbound call at all.
• First ceiling: table growth, not throughput. webhook_deliveries is append-only and the pack ships no retention policy on purpose — the right retention is "at least as long as your longest recovery lookback", which depends on which providers you use (Apple's history goes back 180 days, Stripe's 30, Pub/Sub's 7 by default). At roughly 1–2 KB of JSONB per row, a million deliveries a month is on the order of a gigabyte or two a month. Deciding your own delete from webhook_deliveries where status = 'done' and received_at < now() - interval '<N> days' is a one-line cron, and it is a decision the pack refuses to make for you rather than one it forgot.
• Second ceiling: a single hot resource. Effects on one resource_key serialize on that row's lock. Thousands of events per second spread across many resources is a different problem from thousands against one — the second is the one that needs a redesign, and the redesign is partitioning the resource, not replacing the pipeline.
• Upgrade path: more workers first (free, no code change); then a retention cron; then Postgres vertical scale; and only after all three does replacing the SKIP LOCKED loop with a broker become the right move. The worker loop is the only thing you'd replace — the keys, the ledger and the ordering rules are unaffected.

What it comfortably covers as shipped: a single product taking webhooks from one to five of these providers at the volume a small-to-mid SaaS, storefront or mobile app generates — thousands to low millions of events a month. What it isn't sized for out of the box: multi-tenant ingestion where each tenant has its own provider credentials (the config is per-app, not per-tenant), or a single resource under sustained high-frequency contention.

What it costs to skip the pack (modelled, not measured): building this layer the way most people actually work — one agent on a top-tier model, no cheap-tier routing, no parallel subagents, no prompt-caching discipline — models out at roughly $100–400 in API tokens, driven mostly by an exploration factor: five signature schemes and five sets of retry semantics are a lot of documentation to read badly. That range is this pack's own estimated build volume repriced at Opus-class rates, and it will be replaced with a measured figure when the receipt lands. The honest reading: buy this for the bug classes, not the token arithmetic. The bugs here are the silent, off-by-one, discovered-in-a-month kind.

What you get

Features:

• POST /api/webhooks/[provider] — one route, five verifiers, a 200/401/404/500 contract that never lets an unhandled event type disable your endpoint
• Signature and authenticity for all five: Stripe HMAC with a 300-second tolerance, Shopify HMAC, Apple JWS verified through @apple/app-store-server-library (chain of trust and the App Store marker OIDs), Google's push OIDC token checked for audience and service account, Resend via Svix
• webhook_deliveries — an append-only delivery log keyed on (provider, delivery_id), the layer that makes a duplicate request free
• webhook_effects — the effect ledger keyed on the business fact, the layer that makes a duplicate change impossible even from a different path
• resource_state — the projection plus its ordering watermark
• A SKIP LOCKED worker: N processes, no queue broker, no Redis, no lease table
• Fixed-table backoff, WEBHOOK_MAX_ATTEMPTS and a dead-letter status
• Three backfill sources (Stripe events, Apple notification history, Shopify reconcile) built as request-builder/response-mapper pairs over an injected fetch, so they are testable without a network
• replayDead — re-processing through the identical pipeline, never a second code path
• A delivery simulator with seeded chaos and an outage mode, and 17 acceptance checks against a real Postgres in Docker
• Signature-valid but unparseable payloads persisted rather than dropped

Screens/pages and CLIs:

• /dev/webhooks — read-only pipeline view: counts by provider and status, the newest dead rows. Disabled in production
• POST /api/webhooks/[provider] — the ingest endpoint
• npm run worker — the processing loop
• npm run webhooks:status — counts by provider and status, oldest pending age
• npm run webhooks:replay — replay dead letters by provider or id
• npm run webhooks:backfill — run a provider's recovery source over a range

nextjs-postgres-worker versions lock