Build Webhook Durability & Replay on nextjs-postgres-worker

Every provider you take money through assumes you built a durable webhook

ingestion pipeline, and none of them ship one — Shopify retries 8 times over 4

hours, then drops the event permanently, offers no replay API, and tells you in

its own docs to build reconciliation jobs instead. This pack builds the layer

underneath your handlers for Stripe, Shopify, Apple, Google Play and Resend: two

distinct dedupe keys doing two different jobs, a per-resource ordering

watermark, a real dead-letter, and backfill for the drops each provider

documents — with the acceptance suite and a chaos simulator already written.

**Status: draft. There is no published receipt yet**, so the only proof

currently on offer is one you run yourself: `verify.sh` drops, duplicates,

reorders and delays a multi-provider event history against a real Postgres, then

recovers from a simulated three-day outage, and asserts the final state matches

a clean-delivery oracle exactly.

Example use cases

• **An indie SaaS on Stripe that has been double-crediting accounts and can't

reproduce it.** Stripe documents that it doesn't guarantee event order and

that endpoints may receive the same event more than once. The pack's effect

ledger makes the second delivery free and the reordered one harmless, and the

chaos test is where you find out whether your version actually does that.

• **A Shopify app developer who has been quietly losing orders.** Shopify's

retry budget is 4 hours and there is no replay API; developers have publicly

reported missing `orders/create` webhooks and 20-minute-plus `orders/paid`

delays. This pack ships the reconciliation job Shopify's own best-practices

page tells you to write, keyed so it can never double-count an order you

already received.

• **A mobile team shipping on both stores.** Apple App Store Server

Notifications and Google Play RTDN have completely different failure shapes —

Apple retries 5 times over ~3 days and never retries at all in the sandbox;

Play arrives through Pub/Sub, unordered, carrying a signal rather than state.

One pipeline, one ledger, one dead-letter for both, with the per-provider

differences pinned in a table instead of discovered in production.

• **A team that already bought one of the other packs in this catalogue.** The

billing, email and App Store packs each ship a correct handler for their own

domain. This is the ingestion layer under all of them — see "How it composes"

below, including what it deliberately does not do for you.

• **Anyone who has ever written "TODO: make this idempotent" above a webhook

route.** That TODO is four separate mechanisms, and the pack's whole argument

is that collapsing them into one is the bug.

Scale envelope

Derived from the pinned stack (Next.js 16 route handlers, `pg` 8.22, Postgres

17.10, a `SKIP LOCKED` worker), not from a benchmark. **Nothing here is

load-tested**; the acceptance suite proves correctness, not throughput, and it

says so in ACCEPTANCE.md. Treat every figure as an order-of-magnitude estimate.

• **Ingest is deliberately cheap:** one HMAC or JWS verification plus one

`INSERT … ON CONFLICT DO NOTHING`. On a small managed instance that is

comfortably inside Shopify's 5-second total / 1-second connect budget — the

tightest published response limit of the five providers, and the one the

design targets. The realistic bound at this tier is the web service's request

concurrency, not Postgres.

• **Processing scales by adding worker processes**, with no queue broker, no

lease table and no Redis: `FOR UPDATE SKIP LOCKED` is the entire scheduler.

Workers need no coordination because snapshot handlers are watermark-guarded

and accumulate handlers are commutative — that is a property of the design,

not a tuning parameter.

• **The `enrich` step is the per-event cost that matters.** Google Play

notifications require a `purchases.subscriptionsv2.get` call before any key is

derived, so Play throughput is bounded by that API, not by your database. The

other four providers need no outbound call at all.

• **First ceiling: table growth, not throughput.** `webhook_deliveries` is

append-only and the pack ships **no retention policy on purpose** — the right

retention is "at least as long as your longest recovery lookback", which

depends on which providers you use (Apple's history goes back 180 days,

Stripe's 30, Pub/Sub's 7 by default). At roughly 1–2 KB of JSONB per row, a

million deliveries a month is on the order of a gigabyte or two a month.

Deciding your own `delete from webhook_deliveries where status = 'done' and

received_at < now() - interval '<N> days'` is a one-line cron, and it is a

decision the pack refuses to make for you rather than one it forgot.

• **Second ceiling: a single hot resource.** Effects on one `resource_key`

serialize on that row's lock. Thousands of events per second spread across

many resources is a different problem from thousands against one — the second

is the one that needs a redesign, and the redesign is partitioning the

resource, not replacing the pipeline.

• **Upgrade path:** more workers first (free, no code change); then a retention

cron; then Postgres vertical scale; and only after all three does replacing

the `SKIP LOCKED` loop with a broker become the right move. The worker loop is

the only thing you'd replace — the keys, the ledger and the ordering rules are

unaffected.

**What it comfortably covers as shipped:** a single product taking webhooks from

one to five of these providers at the volume a small-to-mid SaaS, storefront or

mobile app generates — thousands to low millions of events a month. **What it

isn't sized for out of the box:** multi-tenant ingestion where each tenant has

its own provider credentials (the config is per-app, not per-tenant), or a

single resource under sustained high-frequency contention.

**What it costs to skip the pack (modelled, not measured):** building this layer

the way most people actually work — one agent on a top-tier model, no cheap-tier

routing, no parallel subagents, no prompt-caching discipline — models out at

roughly **$100–400 in API tokens**, driven mostly by an exploration factor: five

signature schemes and five sets of retry semantics are a lot of documentation to

read badly. That range is this pack's own *estimated* build volume repriced at

Opus-class rates, and it will be replaced with a measured figure when the receipt

lands. The honest reading: buy this for the bug classes, not the token

arithmetic. The bugs here are the silent, off-by-one, discovered-in-a-month kind.

What you get

Features:

• `POST /api/webhooks/[provider]` — one route, five verifiers, a 200/401/404/500

contract that never lets an unhandled event type disable your endpoint

• Signature and authenticity for all five: Stripe HMAC with a 300-second

tolerance, Shopify HMAC, Apple JWS verified through

`@apple/app-store-server-library` (chain of trust **and** the App Store marker

OIDs), Google's push OIDC token checked for audience and service account,

Resend via Svix

• `webhook_deliveries` — an append-only delivery log keyed on

`(provider, delivery_id)`, the layer that makes a duplicate request free

• `webhook_effects` — the effect ledger keyed on the business fact, the layer

that makes a duplicate *change* impossible even from a different path

• `resource_state` — the projection plus its ordering watermark
• A `SKIP LOCKED` worker: N processes, no queue broker, no Redis, no lease table
• Fixed-table backoff, `WEBHOOK_MAX_ATTEMPTS` and a dead-letter status
• Three backfill sources (Stripe events, Apple notification history, Shopify

reconcile) built as request-builder/response-mapper pairs over an injected

`fetch`, so they are testable without a network

• `replayDead` — re-processing through the identical pipeline, never a second

code path

• A delivery simulator with seeded chaos and an outage mode, and 17 acceptance

checks against a real Postgres in Docker

• Signature-valid but unparseable payloads persisted rather than dropped

Screens/pages and CLIs:

• `/dev/webhooks` — read-only pipeline view: counts by provider and status, the

newest dead rows. Disabled in production

• `POST /api/webhooks/[provider]` — the ingest endpoint
• `npm run worker` — the processing loop
• `npm run webhooks:status` — counts by provider and status, oldest pending age
• `npm run webhooks:replay` — replay dead letters by provider or id
• `npm run webhooks:backfill` — run a provider's recovery source over a range

nextjs-postgres-worker versions lock