Skip to content

← Selected work/09

Notification & Webhook Engine

At-least-once delivery engine — every guarantee boundary designed, implemented, then attacked with chaos failures until it held.

Period
Oct–Nov 2026
Role
System design & guarantee boundaries by hand; platform AI-assisted
Commits
134
Key metric
134 commits · dedup boundary · retry heap + DLQ · chaos-tested
Links
Repo ↗
System diagram
202 = persistedpg row before ackdlqexhausted → replayable01ingesthttp + persist02dedupredis nx · pg unique03channelbounded · 503 full04dispatchtoken bucket05deliverhmac sha256 signretry heap · backoff + jitter · durable scheduleat-least-once by design — duplicates acceptable, loss is not · chaos-tested: 500/429/slow/flap/drop

Problem

“Reliable delivery” is where most systems lie to themselves. Accept an event, lose it in memory, report success — that’s the default failure mode of every naive webhook dispatcher. The brief: build an engine where at-least-once is a system property, not a feature — and prove it by breaking it.

Architecture

An event is persisted to Postgres before the API returns 202 — the row existing IS the acknowledgement. Dedup is layered honestly: a Redis SET NX content marker answers duplicates with zero reads, while a unique constraint on a time-bucketed key holds the hard boundary — same content in the next window is a new event, the way Stripe-style idempotency actually works. A bounded channel applies backpressure (full → honest 503, the durable Pending row plus a sweeper covers the gap). Dispatch is rate-limited per endpoint (token bucket), payloads are HMAC-SHA256 signed per secret, and retries run on an in-memory min-heap with versioned lazy deletion over a durable PG schedule — backoff with equal jitter, exhausted events land in a queryable, replayable DLQ.

Key decisions

  • Postgres is the durability boundary, not Redis — Redis failing open means slowdown, not loss.
  • Bounded channels over a broker — RabbitMQ/Kafka would be a broker to operate for a lesson that’s about queues, not brokers.
  • No IRepository wrapper — DbContext used directly where hot paths need it; a repository abstraction hides exactly the failure modes this project studies.
  • Proof is a run, not a paragraph — MockReceiver serves chaos modes (500/429/slow/flap/drop) and a load generator fires real rps with mixed duplicates.
  • 50/50 build contract — the guarantee boundaries (dedup, retry, DLQ, backpressure) are hand-written and chaos-reviewed; the platform plumbing (SignalR, migrations, dashboard, compose) was AI-generated under an explicit AGENTS.md contract — a working example of the method on /ai.

What it proves

The final acceptance criterion: being able to answer where a message can duplicate and why that’s acceptable, where it can be lost and why that’s not. ~140 files — decisions journaled as ADR notes in PLAN.md, architecture and walkthroughs in docs/.