Backend systems · Design brief
Webhook delivery platform
Asynchronous webhook delivery with durable event storage, bounded retries, and delivery history.
Planned stack: TypeScript, NestJS, PostgreSQL, Docker, and AWS.
Requirements
Event acceptance must remain independent of downstream availability. Once the API acknowledges an event, the system needs a durable record of the event and its pending deliveries.
Delivery outcomes can be uncertain: a receiver may process a request before the sender receives confirmation. The design permits duplicate delivery and preserves the information needed for retries, investigation, and recovery.
Scope
- API-key authentication and endpoint management.
- Event submission with persistent delivery records.
- Asynchronous delivery, bounded retries, and manual replay.
- Delivery history and operational metrics.
Architecture
The API writes accepted events and pending deliveries to PostgreSQL. A dispatcher publishes committed work to a queue; workers execute outbound requests and persist attempt results.
The API and workers have separate deployment lifecycles. PostgreSQL remains the source of truth for event state and delivery history.
- APIEvent ingestion
- PostgreSQLEvent & delivery state
- Queue & workerDispatch & retry
- EndpointReceive webhook
Design decisions
Transactional acceptance
The API commits each event and its pending delivery records in one PostgreSQL transaction before acknowledging the request. A dispatcher publishes pending work to the queue.
Publishing a job and recording that publication are separate operations. A dispatcher restart can therefore produce duplicate queue messages.
Delivery semantics
A receiver may process a webhook before the worker records a successful response. Retrying can repeat an external side effect. A stable event ID allows receivers to deduplicate deliveries; exactly-once execution cannot be guaranteed across this boundary.
Retry policy & isolation
Outbound requests have deadlines. Retryable failures use capped exponential backoff; exhausted deliveries remain available for inspection and manual replay. Per-endpoint concurrency limits prevent one slow destination from consuming the shared worker pool.
Outbound request security
Webhook destinations are untrusted input. Destination validation and egress controls need to cover private address ranges, redirects, and DNS changes. Access controls apply to endpoint configuration and delivery history.
Implementation plan
API & persistence
Endpoint management, API-key authentication, event submission, and transactional delivery records. Integration tests cover validation, authorization, and database constraints.
Delivery processing
Queue dispatch, worker concurrency, retry scheduling, and delivery history. Failure tests cover duplicate messages and interrupted processing.
Deployment
Docker services on AWS, provisioned with Terraform. CI/CD includes migrations, health checks, and rollback; backup and restore procedures cover database recovery.
Operations & security
Structured logs, metrics, and distributed traces across ingestion and delivery. Load and fault tests cover resource saturation, dependency outages, and outbound request controls.
Acceptance criteria
- An acknowledged event remains recoverable after API, dispatcher, or worker failure.
- Interrupted dispatch can be retried without losing pending work.
- Duplicate messages do not corrupt delivery state or erase attempt history.
- Slow or unavailable endpoints do not block deliveries to other destinations.
- Every attempt can be traced to its event, with enough context to explain the result.
- Manual replay preserves the original event and prior delivery attempts.