Design a Notification System

Requirements

Functional: send a notification to a user across multiple channels (push, email, SMS); respect per-user channel preferences; avoid spamming (rate/frequency caps). Non-functional: the system triggering a notification (e.g. "order shipped") should never be slowed down waiting for delivery; must handle each channel's own rate limits and outages independently.

The key idea: decouple triggering from sending

If the order service called the push-notification API directly and synchronously, a slow or down push provider would make checking out slower or fail orders entirely — for a completely unrelated concern. The fix is a queue in between: the triggering service just publishes an event and moves on.

Architecture

Why this matters for reliability

If the SMS provider has an outage, only SMS notifications back up in retry — push and email keep flowing normally, and (critically) the original triggering service was never blocked or affected at all, since it only ever talked to the queue.

Follow-up: how do you prevent duplicate notifications if a worker crashes mid-send?

Use an idempotency key (e.g. a hash of user_id + event_id + channel) so a retried delivery can be detected and skipped if it already succeeded — this is the same pattern used for idempotent payment retries.