System Design Interview - Notification Service

For a curated list of system design interview resources, check out our Helpful Resources for System Design Interviews page.

For a comprehensive list of resources for tech interviews, check out our Best Resources for Tech Interviews page.

1. Introduction

In today’s interconnected digital world, applications constantly need to communicate with their users. Whether it’s a social media mention, a shipping update, or a critical alert, notifications are the primary channel for proactive user engagement. A well-executed notification strategy can dramatically improve user retention and satisfaction, while a poorly designed one can lead to user churn and distrust.

Designing a system that can reliably deliver billions of such messages across multiple channels is a classic and revealing system design interview problem. It tests a candidate’s understanding of asynchronous workflows, distributed systems, scalability, and fault tolerance. This guide provides an expert-level deep dive into building such a system from the ground up.

2. What is a Notification Service?

A Notification Service is a dedicated backend system responsible for sending messages to users through various channels. Its core function is to abstract the immense complexity of communicating with different delivery providers and to manage the end-to-end lifecycle of a notification.

This abstraction is non-trivial. It involves handling:

• Platform-Specific Protocols: APNS (Apple) and FCM (Google) have unique connection and payload requirements.
• Token Management: Securely storing and refreshing millions of device tokens.
• HTML Rendering: Crafting and rendering complex HTML for emails.
• Vendor APIs: Integrating with dozens of SMS and email provider APIs, each with its own rate limits and error codes.

By centralizing this logic, the service empowers product developers to send notifications with a single API call, without needing to be experts in this complex domain.

3. Use Cases

Notification services are ubiquitous and support a wide range of use cases, including:

• Transactional Alerts: Critical, user-initiated updates like order confirmations, shipping statuses, password resets, and two-factor authentication codes. These demand the highest reliability and lowest latency.
• Social Engagement: Social-driven alerts such as new friend requests, likes, comments, and mentions. These are often high volume and require intelligent fan-out.
• Marketing & Promotional: Bulk messages sent to a large user base about new features, offers, or news. These are less time-sensitive but require high throughput and sophisticated targeting/throttling.
• System Alerts: Proactive monitoring alerts sent to users about system health, usage limits, or security events.
• Real-time Updates: Time-sensitive information like sports scores, stock price changes, or traffic alerts.

4. Requirements of the System

Functional Requirements

• Multi-Channel Delivery: Support for Push Notifications (iOS/Android), SMS, and Email.
• Fan-out Capability: A single event must be able to trigger notifications for millions of subscribers.
• User Preference Management: Users must be able to opt-in or opt-out of different notification categories.
• Template-Based Content: Support for pre-defined templates with personalization (e.g., “Hi {{name}}, your order has shipped!”).
• Throttling: Limit the number of notifications a user can receive in a given time window to prevent spamming.
• Analytics: Track delivery status (Sent, Delivered, Failed), open rates, and engagement for product analysis.
• A/B Testing: Allow product teams to test different notification texts or delivery times.

Non-Functional Requirements

• Extreme Reliability: Guarantee at-least-once delivery. No messages should be lost, especially transactional ones.
• High Scalability: The system must scale horizontally to handle growth in users and message volume (billions per day).
• Low Latency: Time-sensitive notifications should be delivered within seconds.
• Security: Protect user data and prevent the system from being used as a spam vector.
• Observability: Implement comprehensive logging, monitoring, and tracing to quickly diagnose issues in a complex distributed workflow.
• Cost-Effectiveness: Minimize costs by batching requests, using cheaper channels first, and optimizing vendor choices.

5. High-Level Design

The architecture is built around a decoupled, asynchronous model to handle high throughput and ensure reliability.

+----------------+   +----------------+   +--------------------+      +----------------+
| Client Service |-->|   API Gateway  |-->| Notification Service |--->|  Metadata DB   |
+----------------+   +----------------+   +--------------------+      |  (PostgreSQL)  |
                                                     |                +----------------+
                                                     |
                                                     v
                                     +------------------------------+
                                     |   Message Queue (Kafka)      |
                                     |------------------------------|
                                     | Topic: push | Topic: email   |
                                     +------------------------------+
                                           |          |
                  +------------------------+          +------------------------+
                  |                                                            |
                  v                                                            v
+-----------------------------------+            +-----------------------------------+
|      Push Worker Fleet            |            |      Email Worker Fleet           |
| (Consumes from `push` topic)      |            | (Consumes from `email` topic)     |
| 1. Fetch Token from Cache (Redis) |            | 1. Fetch Email from DB/Cache      |
| 2. Construct Payload              |            | 2. Render HTML Template           |
| 3. Call APNS/FCM                  |            | 3. Call SendGrid/SES API          |
| 4. Log Result to Analytics DB     |            | 4. Log Result to Analytics DB     |
+-----------------------------------+            +-----------------------------------+
                  |                                                            |
                  v                                                            v
        +----------------+                                           +----------------+
        |   APNS/FCM     |                                           | SendGrid/SES   |
        +----------------+                                           +----------------+

6. Detailed Design

API Design (OpenAPI 3.0 Spec)

The API must be idempotent. We enforce this with a X-Request-ID header.

openapi: 3.0.0
info:
  title: Notification Service API
  version: 1.0.0
paths:
  /v1/send:
    post:
      summary: Send a notification
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                recipient_id:
                  type: string
                channel:
                  type: string
                  enum: [PUSH, EMAIL, SMS]
                template_id:
                  type: string
                template_context:
                  type: object
      responses:
        '202':
          description: Accepted for processing.
      security:
        - bearerAuth: []
components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

Message Queue: Kafka vs. RabbitMQ

Worker Design & Logic

Workers are stateless and part of a consumer group for scalability and load balancing.

# Pseudocode for a generic worker
def main():
    consumer = kafka.Consumer("push_topic", group_id="push_workers")
    for message in consumer:
        try:
            process_notification(message)
            consumer.commit(message.offset) # Mark as processed
        except Exception as e:
            # Log error
            move_to_dlq(message)

def process_notification(message):
    # 1. Deserialize message
    notification_task = json.loads(message.value)

    # 2. Fetch data from cache/db
    user_profile = redis.get(f"user:{notification_task.recipient_id}")
    
    # 3. Apply business logic (throttling, user preferences)
    if not can_send(user_profile):
        return # Drop notification

    # 4. Call 3rd party gateway with retry logic
    send_with_retry(construct_payload(user_profile, notification_task))

    # 5. Log result to analytics DB (e.g., Cassandra)
    log_event(status="SENT", notification_id=notification_task.id)

7. Key Challenge: Reliability & Delivery Guarantees

Ensuring at-least-once delivery is paramount.

1. Persistence: The first step is the Notification Service persisting the task in a durable message queue like Kafka before returning a 202 Accepted response. The API call is synchronous, but the processing is asynchronous.
2. Retries with Exponential Backoff & Jitter: Workers must not hammer a failing 3rd party service. A retry policy should be implemented, for example: wait 1s, 2s, 4s, 8s… with a small random jitter to prevent thundering herds of retries.
3. Dead Letter Queue (DLQ): After 3-5 failed retries, the message is moved to a DLQ (another Kafka topic). This is critical. It prevents a single “poison pill” message (e.g., one with a malformed device token that always fails) from blocking the processing of all other valid messages in the partition. An alerting system must monitor the DLQ size.
4. Tracking Delivery Status: Many gateways (especially for email) provide asynchronous feedback via webhooks. We need a separate service to ingest these webhook events, correlate them with the original notification, and update the status in our analytics database.

8. Scaling & Optimizations

• Database Scaling:

  • PostgreSQL (Metadata): Use read replicas to offload traffic from analytics or services that only need to read preferences. For write scaling, consider vertical scaling or eventually moving to a distributed SQL DB like CockroachDB.
  • Cassandra (Logs/Analytics): Cassandra is built for horizontal scaling. Simply add more nodes to the cluster to increase write throughput. Use a sensible partition key (e.g., (user_id, month)) to ensure even data distribution. Implement Time-to-Live (TTL) on logs to automatically purge old data and manage storage costs.

• Caching Strategy (Cache-Aside Pattern):

  • A worker needing a user’s device token first checks Redis.
  • Cache Hit: The data is returned from Redis.
  • Cache Miss: The worker fetches the data from the source-of-truth DB (PostgreSQL), writes it to Redis with a TTL (e.g., 24 hours), and then proceeds. This ensures the cache stays reasonably fresh.

• Cost Optimization:

  • Batching: For non-urgent notifications, workers can buffer messages for a short period (e.g., 100ms) and send them to providers in a single batch API call, reducing network overhead and cost.
  • Intelligent Routing: Define a cost-based channel priority. Always try to send a Push notification first (virtually free). If it fails or the user has no device, fall back to Email, and finally to SMS (the most expensive).

9. Conclusion

Designing a notification service is a masterclass in building a modern, asynchronous, and resilient distributed system. It’s far more than a simple “send” button. A production-grade system requires a deep understanding of message queues to decouple components, robust retry and DLQ strategies to ensure reliability, and multi-layered caching and database scaling patterns to handle massive volume with low latency. By carefully considering the trade-offs between different technologies (like Kafka vs. RabbitMQ) and implementing strategies for cost and latency optimization, we can build a system that not only meets today’s demands but is also prepared for future growth.