# Designing a Low‑Latency Multi‑Region Real‑Time Chat System

> Learn how to architect a real‑time chat service that scales to millions of users, stays under 100 ms latency worldwide, and handles offline messages gracefully.

- URL: https://ferdousazad.com/blog/designing-low-latency-multi-region-chat-system
- Author: Ferdous Azad (Founder @ DataXgen AI)
- Published: 2026-11-01
- Updated: 2026-11-01
- Tags: system design, real-time chat, scalability, distributed systems, low latency

## Problem statement and requirements  

We need to build a **real‑time chat platform** that powers a consumer‑facing mobile/web app. The service must feel instantaneous, survive network partitions, and keep costs predictable as the user base grows. Below are the functional and non‑functional requirements, together with realistic scale assumptions that guide design decisions.  

### Functional requirements  

| # | Requirement | Description |
|---|-------------|-------------|
| F1 | **One‑to‑one & group messaging** | Users can send messages to a single peer or to a group of up to 200 participants. |
| F2 | **Message ordering** | Within a conversation, messages must be displayed in the order they were sent, even across regions. |
| F3 | **Read receipts & typing indicators** | Low‑latency signals that a message was read or that a participant is typing. |
| F4 | **Offline support** | If a user is offline, messages are persisted and delivered when they reconnect, preserving order. |
| F5 | **Message history** | Users can scroll back to retrieve the last 30 days of chat history. |
| F6 | **Media attachments** | Images, short videos, and audio clips up to 10 MB each. |

### Non‑functional requirements  

| # | Requirement | Target |
|---|-------------|--------|
| N1 | **Latency** | 95th‑percentile end‑to‑end latency < 100 ms for text messages in any major region (NA, EU, APAC). |
| N2 | **Throughput** | 10 k messages / s sustained, with spikes up to 100 k messages / s during events. |
| N3 | **Availability** | 99.99 % uptime (≈ 4.3 min downtime per month). |
| N4 | **Scalability** | Horizontal scaling to support 50 M active users without major architecture changes. |
| N5 | **Consistency** | **Strong ordering** per conversation, **eventual consistency** across regions for read receipts and typing indicators. |
| N6 | **Cost** | Keep per‑user monthly cost < $0.05 for core messaging (excluding media storage). |

### Assumptions (scale numbers)  

* **Active users**: 5 M concurrent, 50 M total monthly.  
* **Average messages per active user**: 20 msg / day → ≈ 100 M messages / day.  
* **Peak concurrency**: 2 M users online during global events.  
* **Geography**: Three primary regions – US‑East, EU‑West, AP‑SouthEast – each with its own data center.  

These numbers are deliberately generous to stress‑test the design while staying realistic for a fast‑growing consumer app.

---

## High-level architecture  

```mermaid
graph TD
    subgraph Edge[Edge Layer]
        LB[Global Load Balancer] -->|TLS| API[API Gateway]
        WS[WebSocket Frontend] -->|WS| API
    end

    subgraph Region[Region (e.g., US‑East)]
        API -->|REST| Auth[Auth Service]
        API -->|WS| ConnMgr[Connection Manager]
        ConnMgr -->|Pub/Sub| MsgBroker[Kafka Cluster]
        MsgBroker -->|Topic: chat.{conv_id}| MsgProc[Message Processor]
        MsgProc -->|Write| ChatStore[Sharded Cassandra]
        MsgProc -->|Write| PresenceStore[Redis Cluster]
        MsgProc -->|Emit| Notif[Push Notification Service]
        MediaSrv[Media Service] -->|S3| ObjectStore[Object Storage]
    end

    subgraph Global[Cross‑Region Sync]
        MsgBroker -- Replicate --> MsgBroker_EU
        MsgBroker -- Replicate --> MsgBroker_AP
        PresenceStore -- Sync --> PresenceStore_EU
        PresenceStore -- Sync --> PresenceStore_AP
    end

    subgraph Client[Client]
        MobileApp -->|WS| WS
        WebApp -->|WS| WS
    end
```

**Key flow**  

1. **Authentication** – JWT issued by Auth Service, verified at the edge.  
2. **WebSocket connection** – Handled by Connection Manager; each client gets a logical session ID.  
3. **Message publish** – Client sends a JSON payload over WS; Connection Manager forwards to Kafka topic `chat.{conversation_id}`.  
4. **Processing** – Message Processor consumes, assigns a **Lamport timestamp**, writes to Cassandra, updates read‑receipt state in Redis, and pushes to the recipient’s active connections.  
5. **Cross‑region** – Kafka’s MirrorMaker replicates topics to other regions; read‑receipt updates are eventually synced via Redis CRDTs.  
6. **Media** – Attachments are uploaded via a pre‑signed URL to Object Storage; message payload contains a reference URL.

---

## Data model  

### 1. Message schema (Cassandra table)  

```cql
CREATE TABLE chat.messages (
    conversation_id uuid,
    partition_key timeuuid,          // Lamport timestamp + UUID for ordering
    message_id uuid,
    sender_id uuid,
    payload text,                    // JSON with text or media reference
    created_at timestamp,
    edited_at timestamp,
    is_deleted boolean,
    PRIMARY KEY (conversation_id, partition_key)
) WITH CLUSTERING ORDER BY (partition_key ASC);
```

* **partition_key** combines a Lamport timestamp (ensuring total order per conversation) with a UUID to avoid collisions.  
* **Time‑series partitioning**: each conversation is a partition; Cassandra’s wide‑row model handles millions of rows efficiently.  

### 2. Presence & read receipts (Redis)  

| Key pattern | Value | TTL |
|-------------|-------|-----|
| `online:{user_id}` | `1` | 30 s (auto‑expire) |
| `typing:{conv_id}:{user_id}` | `timestamp` | 5 s |
| `read:{conv_id}:{user_id}` | `message_id` | ∞ |

Redis stores the latest read pointer per user per conversation. Updates are **idempotent** and replicated across regions using **Redis‑CRDT** for eventual consistency.

### 3. Media metadata (SQL)  

A lightweight relational table (PostgreSQL) tracks media URLs, size, MIME type, and expiration for cleanup jobs.

```sql
CREATE TABLE media (
    media_id UUID PRIMARY KEY,
    conversation_id UUID,
    uploader_id UUID,
    object_key TEXT,
    mime_type TEXT,
    size_bytes BIGINT,
    created_at TIMESTAMP WITH TIME ZONE
);
```

---

## Key components  

| Component | Responsibility | Technology choice | Rationale |
|-----------|----------------|-------------------|-----------|
| **API Gateway** | TLS termination, routing, rate limiting | Envoy + AWS ALB | Layer‑7 routing, easy canary deployments |
| **Auth Service** | JWT issuance, token revocation | Go + OIDC provider (Keycloak) | Low latency, mature ecosystem |
| **Connection Manager** | WebSocket lifecycle, heartbeat, back‑pressure | Node.js (ws) + NGINX | Handles many concurrent sockets, non‑blocking I/O |
| **Message Broker** | Durable ordering, fan‑out to processors | Apache Kafka (3‑zone replication) | Strong ordering per topic, high throughput |
| **Message Processor** | Validation, enrichment, persistence | Java (Spring Boot) + Kafka Streams | Exactly‑once processing with transactional writes |
| **Chat Store** | Long‑term storage, range queries | Apache Cassandra (5‑node cluster per region) | Linear scalability, tunable consistency |
| **Presence Store** | Real‑time presence, typing, read receipts | Redis Cluster + CRDT extension | Sub‑millisecond reads/writes, cross‑region sync |
| **Push Notification Service** | APNs / FCM integration | Go microservice + Firebase SDK | Decoupled from core messaging path |
| **Media Service** | Signed URL generation, cleanup | Python (FastAPI) + S3 | Offloads large payloads from the main path |
| **Monitoring & Alerting** | Metrics, tracing, health checks | Prometheus + Grafana + OpenTelemetry | End‑to‑end observability |

---

## Scaling & bottlenecks  

### 1. WebSocket connection density  

* **Bottleneck**: Each Connection Manager instance can handle ~30 k concurrent sockets (limited by file descriptors and event loop).  
* **Mitigation**: Horizontal autoscaling behind a TCP load balancer; use **sticky sessions** based on user ID to keep a user’s socket on a single manager, reducing cross‑instance fan‑out.  

### 2. Kafka throughput  

* **Bottleneck**: Topic partitions per conversation; too many partitions can overwhelm the controller.  
* **Mitigation**: **Partition by conversation hash** (e.g., first 2 characters of UUID) to achieve ~10 k partitions per cluster, enough to parallelize consumption while keeping per‑partition ordering.  

### 3. Cassandra write amplification  

* **Bottleneck**: High write rate can increase compaction latency.  
* **Mitigation**:  
  * Use **TimeWindowCompactionStrategy** tuned for time‑series data.  
  * Set **replication factor = 3** (one per region) with **NetworkTopologyStrategy**.  
  * Enable **batch writes** only for atomicity within a single conversation (avoid large multi‑row batches).  

### 4. Cross‑region replication latency  

* **Bottleneck**: MirrorMaker introduces ~30‑50 ms lag, which can affect read receipts visible to remote participants.  
* **Mitigation**: Accept **eventual consistency** for read receipts; surface a “Seen by X (syncing…)” UI hint until the remote region acknowledges.  

### 5. Media upload bandwidth  

* **Bottleneck**: Large video files can saturate edge bandwidth.  
* **Mitigation**:  
  * Enforce **client‑side compression** and size limits.  
  * Use **multipart upload** with CDN edge nodes (e.g., CloudFront) to offload traffic from core region.  

### 6. Autoscaling thresholds  

* **Metric**: `messages_per_sec` per region, `ws_conn_count` per manager, `cpu_utilization`.  
* **Policy**: Scale out when any metric exceeds 70 % for 2 minutes; scale in when below 30 % for 5 minutes.  

---

## Trade‑offs  

| Decision | Pro | Con |
|----------|-----|-----|
| **Strong ordering via Kafka per conversation** | Guarantees exact message order, simplifies UI logic | Increases partition count; more metadata overhead |
| **Eventual consistency for read receipts** | Faster local UI updates, lower cross‑region traffic | Users may see stale read status, requires UI hints |
| **Cassandra for chat history** | Write‑optimized, easy horizontal scaling | Complex query patterns (e.g., full‑text search) need a secondary index or Elasticsearch |
| **Redis CRDT for presence** | Conflict‑free merges, no split‑brain issues | Higher memory footprint, limited to simple data structures |
| **WebSocket vs. HTTP Long‑Polling** | Sub‑ms latency, push‑only model | Requires sticky routing, more connection state to manage |
| **Separate media service** | Keeps core path lightweight, leverages CDN | Additional service to monitor, potential consistency gap between media metadata and message store |

Choosing the right balance depends on product priorities: if **ultra‑low latency** is non‑negotiable, we accept higher operational complexity; if **cost predictability** is paramount, we may relax ordering guarantees for low‑traffic groups.

---

## Interview tips  

1. **Start with requirements** – Interviewers love a structured approach. List functional vs. non‑functional items, then add realistic scale numbers.  
2. **Explain ordering guarantees** – Show you understand why per‑conversation ordering matters and how Kafka partitions enforce it.  
3. **Talk about CAP** – Emphasize that chat is **CP** for message ordering but **AP** for read receipts, and justify the trade‑off.  
4. **Use diagrams** – A quick mermaid sketch demonstrates clarity and helps the interviewer follow your mental model.  
5. **Address failure modes** – Discuss how the system recovers from a Kafka broker loss, a Redis partition outage, or a network split.  
6. **Mention observability** – Mention metrics (latency, queue depth, WS connections) and tracing (OpenTelemetry) as part of a production‑ready design.  
7. **Quantify** – When you say “scale to 10 k msg/s”, back it up with a rough calculation (e.g., each Kafka broker can handle ~500 k msg/s, so a 5‑node cluster is safe).  

---

## Lessons from production  

* **Heartbeat matters** – Without a 5‑second WS ping, idle connections linger, exhausting file descriptors. Implement a server‑side timeout and client‑side reconnection logic.  
* **Back‑pressure is essential** – During spikes, the Connection Manager should buffer at most 100 msg per socket; excess messages are dropped with a “retry later” flag to avoid OOM.  
* **Schema evolution** – Use **Cassandra’s flexible columns** for adding new fields (e.g., reactions) without downtime. Keep a version field in the message payload for forward compatibility.  
* **Media cleanup** – Schedule a nightly job that deletes objects older than 30 days **and** not referenced in the `media` table. This prevents orphaned blobs from inflating storage costs.  
* **Testing latency** – Simulate cross‑region traffic with a tool like **tcpcopy** or **chaos‑mesh** to verify the 100 ms SLA under realistic network jitter.  

---

## Key takeaways  

- Real‑time chat demands **strong per‑conversation ordering** and **sub‑100 ms latency**; Kafka + Lamport timestamps provide a reliable ordering backbone.  
- **Cross‑region replication** is best kept eventual for low‑priority signals (read receipts, typing) to preserve latency budgets.  
- A **sharded Cassandra** store handles massive write loads while supporting range queries for history retrieval.  
- **Redis with CRDTs** offers conflict‑free presence tracking across data centers without a single point of failure.  
- Horizontal scaling of **WebSocket front‑ends** and **Kafka brokers** is the primary lever for handling traffic spikes.  
- Trade‑offs between consistency, cost, and complexity must be articulated early; the right mix depends on product SLAs.  
- In interviews, frame your answer around requirements, CAP considerations, failure handling, and concrete metrics.