Our matchmaking system started life as a single Redis sorted set. Player skill ratings went in, queue timeouts came out. At 50,000 concurrent users it worked fine. At 500,000 it started to crack — and at 1.2 million, which is where we peaked last December, it buckled completely. P99 queue times ballooned to 47 seconds during prime time. Players in Southeast Asia were routinely landing on servers in Frankfurt. We had built ourselves into a corner, and we needed to rebuild fast.
The Old Architecture and Its Hidden Debt
The original system used a single authoritative matchmaking process — a monolithic Go service we internally called the Broker. The Broker polled Redis every 100ms, scored candidate pairs, and dispatched match assignments to regional game-server clusters. It was simple, auditable, and elegant for a team of four. It was not designed for a team of 140 serving two million daily active players across six continents.
The specific failure modes were instructive. First, the Broker became a write-hotspot: every player state update — skill deltas, ping measurements, game-mode preferences — funneled through a single Redis primary. At scale, replication lag meant regional replicas were reading stale data. A player finishing a match in Tokyo might not appear in the Singapore replica's pool for up to 800ms, creating phantom availability gaps that inflated queue times.
“We were essentially running one matchmaker for the entire planet and wondering why it felt slow. It is the engineering equivalent of funneling all global air traffic through a single control tower.”
— Nexus Infrastructure Lead
The New Design: Federated Session Brokers
The core insight driving the new architecture is that most matches are regionally local. A player in São Paulo almost never benefits from being matched against someone in Warsaw. Ping alone makes it undesirable. So instead of one global Broker, we decomposed into a mesh of regional Brokers that each own their geographic catchment area, with a thin Global Orchestrator handling cross-region overflow and tournament-mode play where latency constraints are relaxed.
Regional Broker Design
Each Regional Broker is a stateful Rust service deployed as a three-node Raft cluster within a single AWS region. Player presence is written to the local leader and replicated synchronously to the two followers before acknowledgement — so reads from any node are always consistent. The pool is modeled as a multi-dimensional k-d tree keyed on skill rating, ping percentiles to available server nodes, and preferred game modes. Candidate matching runs on each heartbeat (currently 50ms) using an approximate nearest-neighbor query that returns candidates within an acceptable skill and latency radius.
Cross-Region Overflow
If a player's local queue exceeds a configurable wait threshold (default: 8 seconds), the Regional Broker publishes an overflow request to a Global Orchestrator via a low-latency NATS JetStream topic. The Global Orchestrator holds a bloom-filter summary of every region's available player pool, refreshed every 500ms. It identifies candidate regions, brokers a temporary match-ticket exchange, and lets the two regional Brokers negotiate a cross-region assignment. Players always know their expected ping before confirming — they can accept or re-queue locally.
Key Engineering Decisions
- Rust for the Broker core — deterministic latency, no GC pauses, safe concurrency via async-std
- Raft over Paxos — simpler leader election, mature tooling with openraft crate
- k-d tree over flat scoring — O(log n) candidate search vs O(n), critical at 200k+ players per region
- NATS JetStream for cross-region messaging — persistent, replay-capable, sub-1ms regional RTT
- Prometheus + Grafana with SLO-based alerting — P50/P95/P99 queue time, skill-delta distributions
- Shadow mode rollout — new Broker ran in shadow for six weeks, emitting metrics without affecting live matches
We ran the new and old systems in parallel for six weeks, validating that the new Broker's match decisions agreed with the old system more than 97% of the time before cutting over traffic.
Results After 90 Days
The results exceeded our targets across every metric we track. P99 queue time dropped from 47 seconds to 17.4 seconds globally — a 63% reduction. P50 (median) queue time fell to under 4 seconds in all tier-1 regions. Cross-region matches, which previously accounted for 23% of all sessions, now represent only 6%, meaning players are landing on lower-ping servers as a default rather than as an exception.
- P99 queue time: 47s → 17.4s (−63%)
- P50 queue time: 12s → 3.8s (−68%)
- Cross-region match rate: 23% → 6%
- Skill-delta variance (fairness metric): −41%
- Broker service p99 CPU: unchanged despite 3× traffic growth
- Zero unplanned downtime in 90 days post-cutover
What's Next
The new architecture gives us foundations we did not have before. Next quarter we are shipping dynamic pool segmentation — the ability to spin up ephemeral sub-pools for limited-time game modes (tournaments, seasonal events, Olympic qualifiers) without polluting the global ranked pool. We are also prototyping ML-assisted skill estimation that replaces the current Elo-variant with a gradient-boosted model trained on match outcome sequences, which our internal benchmarks suggest will further reduce skill-delta variance by another 20–25%.
Building infrastructure for global competitive gaming at this scale is genuinely hard. We hope this write-up saves another team somewhere a few months of iteration. If you are working on similar problems, we are hiring — the link is in our Careers section.