Exam-Peak High Availability: Redis Sentinel for 45,000 to 70,000 concurrent users

From December 2021 to October 2023 I was a senior backend engineer at Gozine2 in Tehran. The central piece of work was a Redis Sentinel high-availability setup that I designed and deployed; it sustained 45,000 to 70,000 concurrent users at exam peaks. Around it sat query and API tuning, load testing, release work and a data science team that I founded.

Context

Exam platforms live by the calendar. Load that is ordinary for most of the week arrives all at once when an exam opens.

Before the senior role, I joined Gozine2 in June 2021. In those first months I redesigned the reporting system, which reduced database load and sped up frequently used reports, and I introduced structured logging with Serilog. Both connect to the peak problem in a general way. Reports that put less load on the database leave more room for exam traffic, and structured logs make it much easier to understand what a system did under load.

The engineering problem

Exam traffic is bursty and synchronized. Users start together, the same pages and APIs are hit in the same order, and the last minutes before a deadline bring another wave of writes. In general, three things follow from that:

  • Hot paths dominate. A small number of APIs carry most of the load. An inefficiency there is multiplied by every user online.
  • Shared dependencies become single points of failure. A cache that every request touches is a performance win right up until it goes down. Then every request falls through to the database at the worst possible moment.
  • The peak is the wrong time to find out. Problems need to surface under simulated load beforehand.

What I built

  • Redis high availability. I designed and deployed a Redis Sentinel setup for the exam-peak traffic.
  • Database and API tuning. I optimized SQL Server queries, caching paths and hot APIs.
  • Load testing ahead of peaks. I used Locust load tests to find bottlenecks before the exam peaks arrived.
  • Release reliability. I improved release reliability with Dockerized services, structured logging and CI/CD.
  • A data science team. I founded and led a four-person data science team that built ranking, scoring and recommendation algorithms.

Design notes

The reasoning below is general: how such systems behave and why the tools fit, not the internal layout of Gozine2's setup.

Why Redis Sentinel

A single Redis instance is fast and simple, and it is also one machine. Sentinel adds a group of monitoring processes next to a primary and its replicas. When enough Sentinels agree that the primary is unreachable, they promote a replica and tell clients where the new primary is. The application keeps the same programming model; it asks Sentinel for the current primary instead of using a fixed address.

The trade-offs:

  • Redis replication is asynchronous, so a write acknowledged just before a failover can be lost. Data that must not be lost belongs in the database, with Redis as a cache or a fast secondary copy.
  • Sentinel decides by vote: a quorum must agree that the primary is down, and a majority of all Sentinels must authorize the failover. In practice that means at least three Sentinel processes on separate machines, so that losing one machine still leaves a majority.
  • Clients need Sentinel support to find the current primary, and they have to cope with the short window while a failover is in progress.
  • Sentinel gives availability, not sharding. All data still lives on one primary. Redis Cluster spreads data across nodes but brings its own complexity, so when the dataset fits on one machine, Sentinel is often the simpler choice.

Load testing with Locust

Locust describes user behavior in Python code, so a test can follow a realistic exam flow (sign in, open the exam, submit answers) rather than hammering a single URL. That matters because exam bottlenecks often appear between steps, such as a query that is fine alone but slow when many submissions arrive together.

Queries, caching paths and hot APIs

Tuning for peak load is mostly about the few endpoints that carry the traffic. The general levers are doing less work per request (better queries), avoiding repeated work (caching values that many users read), and making sure the cache path itself does not become the next bottleneck. Load tests show which lever matters for which endpoint.

Releases, logs and messaging

Docker images give the same artifact in testing and in production, and CI/CD turns a release into a repeatable process. Serilog writes structured events with named properties, so logs can be filtered by request, user or endpoint instead of searched as plain text. RabbitMQ is also part of the stack. In general, a message broker lets slow or non-urgent work move off the request path, so it does not compete with users during a peak.

Stack

  • .NET 6
  • SQL Server
  • Redis Sentinel
  • RabbitMQ
  • Docker
  • Serilog
  • Locust for load testing

Outcome

  • The Redis Sentinel setup sustained 45,000 to 70,000 concurrent users at exam peaks.
  • The earlier reporting redesign reduced database load and sped up frequently used reports.

Related

The adaptive testing platform case study covers my later assessment work at NovaTech. For backend performance and high-availability work, see Work with me.

Frequently asked questions

How many concurrent users did the Redis Sentinel setup handle?

The Redis Sentinel high-availability setup Mahdi Molavi designed and deployed at Gozine2 sustained 45,000 to 70,000 concurrent users at exam peaks.

How were bottlenecks found before exam peaks?

He used Locust load tests to find bottlenecks before peaks, and optimized SQL Server queries, caching paths and hot APIs.

What else did he lead at Gozine2?

He founded and led a four-person data science team that built ranking, scoring and recommendation algorithms.

More in this section

Projects