Most web traffic follows the clock. Exam traffic follows the calendar. For weeks the system runs at a fraction of its capacity, and then, at the start time printed on every admission card, tens of thousands of students open the same exam in the same minute. The monthly average says almost nothing about that minute, and that minute is the only one anybody remembers.
At Gozine2 I designed and deployed a Redis Sentinel high-availability setup that carried 45,000 to 70,000 concurrent users at exam peaks. This note is about the reasoning behind that kind of setup: what Sentinel gives you, when Redis Cluster fits better, how to configure the .NET client, and the caching and testing habits that matter more than the topology.
The shape of the problem
Two properties make exam platforms unusual. The spike is synchronized: users arrive together because the exam has a start time. And it is predictable: you know the date weeks ahead, so you can warm caches, run load tests and freeze risky deployments.
Synchronized arrival is what hurts a cache. In the first minute everybody asks for the same few things: the exam definition, its sections, the question set, the timing rules. If those keys are cold, or expire together, every request falls through to SQL Server at once. And if the Redis primary dies inside that window, a replica has to take over within seconds, without anyone editing a connection string.
What Sentinel does, and what it does not
Redis Sentinel is a separate process that runs alongside your Redis servers. A typical deployment has one primary, one or more replicas, and at least three Sentinels on independent machines. Four of its jobs matter here:
- Monitoring. Each Sentinel pings the primary and the replicas. If the primary stops answering for longer than
down-after-milliseconds, that Sentinel marks it as subjectively down. - Agreement. When enough Sentinels agree (the quorum you configure), the primary is considered objectively down.
- Automatic failover. The Sentinels elect a leader, which promotes the most suitable replica and points the other replicas at it. The election needs a majority of all Sentinels, which is why three is the practical minimum.
- Service discovery. Clients never hard-code the primary's address. They ask the Sentinels who the primary is for a service name, and follow the answer when it changes.
What Sentinel does not do is make Redis durable. Replication is asynchronous, so a write acknowledged by the old primary may never reach the replica that gets promoted; the Redis documentation says so plainly. For a cache that is acceptable, as long as everything in Redis can be rebuilt from the source of truth. If losing a write would hurt, that write belongs in SQL Server first.
Sentinel or Cluster?
Redis Cluster solves a different problem. It shards the keyspace across several primaries using 16,384 hash slots, each primary with its own replicas, so you get more memory and write throughput than one machine offers. The price is a set of constraints: multi-key commands, transactions and Lua scripts only work when all their keys map to the same slot, so keys get designed around hash tags such as {exam:42}:sections. Only database 0 exists, and resharding becomes an operation your team has to understand.
My rule of thumb: if the working set fits comfortably in one machine's memory and one primary can handle the command rate, Sentinel is the simpler and safer choice. Exam caches often fit that profile. The hot data is the content of a limited number of exams, read by very many people, and the load is far more read-heavy than write-heavy. Move to Cluster when measurements show that a single primary is the limit. Replica reads with CommandFlags.PreferReplica are a middle step, for data where slight staleness is harmless.
Configuring StackExchange.Redis for Sentinel
In StackExchange.Redis you list the Sentinel endpoints, not the Redis servers, and pass the service name from sentinel.conf. Setting ServiceName switches the client into Sentinel mode: it asks the Sentinels for the current primary, connects to it, and follows a failover when one happens.
using StackExchange.Redis;
var options = new ConfigurationOptions
{
ServiceName = "exam-cache", // matches "sentinel monitor exam-cache ..." in sentinel.conf
AbortOnConnectFail = false, // keep retrying in the background instead of failing startup
ConnectTimeout = 5000,
SyncTimeout = 2000,
};
options.EndPoints.Add("sentinel-1", 26379);
options.EndPoints.Add("sentinel-2", 26379);
options.EndPoints.Add("sentinel-3", 26379);
var redis = ConnectionMultiplexer.Connect(options);
redis.ConnectionFailed += (_, e) =>
Log.Warning("Redis connection failed: {EndPoint} {FailureType}", e.EndPoint, e.FailureType);
redis.ConnectionRestored += (_, e) =>
Log.Information("Redis connection restored: {EndPoint}", e.EndPoint);
builder.Services.AddSingleton<IConnectionMultiplexer>(redis);
The multiplexer is designed to be created once and shared, so it is a singleton; one per request is a classic way to run out of sockets. AbortOnConnectFail = false keeps a Redis outage at startup from taking the whole service down (on the current 3.x client; with Sentinel on 2.x, the first connect can still throw if no Sentinel answers). Logging the connection events, with Serilog in my case, gives you a timeline when you later ask what happened during a failover. Authentication and TLS go on the same options object, with their values read from configuration.
Then test failover on purpose. Stop the primary in staging while a load test runs and watch the next few seconds. Some requests will fail during the switch. Cache reads should treat a Redis error as a miss instead of returning a 500, but every miss is now a database query, at exactly the moment the database needs protection.
Cache-aside without a stampede
Cache-aside is the usual pattern: read from Redis, and on a miss load from the database and write the value back with a TTL. The weak spot is the miss. When a popular key expires, every request that sees the miss queries the database at once. Three defenses combine well:
- Locking. One caller rebuilds the key; the rest wait briefly and read the result.
- Early refresh. Hot keys are refreshed shortly before they expire, by a background job or probabilistically on read, so they never go cold.
- Jitter. A small random amount is added to each TTL, so keys written together do not expire together.
Here is a compact version of the first and third ideas:
public sealed class CacheAside(IConnectionMultiplexer redis)
{
private readonly IDatabase _db = redis.GetDatabase();
public async Task<T> GetOrLoadAsync<T>(
string key, TimeSpan ttl, Func<CancellationToken, Task<T>> load, CancellationToken ct)
{
var cached = await _db.StringGetAsync(key);
if (cached.HasValue)
return JsonSerializer.Deserialize<T>(cached.ToString())!;
var lockKey = $"lock:{key}";
var token = Guid.NewGuid().ToString("N");
if (await _db.LockTakeAsync(lockKey, token, TimeSpan.FromSeconds(10)))
{
try
{
cached = await _db.StringGetAsync(key); // another caller may have filled it
if (cached.HasValue)
return JsonSerializer.Deserialize<T>(cached.ToString())!;
var value = await load(ct);
await _db.StringSetAsync(key, JsonSerializer.Serialize(value), WithJitter(ttl));
return value;
}
finally
{
await _db.LockReleaseAsync(lockKey, token);
}
}
for (var i = 0; i < 20; i++) // someone else is loading: wait for it
{
await Task.Delay(50, ct);
cached = await _db.StringGetAsync(key);
if (cached.HasValue)
return JsonSerializer.Deserialize<T>(cached.ToString())!;
}
return await load(ct); // stop waiting and load directly
}
private static TimeSpan WithJitter(TimeSpan ttl) =>
ttl + TimeSpan.FromMilliseconds(Random.Shared.Next(0, (int)(ttl.TotalMilliseconds * 0.1)));
}
The lock only reduces load. If it fails in some edge case, two callers load the same data and nothing breaks; when it works, a thousand identical queries become one. If you can take a new dependency, HybridCache (the Microsoft.Extensions.Caching.Hybrid package, announced with .NET 9 and stable since March 2025; it also runs on .NET 8) gives you per-process stampede protection and a local memory layer in front of Redis.
For the hottest keys I prefer not to depend on expiry at all. If an exam starts tomorrow morning, a background job can load its definition and question set tonight and refresh them whenever the exam is republished. Expiry then only matters if that job fails.
TTLs, and what not to cache
A TTL is a statement about how wrong you are willing to be, and for how long. Content that changes only on publish, like exam definitions and question text, can take a long TTL plus explicit invalidation. Periodically recomputed aggregates and rankings should expire on the recompute interval. Small reference data can live long, ideally with an in-process memory cache in front of Redis, because a network round trip is still a round trip.
Some things I keep out of the cache, or never treat as the only copy:
- Submitted answers, scores, anything financial. A failover can lose the last writes. The database is the record; the cache is a copy.
- Time-critical decisions. Whether an exam window is open, or how much time a student has left, should come from server time and durable state.
- Low-reuse data. Values read once per user mostly cost memory and drag the hit ratio down.
- Large values. Redis executes commands one at a time, so a big value makes every other client wait, and it eats bandwidth when you have none to spare.
Rehearse the peak
Because the peak is predictable, it can be rehearsed. At Gozine2 I used Locust load tests to find bottlenecks before the peaks arrived. Locust scenarios are plain Python, easy to write and review, and they can produce the "everyone arrives at once" shape that matters for exams. A minimal scenario against a staging API:
import random
from locust import HttpUser, task, between
class ExamTaker(HttpUser):
wait_time = between(5, 20) # students read before they answer
def on_start(self):
resp = self.client.post("/api/exams/demo/sessions", name="start session")
self.session_id = resp.json()["sessionId"]
@task(1)
def exam_info(self):
self.client.get("/api/exams/demo", name="exam info")
@task(5)
def next_question(self):
self.client.get(f"/api/sessions/{self.session_id}/next", name="next question")
@task(5)
def answer(self):
self.client.post(f"/api/sessions/{self.session_id}/answers",
json={"choice": random.randint(1, 4)}, name="submit answer")
locust -f exam_day.py --headless --host https://staging.example.com \
-u 5000 -r 500 --run-time 20m --csv exam_day
The endpoints and numbers are placeholders; derive real ones from the registration count of the exam you are preparing for, and run Locust distributed (--master, --worker) once one machine cannot generate enough load. The spawn rate matters as much as the user count, because the ramp is the dangerous moment, when caches are cold and connection pools are still growing. Read percentiles, not averages, and watch Redis and SQL Server during the test. An average hides exactly the slow requests that make students hit refresh, and every refresh is more load.
SQL Server is still on the hot path
Redis takes reads off SQL Server, but writes still land there, misses still go there, and right after a failover the database briefly carries the read load alone. Alongside the Redis work at Gozine2, I optimized SQL Server queries, caching paths and hot APIs. The method barely changes: find the queries behind the hot endpoints (Query Store is the easiest place to start), give each one an index that covers it, and remove work the request does not need.
On the EF Core side, project hot reads into small DTOs instead of loading full entities, and add AsNoTracking when you do load entities for read-only work. For queries that run on every request, a compiled query helps:
private static readonly Func<ExamDbContext, int, Task<ExamSummary?>> GetExamSummary =
EF.CompileAsyncQuery((ExamDbContext db, int examId) =>
db.Exams
.Where(e => e.Id == examId)
.Select(e => new ExamSummary(e.Id, e.Title, e.StartsAtUtc, e.DurationMinutes))
.FirstOrDefault());
EF Core already caches query translation, but it still inspects the expression tree on every call to find the cached plan; a compiled query skips that step. The saving per request is small, and on an endpoint every student hits in the same minute small savings add up. Measure before and after. If the profile never showed query preparation as a cost, keep the code simple.
If your system has a calendar-shaped peak like this and you want a second pair of eyes on it, that is the kind of backend and performance work I take on. The next note moves from the cache to what students are actually doing: adaptive tests, where the server chooses every next question.
This note is also available in Persian: read the Persian version.