Scalable Matchmaker Performance with Cloud Memorystore for Redis Read Replicas
Jon Foust
Developer Advocate, Google
Scalable Matchmaker Performance with Cloud Memorystore for Redis Read Replicas
Online games have grown in popularity lately, and with the influx of players in multiplayer games, meeting the increased demand is a known issue within the gaming industry. We developed Open Match, an open source multiplayer matchmaking framework, to solve this issue by providing core services commonly found in most matchmaking services and hosting them on Kubernetes. Open Match enables developers to focus more on match logic and less on running at scale.
Open Match uses a Bitnami open source Redis image but allows for using any key-value store. In the architecture diagram below you can see how Open Match interacts with a state store:
Open Match Architecture
While Open Match is designed to scale with player activity, by default, all reads and writes are performed on the primary Redis node, which can potentially approach capacity limits. Moving all read operations to replicas can reduce the stress on primary, but also introduces concerns with how well this approach scales and the potential for Open Match to return outdated matchmaking results.
Earlier this year, Cloud Memorystore for Redis read replicas launched, and Open Match was the perfect candidate to test running at scale. Cloud Memorystore for Redis read replicas expose a read endpoint for load balancing read queries across up to five replicas over multiple zones. This allows for high availability and minimal application downtime. Memorystore also supports Redis 6, which introduced multithreaded I/O on M3 or higher configurations. At scale, this performance could support up to a million players looking for (and finding!) matches.
Running at Scale
The example configuration provided in the Open Match repository uses Redis Sentinel with three replicas for high availability with all reads and writes performed on the primary node. With a quick swap in configuration, we will confirm that Memorystore for Redis standard instance with read replicas could be used as a drop-in replacement with no code changes. Although Memorystore supports an even high number of read replicas, to get a close comparison to the Sentinel example configuration, three replicas were used with Memorystore.The Open Match repo contains scale tests for benchmarking. To test the compatibility of read replicas and really give the system a workout, we ran the benchmark with 15000 simulated players requesting a match per second.These numbers represent a surge of players entering matchmaking as a game becomes a viral hit. Here are some metrics confirming that the application was stable and performed well, gathered over several days.
Confirming Redis Replicas Are Up-to-Date
Open Match tracks the player's process through the matchmaking system as a status field in Redis, and out-of-sync Memorystore replicas could lead to a variety of problems. Some examples of players who should no longer be considered for matchmaking include:- Players who have already been matched
- Players who have chosen to leave the queue
The chart shows a measured latency under one second behind the primary node. With this little latency, replicas are up-to-date, matchmaking requests are completed, and players are off to enjoy their games.
Memory Usage
As players enter the queue and are waiting to be matched, Open Match stores them in Redis. If matchmaking slows down due to a performance bottleneck, the number of players waiting can start to increase. An important consideration is making sure the number of players waiting doesn't outstrip the available memory for Redis.
Open Match Memorystore Keys
As the chart shows, the number of keys slowly increased to 2k during our tests. There it remained steady and the application operation became stable, with players being matched at approximately the rate they enter the matchmaking queue. From this metric, we can see that the replacement of a Redis Sentinel configuration with our Cloud Memorystore for Redis configuration using read replicas is operating normally and doesn't seem to have any issues with getting all players into matches.
Conclusion
In this test, we demonstrated how three Cloud Memorystore for Redis read replicas can serve as a drop-in replacement for Redis Sentinel while giving simulated players a similar matchmaking experience. Cloud Memorystore for Redis supports additional read replicas, which could improve Open Match performance even further. This capability will help an online game smoothly and rapidly scale to handle matchmaking requests in the event of a player surge.
You can learn more about Cloud Memorystore for Redis read replicas in the documentation.
To learn more about Open Match, check out the GitHub repo, explore the documentation for tutorials and samples, and join the Slack channel to connect with developers and contributors.
Follow Jon on Twitter for more Open Match and gaming content.
