Handles over one million users in a day with less than 0.08% error rate
Respond to all requests in under 60 milliseconds
With Google Cloud, Indonesia's Ministry of Education, Culture, Research, and Technology (MoECRT) is able to scale its evaluation program, the Asesmen Nasional, effectively with rapid responses and no downtime.
With Google Cloud, Indonesia's Ministry of Education, Culture, Research, and Technology (MoECRT) is able to scale its evaluation program, the Asesmen Nasional, effectively with rapid responses and no downtime.
Working with different parties would require real collaboration, so we assembled a task force consisting of the Cloud platform, Core QA, Software Development Engineers in Test, and Technical Program Management teams. A combination of engineering and program managers ensured the task force always had a holistic view when solving problems.
Muhammad Saiful Islam
Cloud Platform Architect at GovTech Edu
An annual education program by Indonesia's Ministry of Education, Culture, Research, and Technology (MoECRT) that aims to improve the quality of education by evaluating student learning outcomes, teaching quality, and the learning environment, the Asesmen Nasional welcomes a significant number of students to its two-day online/semi-online event.
Calling for requires extensive collaboration with a clear view on everything, the Asesmen Nasional is a joint effort between three entities: Pusat Asesmen Pendidikan, which owns and develops the whole Asesmen Nasional program; Pusat Data dan Informasi, which manages the IT infrastructure for the ministry, including the cloud; GovTech Edu, which supports the ministry under Pusdatin coordination to build and run technology-based ecosystems, and Radya Labs, a digital agency that leverages cloud solutions to support the scalability improvements of the assessment system.
"Working with different parties would require real collaboration, so we assembled a task force consisting of the Cloud platform, Core QA, Software Development Engineers in Test, and Technical Program Management teams. A combination of engineering and program managers ensured the task force always had a holistic view when solving problems," says Muhammad Saiful Islam, Cloud Platform Architect at GovTech Edu.
Following an architecture review at the start of 2023 with Pusmendik and Radya Labs utilizing Google Kubernetes Engine (GKE), it became clear that there were multi-disciplinary challenges when it came to edge cases and scalability. A more efficient infrastructure was needed, and that required the use of Google Cloud services across the various touchpoints.
The scalability of Google Cloud and its pay-as-you-go model became the key foundation for the teams involved to build from.
The new way to create a more efficient, robust infrastructure with key insights
With the key information at hand, the teams understand what needs to be addressed ahead of the Asesmen Nasional every year. As a start, running on the Google Cloud infrastructure allows the team to focus fully on the application side.
On the backend, the database runs on Cloud SQL PostgreSQL. Using an RDBMS solution allows Radya Labs to keep using the paradigm familiar to Pusmendik as the program owner. It also has a Query Insight feature, which enables the team to identify less-performant queries without additional cost.
In addition, the student response database is sharded, and the application side modified to incorporate the sharding logic. This helps ensure that the application side is stateless, as the platform utilizes pod and cluster autoscaling.
The team has also introduced a message queue to log students' actions utilizing Cloud Pub/Sub, which can then be stored in a single database instance, with no need for sharding despite heavy incoming traffic. A combination of Cloud Storage and Cloud CDN is used to offload static asset traffic from the application, which makes it possible to maximize the application's resources to handle the most essential function of allowing students to fetch questions and respond to them.
The team can utilize the built-in Cloud Monitoring and Cloud Logging stacks to observe and monitor generated metrics and logs. Logs are also emitted from the application side and captured by Cloud Logging, and are compatible with the Error Reporting feature to identify new errors as they come up, allowing tracking and prompt rectification. All of these help to establish a baseline and iterate between performance tests as the team continues to refine the platform.
Based on the performance tests, the team can fine-tune the capacity required and make an informed forecast in preparation for upcoming sessions with more students. The robust system and observability afforded means that the teams can monitor performance and uncover issues that are straightforward fixes. This includes eliminating slow database queries as much as possible to reduce latency and, eventually, limiting the log queue, and disabling several system logs based on a test canary deployment memory profile. The latency drops from seconds to milliseconds, and the error rate, too, drops significantly.
Based on the adjustments, the system can handle over one million users on a single day, with traffic peaking at ~80,000 requests per second with ~30ms p95 latency. This is only possible through the collaborative effort between the Pusmendik operational team's meticulous scheduling and its technical team's robust preparation for the massive traffic through Google Cloud.
The Ministry of Education, Culture, Research, and Technology is the organization responsible for education, culture, research, and technology affairs in Indonesia, and runs the annual Asesmen Nasional to improve the quality of education.
Industry: Education
Location: Indonesia
Products: Cloud CDN, Cloud Logging, Cloud Monitoring, Cloud Pub/Sub, Cloud SQL, Cloud Storage, Error Reporting, Google Kubernetes Engine (GKE)
