Vaccinating a nation: Vaccination app delivery in 30 days with Cloud Spanner

As the most specialized provider of cloud computing solutions in Poland, Chmura Krajowa (OChK) works to accelerate the digital transformation of Polish businesses and public institutions. In November 2020, the Polish government handed us a formidable challenge: starting from scratch and within 30 days, design and deploy an application to help vaccinate every citizen in Poland against COVID-19. Using Google Cloud products like Cloud Spanner to power the application, we met our goal, and the citizens of Poland are now better protected from the coronavirus pandemic. 

Defining the challenge

We were under considerable pressure to deliver an application that worked as expected and ran without errors or downtime, all with citizens, the government, and the media watching. 

Because the business requirements of the vaccination programme kept evolving in response to the changing situation, the system had to be modified with particular agility. The time pressure was exceptionally high – changes and new functionalities were implemented within hours.

The solution required three systems in one platform:

• One for 100,000 medical workers at 9,000 vaccination sites to run their own site-specific calendars and manage vaccination schedules. This was a complex undertaking, as vaccinations often required multiple doses,and people could choose different vaccines depending on their age and actual legislation at the given moment. 

• One for call centers where live operators could schedule appointments for callers. At the peak, about 2,000 operators worked 24/7 in shifts to field calls from across Poland.

• One for 36 million eligible users to go online via the web, a mobile device or an SMS gateway and Interactive voice response (IVR) to schedule their own appointments using a country-wide authentication scheme and to access follow-up care and resources. 

The scalability’s requirements were unpredictable for the third system for 36 million eligible users. The government could plan for the number and behavior of trained medical workers and call center operators, but not for the number and behavior of citizens scheduling their own appointments at the peak of a worldwide pandemic. This became more challenging as the vaccine roll-out progressed and more people became eligible, including those with greater familiarity with the internet and technology. On days when eligibility widened, huge numbers of citizens wanted to be among the first to get their vaccinations from a limited set of available calendar slots. 

Partnering with Google Cloud

To succeed within such a narrow timeframe, we needed to collaborate with a capable and experienced cloud provider. Given our experience working with Google on earlier projects for the Ministry of Health, Google Cloud was the clear and easy choice as it allowed us to focus on the project details, while being able to entrust virtually all infrastructure and scalability needs to Google.

Using Google-native Go programming language, we employed a wide range of Google Cloud solutions in building the application, including Google Kubernetes Engine (GKE) for backend services, like scheduling vaccination site staff, as well as external services and web requests.

Spanner played a key role in the application’s architecture, meeting the project’s most critical needs:

• High availability to avoid downtime due to maintenance.

• Strong consistency due to the transactional nature of the reservation system so that everyone has the same view of the data at a given point-in-time.

• Horizontal scalability to meet the demands of a program designed to be used by millions of citizens.

Of these three needs, scalability was the most important. We felt secure knowing that during peak traffic, we would not have to worry about the database scalability. 

Architecting the solution

The Spanner database schema consists of 30 tables, some of which are particularly crucial, like the table used by every vaccination site to define its calendar and specify appointment slots.The Ministry of Health workers at the vaccination sites are responsible for generating slots for a specified time span, using the back office system we provided them. The amount of data in  the table grew quickly to hundreds of millions of rows with thousands of rows per second. 

Integrating the suite of Google Cloud data solutions

Other Google products in the application stack include Memorystore for Redis for caching, storing user sessions and rate limiting. While it’s unfeasible to cache the available slots because they change so quickly, it’s easier to cache empty search results for given criteria, so if a user is looking for a certain location where there are no slots at a given time, this information can be temporarily cached to speed up the response. 

The stack also uses Pub/Sub for asynchronous messaging and Firestore for maintaining application configuration. For reporting, we run a Dataflow job that mirrors the Spanner database into BigQuery, and then another job mirroring data into the Ministry of Health’s internal data warehouse .  

For day-to-day internal reporting, we use Data Studio connected directly to BigQuery. The Data Studio reports are also used by external parties responsible for creating vaccination strategies to help manage the availability of vaccines and allocate resources.

Meeting the challenge 

We definitely see this project as a success. We deployed and managed it in record time without any major errors or outages, our client is satisfied, and most Polish citizens have now been vaccinated using our system either directly or indirectly. Projects don’t get any more exciting or challenging, and this one directly benefited the public health of our nation, which is extremely important to us. With Google Cloud products like Spanner, GKE, Pub/Sub, Dataflow, and BigQuery underpinning this critical application, we were able to deliver on our promise.

Learn more about Chmura Krajowa (OChK) and Cloud Spanner.