Overview of Google Compute Engine for Cloud Developers
Google Compute Engine provides a platform for developers to run their large-scale workloads on Linux virtual machines hosted on Google's infrastructure. With access to Google’s cloud technologies, it is now possible for developers to build globally distributed applications by leveraging Google’s global network. For developers who are experienced with building applications on infrastructure as a service, many of the common design patterns and technology are a perfect fit for Compute Engine. This paper highlights the compelling features of Compute Engine that allow developers to architect scalable and globally distributed systems.
Overview for Cloud Developers
As developers continue to build applications in virtualized environments, their knowledge of design patterns, implementation challenges, and maintenance techniques continues to expand. Although a significant portion of this knowledge is applicable to Compute Engine, such as best practices for building robust systems, there are many advantages of the platform that might be unfamiliar to experienced cloud developers. The following sections highlight the key differentiators of Compute Engine and discuss how to augment existing knowledge to build next-generation global applications.
When developing the core infrastructure of Compute Engine, Google focuses on delivering consistent performance because it significantly improves application stability and reduces operational concerns. As a result, the noisy-neighbor problem, where one tenant noticeably impacts the performance of another tenant, is rare on Compute Engine. For example, a physical disk spindle is currently dedicated to a virtual machine for every four cores within an instance. This physical scratch disk is optional and added to machine types with a ‘-d’ in the name. Persistent disk is the other disk option and is designed to provide consistent performance.
A common practice when building on virtual environments is to spin up a new virtual machine, run performance tests, and delete the machine if it provides poor performance. This practice is not required on Compute Engine because every instance and persistent disk provides the same level of performance. One way Google solves this problem is by designing virtualization software from the ground up to provide excellent tenant isolation. Additionally, all Compute Engine n1 instances are backed by the same hardware configurations, which provides the uniformity expected from high-performance infrastructure. The resulting deterministic performance results in more stable systems, fewer operational concerns, and lower costs due to reduced number of virtual machines required to handle the variance.
Global Private Network
The global network of Compute Engine enables new, more efficient cross-region designs. Developers immediately recognize the benefits when they start new projects. The low-latency, high-bandwidth network between regions enables developers to create global applications by viewing data centers as a globally connected grid instead of isolated regions. All instances within a Compute Engine project share a global, private network and each instance can refer to any other instance by hostname or internal IP address. Since traffic travels over Google’s private fiber network, performance is significantly higher and more predictable when compared with the alternative of cross-region communication over the public Internet. Another benefit of the global network is that Google Cloud Storage is accessible globally by instances in any region without traveling over the public Internet. Similarly, if a persistent disk snapshot is created from a disk in a specific zone, the snapshot can be used to create a new persistent disk in any other global region.
All instances in a Compute Engine project communicate over project private networks. Each network can be set up with unique firewall rules that provide flexibility and security in a network configuration. For example, database virtual machines can be configured to accept no external traffic and only receive traffic from other virtual machines with a custom-defined web server tag. Google’s global network supports existing best practices for building highly available systems. Additionally, cutting-edge developers now have an opportunity to push the boundaries of what is possible in cross-region deployments by utilizing the global high-performance network.
To provide impressive technology and network performance, Google regularly takes each Compute Engine zone offline for a period up to two weeks approximately twice a year for maintenance and upgrades. As a result, all virtual machines in a specific zone during a maintenance window are terminated. Although all persistent disks within the zone are preserved, they are inaccessible during the maintenance window. Developers of robust systems should design for high availability and utilize Google-provided technologies to simplify the process of moving applications between zones. By building robust systems and following the best practices provided in this paper, it is a straightforward process to handle maintenance windows and move application components between zones.
When building an application on Google Cloud Platform, developers create a project for managing billing and developer access to multiple services. Compute Engine exists as a service within a standard Google Cloud Platform project. This configuration enables Compute Engine to seamlessly integrate with other Google Cloud Platform services through the use of service accounts. For example, when creating an instance, command-line tool access to Cloud Storage can be configured through service accounts by simply specifying permissions. Instances that need only to download from Cloud Storage can be configured in read-only mode to prevent any chance of accidentally deleting critical data.
The Compute Engine resources within the project can be accessed through a RESTful API utilizing OAuth 2.0, command-line utility, and web console. Each of these methods for accessing Compute Engine resources have unique use cases. Reviewing the complete details in the Compute Engine developer documentation is recommended. The web console is the easiest tool for developers who are learning Compute Engine. However, when building production applications, the command-line utility is important for scripting large deployments, and the API is utilized from other applications.
Virtual machines are the core components of Compute Engine and the key building blocks when creating distributed systems. New Compute Engine developers often find that the time it takes to create an instance is consistently fast. Additionally, the performance of each instance is consistent. Developers no longer need to test the performance of each machine they create to verify their instance is performing as expected.
The many configuration options available when creating an instance are fully detailed in the developer documentation. Many of these options, such as instance type, are familiar to cloud developers. Other options, such as metadata, startup scripts, and custom images, may be unfamiliar features, but are critical for building highly available systems. Metadata can be attached to an instance as key value pairs during creation or runtime and can be accessible from the instance. Metadata is used to specify instance and project-specific data that can be utilized for configuring functionality and registering with additional services. One of the most important instance metadata key-value pairs is a startup script that can be stored in Cloud Storage or directly in the metadata. The script automatically executes once an instance is running and is useful for configuring the instance and loading the required packages and software components. If packages are required for many instances in a default configuration, a custom image can be created and used to decrease the time required for configuration. The custom image is a modified version of a base operating system, and new instances can be directly created from a custom image. Additionally, custom images can be made available for usage in any region. As a result, a custom image created in North America can be used to create an instance in Europe.
One interesting feature of the Compute Engine command-line utility is the ability to move an instance and all associated persistent disks to any of the available global zones with a single command. However, it is important to remember that all information on scratch disks is deleted and not transferred with the instance. During command execution, snapshots of the instance’s persistent disks are created. Once they complete successfully, persistent disks in the original zone are deleted. The snapshots are used to create new persistent disks with the same names in the new zone and are automatically deleted once the new disks are created. This command is useful when managing large-scale projects, as it reduces the number of commands required to manually create snapshots and recreate virtual machines.
When building cloud applications, a block-store file system that lives beyond the life of an instance is frequently required. Google provides persistent disks that are bound to a zone and can be attached to any virtual instance within the same zone. All data on disk is automatically encrypted. Details of the encryption mechanism are available in the developer documentation. One of the most notable features of a persistent disk is that performance is consistent by design. Consistency is very important, because it allows developers to expect a specific level of performance when building complex systems. If multiple instances need to access files on a persistent disk, the disk can be mounted in read-only mode to multiple instances in the same zone.
Another unique feature of persistent disk is that snapshots of disk contents are usable globally within a project. A persistent disk in Europe can be easily moved to a new zone in North America by utilizing API calls or the Compute Engine Web Console. Persistent disks can also be used as the root disks for virtual machines, which allows an entire virtual machine’s state and configuration to be maintained beyond the life of the virtual machine and moved to any global zone.
The other block-store option available on Compute Engine is a local scratch disk that lives and terminates with the specific instances. Therefore, it should only be used for storing non-mission-critical data or information that can be recovered through other sources. All data on scratch disk is encrypted at rest, and information cannot be retrieved from the disk once the instance terminates. If heavy utilization of scratch disk is required, four or eight core instances are recommended, because a dedicated spindle is currently provided for every four cores. The dedicated spindle provides consistent performance, as it eliminates the possibility of another tenant heavily utilizing the same spindle. When building systems that provide data redundancy at a level higher than the disk, in some cases scratch disk provides similar or better performance at a lower price than a persistent disk. However, evaluating persistent disk performance for each application is highly recommended, as the benefits can simplify the design of other components and processes.
As Google Cloud Platform continues to add functionality and launch innovative new products, deploying scalable cloud applications becomes easier than ever. For example, the challenges associated with architecting around maintenance windows are reduced through new features and functionality. Features such as global persistent disk snapshots and persistent root disks are just early examples of how Google continues to provide important Compute Engine functionality. Additionally, Compute Engine leverages standard Google data-center hardware, which greatly simplifies the process to create additional zones and regions as the service grows. This paper presents a snapshot of the functionality as of June 2013. Please sign up for the Google Cloud Platform newsletter to stay up to date on the rapid deployment of new features.
Developers now have access to a global interconnected grid of data centers to build applications that can achieve true high availability and are no longer constrained to a single region. Google Compute Engine is the best place for smart developers who require virtual machines and want to be on the leading edge of building the next generation of cloud applications. Developers are now empowered to push the boundaries of what is possible by combining new ideas with the speed, consistency, and innovation of Google Cloud Platform.