Google Cloud Platform
BigQuery: introducing powerful new enterprise data warehousing features
In today’s world, data powers the most amazing applications to make businesses successful. Data forms the basis for today’s business intelligence, as well as the foundation for new machine learning endeavors.
Google BigQuery, our cloud data warehouse, is a core part of Google Cloud Platform's data and analytics solution. BigQuery’s serverless architecture continues to redefine what it means to be fully managed. It scales seamlessly, analyzes terabytes of data in seconds and supports concurrent data queries for large organizations, without requiring capacity provisioning or systems tuning. As always, BigQuery is highly available, and encrypts all data at rest.
Today, we’re excited to announce an update to BigQuery that dramatically advances cloud data analytics for large-scale businesses.Easing your move to the cloud with BigQuery’s latest features enable it to seamlessly integrate with a wide array of popular business intelligence and enterprise data integration tools. With today’s update, BigQuery is easier to use by the millions of developers and business analysts who are familiar with SQL queries, paving the way for businesses to migrate their analytical workloads to Google Cloud Platform (GCP).
- BigQuery support for Standard SQL, implementing the SQL 2011 standard, is now generally available.
- New ODBC drivers make it possible to use BigQuery with a number of tools ranging from Microsoft Excel to traditional business intelligence systems such as Microstrategy and Qlik.
More control and visibilityWhen handling critical business data, data management becomes a serious challenge. To address this, BigQuery now offers:
- The ability to update, delete and insert rows and columns in BigQuery datasets using Standard SQL
- Integration with Cloud Identity and Access Management to manage fine-grained security policies for BigQuery users and resources
- Monitoring through Google StackDriver to track workload performance and usage
- Query sharing via links, to foster knowledge sharing and collaboration within organizations
Rich partner ecosystemChoosing the right technology partners is important for any business. From data management and development to visualization, partners make it simple for our customers to access a broad set of products and integrate our data analytics platform into their everyday business operations. We continue to expand the partner ecosystem for BigQuery, including:
- Enterprise data integration partners, such as Informatica and Talend, integrate with BigQuery to connect it to a diverse set of cloud and on-premises data sources
- Business intelligence partners, such as Tableau, iCharts and Looker, offer reporting and dashboarding tools to help businesses visualize their data in BigQuery for all types of users
How businesses are using BigQuery todayGoogle has invested almost two decades of engineering in data and analytics, a core part of which is our internal data warehouse, Dremel. Six years ago, we began to offer Dremel to Google Cloud customers under the name Google BigQuery. Since then, we’ve seen businesses come to rely on the simplicity and power of BigQuery much in the same way Google has.
Coca-Cola European Partners, with 300 million consumers and 25,000 employees, is leveraging the power of IoT and BigQuery to drive the sale of Coca-Cola products through its retail partners, including convenience stores, restaurants and other locations. Using BigQuery, the organization analyzes and segments data collected from beacons located throughout its large network of distributors. These insights are then used to retarget opted-in consumers using mobile media app advertisements.
The New York Times deployed BigQuery to help its analysts determine the impact of content on digital readers. Compared to their previous cloud and hardware data warehouses, BigQuery's performance and reliability have changed the nature of the job itself. According to Justin Stile, Director of Analytics: “The speed of BigQuery allows an analyst to really dig into the data and look beyond the results returned. The focus moves from the rote nature of setting up and optimizing long running queries to understanding the results returned and drilling down further. Analysts no longer have to decide if the understanding they are hoping to gain is worth the time it will take for the query to return a result.”
Viant, a leading advertising technology company, uses BigQuery to provide data and information to its advertising customers, including multinational corporations. On a typical day, the company tracks hundreds of millions of transactions and helps its customers turn that raw data into actionable intelligence. A common query is to understand a set of actions from a unique user that led to a certain event, for example, identifying which page he or she visited before completing a purchase. According to Viant, before BigQuery, using an on-prem database, that query took 24 hours to complete; with BigQuery it takes 10 seconds. Customers need real-time information to make fast decisions and don’t want to wait for those insights.
Other GCP data processing servicesBigQuery is complemented by a rich set of data processing services on GCP. The two most commonly associated with BigQuery are:
- Google Cloud Dataflow is a fully-managed service to execute data processing pipelines. As a complement to partners’ enterprise data integration tools, Dataflow is ideally suited for high-throughput data preparation, especially in streaming mode. Examples include ingesting application logs, mobile app events or IoT data streams, preparing them on-the-fly using advanced windowing mechanisms, and saving them in BigQuery. The upcoming 1.8 release of the Cloud Dataflow SDK will support BigQuery Standard SQL and the new BigQuery data types.
- Google Cloud Dataproc pairs the versatility of Spark and Hadoop ecosystem with the ease-of-use and efficiency of GCP. Cloud Dataproc clusters are an excellent complement to BigQuery, allowing you to perform advanced analytics using tools from the Apache big data ecosystem. For example, if you’re running graph analysis on data in BigQuery, you can use your favorite Spark graph library via Cloud Dataproc. To better support enterprise requirements, Dataproc recently released Hadoop High Availability deployments, integration with Cloud IAM and audit logging.