Finding data insights faster with BigQuery and GCP Marketplace solutions
John Day
Product Marketing Manager, Google Cloud
There are plenty of trends and hot topics in the enterprise technology market today. One common area we hear about from users is that there’s a lot of data to collect, manage, and analyze. And whatever industry you’re in, you probably want to do something more with your data. We built BigQuery, one of the important tools in the Google Cloud Platform (GCP) arsenal, to provide serverless cloud data warehousing and analytics with built-in machine learning to meet modern data needs. (While we’re on the topic, we’re pretty excited that Google was recently named a leader in The Forrester Wave™: Cloud Data Warehouse, Q4 2018.)
The journey to a fully functioning data warehouse starts with identifying and moving data. We often hear developers ask: “How do we get our data into BigQuery?” Google Cloud partners with several extract, transform, and load (ETL) providers like Informatica, Talend, Stitch, and Matillion to make it easy for you to get started with BigQuery. Another essential tool for GCP users is the GCP Marketplace, which offers other options to get data into BigQuery.
What you’ll find in the GCP Marketplace
The GCP Marketplace offers many production-ready commercial and open source tools that work well with Google Cloud services. All the solutions in the GCP Marketplace are easy to deploy, verified against vulnerabilities, and optimized to run on GCP. They’re also designed to be easy to manage.
For BigQuery in particular, there are useful ETL and analytics tools available in GCP Marketplace, including:
Matillion ETL for BigQuery: Matillion transforms data, across its various locations and forms, into BigQuery, enabling users to make informed, data-driven decisions
SuperQuery AI: This powerful SQL integrated development environment for BigQuery includes AI that optimizes queries in real time
Striim: This is an end-to-end data integration platform that enables continuous ingestion of real-time data
Looker: Looker connects directly to your BigQuery instance and optimizes data discovery within BigQuery itself
GCP Marketplace also offers our Google-built ingest tool:
Cloud Dataflow: This offers simplified stream and batch data processing, with equal reliability and expressiveness
You can also find public datasets in GCP Marketplace, which carry only the cost of querying the data, with 1TB free per month. Datasets are useful for many reasons, ranging from incorporating practical insights from the data into your business to training machine learning models. Some real-life use cases include:
Plot the path of a hurricane with real time data using the Geostationary Operational Environmental Satellite dataset from NOAA.
Analyze historical patent data to find trends for certain industries, track expirations, or look for owners with Google Patents Public Datasets.
How one large enterprise tackled BigQuery data ingestion
At Next ‘18 this year, we showcased the story of a business intelligence architect at QuoteCenter, a subsidiary of home improvement giant Home Depot, who used BigQuery and Matillion to transform the company’s business analytics. They were using on-premises hardware and were relying on periodic snapshots of data, which was not providing the speed or scalability that they needed to analyze data properly. The business decided that BigQuery would be right for them, but they needed to load data from many sources into the service. You can see a simple architectural diagram of the services they used and data sources here:
After a quick Google search (useful tool, right?), they found Matillion ETL on GCP Marketplace. The architect and his team liked the usage-based pricing model of Matillion. As with any commercial service launched from GCP Marketplace, support is offered by the partner, and usage charges are combined with charges for other Google Cloud services you may be using, so there’s just one bill.
Getting GCP projects launched quickly
BigQuery is one area where GCP Marketplace can help you get started, and there are plenty of others. We’ve heard from users looking for a fast way to begin a new AI project who find our deep learning virtual machine template in the GCP Marketplace particularly useful. Starting with this template instead of configuring a GPU-enabled VM by yourself can save time and energy that can otherwise be spent building your next application. You may be searching for ways to accelerate hybrid development within your organization and find Kubernetes applications that can be deployed quickly to Google Kubernetes Engine (GKE) or anywhere you have a Kubernetes cluster.
Whatever project or initiative you’re working on, you can choose from more than 1,000 commercial, open source, and Google-built tools in the GCP Marketplace. There’s also new AI-powered technology that will examine your Google Cloud environment and give you recommendations for GCP Marketplace tools. In addition, you can reach the GCP Marketplace from within Google Compute Engine and Kubernetes Engine. You can see here what the view into GCP Marketplace from Compute Engine looks like.
Getting started with BigQuery
If you are working on your own BigQuery project, check out the solutions on GCP Marketplace and find what’s right for you. And to see what’s new, what’s popular, and what is recommended for your environment, visit the GCP Marketplace page in the Google Cloud Console.
