How to get real-time, actionable insights from your Fastly logs with Looker and BigQuery

Editor’s note: Fastly, whose edge cloud platform offers content delivery, streaming, security and load-balancing, recently integrated its platform with Looker, a business intelligence tool. Using Google BigQuery as its analytics engine, you can use Fastly plus Looker to do things like improve your operations, analyze the effectiveness of marketing programs — even identify attack trends.

This past August we announced a deeper integration between Google Cloud Platform (GCP) and Fastly’s edge cloud. In addition to using Fastly to improve response times for applications built on GCP, Fastly customers can stream Fastly logs in real-time from the edge to a number of third parties for deeper analysis, including Google Cloud Storage and BigQuery. We're now expanding upon this partnership by integrating Looker, a powerful business intelligence tool, into our offering.

Looker can analyze Fastly log data on its own or combine it with other data sources in BigQuery such as Google Analytics, Google Ads data or security and firewall logs, allowing customers to run queries against these data sets and present findings in dashboards to facilitate better business decisions.

As part of this collaboration, we created a “Looker Block” for Fastly Log Analytics in BigQuery, to help you get up and running quickly with key visualizations and metrics. Think of Looker Blocks as analytical patterns that can be used as a starting point for modeling a data source. They include dashboards and key metrics that can be explored ad-hoc to build new customized reports. The Fastly Looker Block can be extended to account for specific Fastly logging use cases while also connecting to other data sources in BigQuery for more comprehensive analysis.

Looker runs all analytics in BigQuery — data is never moved from the source — leveraging BigQuery’s performance and features directly. This functionality is made possible via Looker’s modeling layer, LookML, which serves as an abstraction of SQL.

Here are some common use cases for GCP customers who wish to take advantage of both Fastly and Looker:

DevOps - Fastly streams 100% of logs from the edge to BigQuery in real time, providing insights into web and app usage. Using Looker dashboards, you can correlate the most popular URLs, website and app activity by country, and activity by client device. You can then use this information to see which content is gaining the most traction where, and what devices it’s being consumed on.

Leveraging BigQuery analytics, Looker can also analyze Fastly log data and create dashboards to use for troubleshooting. Here, Looker can illustrate failed requests by geo / datacenter, and country, or the slowest URLs. You can also use these dashboards to troubleshoot connectivity issues, pinpoint configuration areas that need tuning, and identify the cause of service disruptions.

Marketing/Digital Advertising - Looker can cross-reference Fastly log data with other data sources for broader insights. For example, by combining Fastly app activity by country with Google Ad data, marketers can discover where engagement is higher and which users are more likely to consume their ads.

Security - You can also use Looker to help visualize Fastly’s real-time logs for insights into live attack trends. Fastly’s Web Application Firewall (WAF) logs can be fed into Google BigQuery. Looker then pulls that data to create dashboards illustrating trends in attacks, breakdown of attacks over time, spikes in attacks from a given attacker, and more.

If you need to get started with Looker, you can request a demo. Once you’re using Looker, follow the documentation to connect BigQuery to your Looker instance. Make sure Looker has access to your Fastly data and any other data sources you’d like to explore (e.g., Google Analytics, Google Ads data, security or firewall log data).

Another way to get started with Looker and Fastly is to use the Log Analytics by Fastly Block. You can either download the entire block into Looker by following the directions, or selectively migrate pieces of the block by simply copying and pasting the block LookML into your Looker instance. Then customize your LookML model to account for any custom metrics relevant to your business within the Fastly logs data (or any other data you’ve made available to Looker in BigQuery).

Now that you are set up with Fastly, BigQuery and Looker you’re ready to get real-time insights into how your web and mobile traffic is performing and better understand users interactions with your applications. Have questions? Please contact us.