Sub-second Looker dashboards with Firebolt

Performance issues are a data warehouse problem

Looker helps analysts break out of data silos by providing governed and live access to all data. However, live access to data comes at the price of being dependent on underlying data warehouses and databases for performance. In situations where the underlying database can’t process the queries fast enough, you could end up with dashboards that take tens of seconds to minutes to refresh.

You can employ many Looker features to help improve performance, from datagroups, caching, and persistent derived tables, to Looker SQL optimization that helps maximize data warehouse query performance.

But at the end of the day, performance is a data warehouse problem, not a Looker problem. Caching and other optimizations may not be enough when Looker still has to wait for a query to get data for the first time from the data warehouse.

That’s why we’re excited to announce that Firebolt, the cloud data warehouse offering extreme speed and elasticity, is an officially supported Looker dialect. This new support gives all Looker customers access to the sub-second performance that Looker and Firebolt can deliver at any scale.

Looker and Firebolt

Our joint customers solved their performance challenges one project at a time by using Firebolt side-by-side with their existing data warehouse.

Firebolt is a complete redesign of the cloud data warehouse. With a decoupled storage and compute architecture, we designed it from the ground up to deliver 10x or greater improvements in speed, scale, and cost efficiency. Almost all of our customers used another cloud data warehouse before they added Firebolt.

Take Appsflyer, one of the leading mobile analytics and attribution vendors. Over 12,000 companies rely on its analytics. Internally they have over 1,000 Looker users analyzing tens of terabytes of data, and billions of rows. When they could not deliver a specific report to executives, they decided to solve the problem differently. Their current data warehouse couldn’t perform the bigger queries they needed, and the smaller ones were taking a minute or more to return results.

They decided to implement Firebolt specifically for that set of executive reports. Appsflyer took the existing schema from their data warehouse, created the tables in Firebolt, ran SQL ingestion scripts, and modified LookML to point to Firebolt. The dashboards didn’t need to change.

They achieved sub-second performance for every query, with 33-183x¹ faster performance in the first project, and 1/5th the cost of any of the alternatives. The performance, and price-performance gains have only improved over time.

What were some of the key reasons for the performance improvements?

• Sub-second data access: Firebolt has a unique storage and data access layer with F3 (Firebolt File Format) that does not fetch entire segments or partitions from S3, only exact data ranges. The result is much more intelligent data access with 10x or faster data access times.
• Query optimization: Firebolt takes Looker SQL and optimizes it even more specifically for Firebolt. It turns it into a physical query plan — using indexing, multi-level pushdown optimization, cost-based optimization, and just-in-time compilation — that delivers fast performance each time.
• Indexing: Indexing matters. Firebolt is the only cloud data warehouse that has indexes for predicates (tables), text searches, joins and aggregations. These indexes deliver much faster data operations with much less computing needed per query.
• Query execution: Firebolt’s query execution engine takes all the query optimization and combines it with vectorized processing, LLVM, and a host of other innovations to deliver really fast execution of standard Looker SQL.

How to learn more

Visit the Firebolt and Looker page to learn more about Firebolt and Looker integration, see Looker and Firebolt in action, or attend a Firebolt and Looker overview.

If you’re interested in how Appsflyer uses Looker and Firebolt, you read more about their story, or watch their webinar.

Or you can just do what Appsflyer and others did:

• Pick a Looker dashboard where you’d like to see better performance
• Register for a 1:1 session with a solution architect and our account team
• See what Firebolt can do for your specific use case

^{1. www.firebolt.io/resources/appsflyer-case-study}