Opening doors, embracing change with cloud data warehouses

Moving from a traditional on-prem data warehouse platform to a cloud data warehouse, such as Google Cloud’s BigQuery, is about more than just adopting new technology. It also provides an opportunity to revisit existing practices and adopt new ways of working. We’ve noted some typical changes that teams undergo to support data warehouse migration to the cloud. 

To get a framework for change management and best practices for moving to the cloud, you can find more details in the Managing Change in the Cloud whitepaper.

People form the core of any change management initiative. For a data warehouse migration initiative, we’ll focus on three key stakeholders who are involved in day-to-day usage and management (though there may be other ancillary roles involved): 

• Data consumer—typically a data analyst, data scientist, or business user 

• Data enabler—typically a data engineer or ETL developer 

• Data administrator—typically a database/data warehouse administrator

Let's take a look at each stakeholder and see how their responsibilities change as their organization transitions from on-prem to cloud. 

Data consumer 

Data analysts and data scientists have a lot to gain from migrating to a cloud data warehouse— easier availability of new datasets, new algorithms to play with, and lower-latency access options in the cloud. Most often, during the migration, existing reporting applications and BI tools are initially kept as is to ensure minimal business disruption—making it relatively easy for data consumers to adopt the changes. Data consumers, however, will likely want to ensure that their existing reports and use cases are tested on the new platform. It is, therefore, advisable to require their participation in any data validation efforts. You might create a data validation team, for example. 

Post-migration, data consumers can find new ways of working once they have access to new datasets, like converting batch reporting into real-time dashboards, and finding ways to make machine learning a reality for the organization. New platform capabilities end up becoming the most important catalyst for change in this role. The more ambitious the data consumers are, the greater the change will be for other roles in the organization.  

Data enabler

The technical abilities of the data enabler are vital to how successfully cloud is adopted into the organization. These individuals own the data pipelines and are deeply involved in any reengineering efforts needed to migrate workloads to the cloud. The learning curve can be steep depending on the technology stack that’s adopted; proper resource planning will be crucial for these data enablers. Migration to the cloud in and of itself could be a long-term undertaking, so it’s worth considering taking the time to plan, re-tool and automate the migration. 

As data enablers are refactoring and redesigning data pipelines and working together with data consumers, they have plenty of opportunities to rethink how business processes should change to take advantage of real-time data ingestion, new data modeling techniques, or new persistent stores that cloud technologies offer. Data enablers might also revisit old asks that were not possible to deliver due to scale, data format complexity, or ETL complexity. Cloud technologies can be better suited to overcoming such challenges through capabilities such as storage and compute resource elasticity, support for numerous data format and storage options, as well as a wide variety of tools and libraries to process data. Beyond that, adopting a DevOps model, application containerization, or a serverless compute model are avenues that data enablers (along with data administrators) can explore to improve the speed with which changes are applied and new insights are delivered to users.

Data administrator 

Database administrators continue to play a vital role even if the enterprise data warehouse is hosted in the cloud. As owners of the platform, administrators must get acquainted with cloud technology capabilities at the earliest and drive the enterprise transformation with key stakeholders. As database administrators move data workloads into the cloud, there continue to be important focus areas: governance overview during and post migration, data integrity, and SLAs such as RPOs and RTOs. If the organization adopts a vendor-managed cloud data warehouse, database administrators will have fewer platform procurement and database-tuning tasks. But cost vs. performance optimization is still a critical area for administrators to be fully involved in. (Check out ESG’s report on how migrating to BigQuery can lower your three-year TCO up to 52%.) 

Database administrators are also freed up to gain a deeper understanding of the storage, network, and compute options that cloud offers and look for ways to optimize enterprise workloads in the cloud. Administrators can adopt new ways of working with machine learning notebook environments, new query tools, containerization, hybrid cloud, and more—there’s plenty to learn and optimize for! 

Organizations that adopt these new skills and processes have a greater chance of success in delivering business results in the cloud. For more information on how to streamline your enterprise migration path, check out our data warehouse migration guide and framework. You can also apply for our data warehouse migration offer for help with the full migration process.