Confetti cannons or fire extinguishers? Here’s how to secure cloud surprises

One of cybersecurity’s most hallowed mantras echoes throughout the industry, as if carried by a dark corridor in a lost city. “Shift left!” it thunders. “Get security involved early!” Yet many security professionals find themselves playing catch-up. Ignored for new initiatives, we're often only invited when the champagne's popped and the new product is about to hit production — a moment better suited for confetti cannons than fire extinguishers.

Cybersecurity professionals would love to be part of the celebration and blast off a few party poppers, too — but there’s no time for that after finding the lost cities of cloud deployed without oversight and outside of our tools and governance processes. Fire extinguishers it is, then.

Finding these previously-unknown environments can set off panic. We know that security vulnerabilities likely lurk in these environments and they must be fixed, and fast.

Getting started when security is invited late to the party is never easy, but the good news is that there is a playbook for exactly what to do in these situations. Following these eight tasks, in order, can help security teams understand and quickly bring projects they inherit running on Google Cloud under control.

As you go through these steps, keep in mind that they might break the project. A good rule of thumb is to make good-faith efforts to communicate with impacted users and stakeholders before each action you take.

Step 1: Get access to the project

You need access to the environment as an Organization Administrator. Try to find the owner in your organization who set up the project, and ask them to grant you the privileges you need.

• If you can’t, contact Google Cloud Support, explain the situation, and request access (and escalate) to authorized contacts be granted access.
• If the project isn’t part of an Organization, contact Google Cloud Support and let them know.
• It’s very possible Cloud Support already has a list of security recommendations for this environment. Google Cloud continuously monitors its platform for abuse and reports things we find to Essential Contacts if they can be identified. Be sure that they provide you with any other security advisories or problems that the Cloud Support team has separately discovered in the environment.
• Before you hang up, make sure they update their contact information for your organization with the latest POCs. It can help Cloud Support reach out to quickly in the case of future incidents or concerns.

Step 2: Bring the environment under control

Bring the project under control of the resource manager and isolate it.

• If you don’t have an Organization, create one. If your company uses Organizations, figure out which Organization this environment should belong to, and add it.
• If you have an Organization with projects, review or create your resource hierarchy to follow our Enterprise Security Foundations best practices.
• Develop a plan to migrate other existing folders and projects to your Organization.
• Identify and apply missing organization constraints to address compliance gaps and exposure of potentially misconfigured internet-facing services (such as publicly-accessible Cloud Storage buckets and applications),
• Assess and apply restrictions to isolate resources in these environments via Identity Access Management (IAM), network-based restrictions, and VPC Service Controls. The goal is to establish a “least privilege” policy restricting what principals can do in the project or new Organization they are now a part of.
• Migrate the environment to your properly secured and prepared Organization.
• Snapshot Virtual Machines (VMs) and move them to storage services for future investigation. Use object retention lock capabilities to ensure VMs cannot be modified after they are moved.

Step 3: Implement least privilege access for principals

Inspect Cloud Identity and address overly permissive users and roles.

• Using our guide for establishing appropriate administrative access to a new organization, ensure that all Google Cloud users are Managed Users in your Cloud Identity domain, synced from your existing Identity Provider. Users should have the least access necessary.
• Enforce the Domain Restricted Sharing Organization Policy so that only managed users in your organization (and other organizations you allowlist) can access the right resources.
• Check for non-Organization members who have IAM access to your resources using IAM Policy Analyzer. If there are legitimate users from trusted partner domains, add them to your Domain Restricted Sharing policy. If not, revoke their access.
• Eliminate the use of Basic Roles, like Editor and Owner, as much as possible. Redeploy policies with Google Cloud recommended policies for each service.

At this stage, the environment should be under your control and isolated from other projects. IAM should also be under more reasonable control. Now, security teams have a window of opportunity to assess what to do next.

Step 4: Enable MFA and address security hygiene opportunities

Enable MFA starting with administrative accounts and address any outstanding, high-risk vulnerabilities.

• Implement two-step authentication on all accounts with access to public cloud resources. If you must prioritize, start with accounts that have privileged access.
• Enable Security Command Center Premium, and then:
  • Use IAM Recommender to apply least-privilege to power users.
  • Patch vulnerable software and remediate vulnerabilities found by Security Health Analytics (also available in the Standard Tier of Security Command Center.)
  • Evaluate and confirm the appropriateness of any Sensitive Actions via Sensitive Action Service and restrict access accordingly.
  • Use other vulnerability-detection capabilities in SCC and remediate issues accordingly.
• Identify and remove all use of basic and primitive roles immediately. Redeploy policies with Google Cloud recommended policies for each service.

You can run the following steps in parallel.

Step 5: Conduct a compromise assessment

Take definitive steps to find and evict bad actors hiding in projects.

Conduct a formal Compromise Assessment of the environment. This review will provide reliable indication that threat actors don’t have control over the environment.

Step 6: Verify or enable logging, and centralize logs

Verify existing logging settings, enable the missing logs, and centralize into a SIEM, log management, or similar system.

• Review Cloud Audit Log and other logging settings for the environment, and implement logging based on Google Cloud best practices.
• Centralize the admin activity audit logs you need for security purposes from all projects in the org to a centralized project managed by the security team.
• Enable detections to run on the collected logs (use the blueprint as an example).

Step 7: Set up Cloud Asset Inventory

Generate an Inventory of Systems in the environment, and conduct a vulnerability and configuration-weakness assessment.

• Use Cloud Asset Inventory (CAI) to track all cloud assets in the environment. CAI offers a comprehensive view of cloud assets, including resource configurations and access control information.
• Use the Risk Manager assessment as a guide to uncover other areas of risky use of Google Cloud. Security teams should exercise their judgment to address the most pressing issues first. Prioritize and fix the riskiest issues based on SCC recommendations.

Step 8: Minimize use of service account keys

Find all service account keys in use, and transition to secure alternatives where possible.

Service accounts are integral for inter-application communication within cloud environments. However, the use of long-lived static keys for authentication poses a significant security risk, akin to leaving your home’s front door unlocked.

• Identification: Use CAI to identify all service account keys in the environment.
• Transition: Plan and execute a careful transition from static keys to short-lived, dynamic credentials. This process requires thorough testing to ensure applications continue to function as intended.
• Mitigation: If you still need service account keys in your environment, follow Google’s best practices.

Step 9: Find and secure valuable data

Find your most important data and apply security controls.

• Use our Sensitive Data Protection (SDP) service to locate sensitive data in your project. Use SDP data profiles to find most sensitive and available infoTypes.
  • Establish data-risk tiers: Classify data into different risk levels based on its sensitivity.
  • Eliminate unnecessary sensitive data: Identify and remove sensitive data that is not needed for business purposes.
  • Obfuscate sensitive data: Use techniques such as encryption, masking, and tokenization to protect sensitive data from unauthorized access.
  • Implement risk-appropriate data encryption: Encrypt sensitive data at rest and in transit using strong encryption algorithms and keys.
  • Scan for hard-coded credentials: Regularly scan systems and code for hard-coded credentials, such as passwords and API keys, and remove them.
  • Secure workflows: Analyze data workflows and implement appropriate security measures to protect sensitive data throughout its lifecycle.
  • Prioritize remediation on projects and folders where the most sensitive data resides, and where the data is of the highest levels of sensitivity.

Google Cloud has sophisticated features to help secure your projects and organizations. No two organizations’ needs are identical, even from project to project, so it’s important to know the above steps and practice applying them to different situations you might find yourself in. Incorporate the steps that work best for you into your new project provisioning and incident response plans. Importantly, be sure to follow CIS Google Cloud Benchmarks for running a secure configuration baseline over time.

The next time your organization is presented with a mysterious cloud project in the middle of a party, be ready to take these steps to secure these environments. With practice, they can help you trade those fire hoses and for the far more fun confetti cannons, and much more quickly at that. For more security guidance, visit our CISO Insights Hub.

Party on!