Google Cloud recommendations for investigating and responding to the Apache “Log4j 2” vulnerability

Editor's note: This post was updated on 1/14/21 at 1:12pm PST.

In this post, we provide recommendations from the Google Cybersecurity Action Team and discuss Google Cloud and Chronicle solutions to help security teams to manage the risk of the Apache “Log4j 2” vulnerability (CVE-2021-44228 and CVE-2021-45046).

For the latest updates on our assessment of the potential impact of the vulnerability on Google Cloud products and services, please visit Google Cloud’s security advisory page.

Background 

The Apache Log4j 2 utility is an open source Apache framework that is a commonly used component for logging requests. On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j version 2.15 or below to be compromised and allow an attacker to execute arbitrary code on the vulnerable server. On December 10th, 2021, NIST published a critical CVE in the National Vulnerability Database identifying this as CVE-2021-44228. The official CVSS base severity score has been determined as a severity of 10. We strongly encourage customers who manage environments containing Log4j 2 to update to the latest version or take the mitigation actions outlined in this post. The Cybersecurity and Infrastructure Security Agency (CISA) released additional recommendations for immediate steps regarding this vulnerability.

Technical Details

The “Log4j 2 vulnerability” can be exploited by sending specially crafted log messages into Log4j 2. The attacker does this by leveraging the Java Naming and Directory Interface (JNDI) - which is a Java abstraction layer included by default in the Java SE Platform that allows a Java application to retrieve data from directory services (such as LDAP). JNDI uses HTTP URIs to resolve to a specific directory service  - and the URI can be adjusted as needed to resolve to the right directory location. Using Log4j 2, an attacker sending a log message with a specially crafted URI can cause the application to execute arbitrary code (such as by providing a Base64 encoded script in the path). 

This is due to specific behavior in Log4j 2 that allows for the input of variable data into the log (called Lookups). In a Lookup, the reference is queried and evaluated for input into the log. By using this feature during an exploit, the attacker uses the URI input to instruct Log4j 2 to resolve an object they input (such as an encoded script). This issue has been resolved in the latest version of Log4j 2. 

Updates and Recommendations to Identify, Detect & Protect

Customers should look to upgrade to the latest version of Log4j2 as soon as possible. Taking this action (and following additional advice from the Apache foundation) is your best immediate action. 

We strongly recommend Google Cloud Marketplace and Google Workspace customers review all third-party apps and solutions accessing your data and validate the security status of those apps and solutions with their respective vendors.

In addition to updating Log4j 2, some Google Cloud Security products can help detect and temporarily mitigate exploitation until the patch can be applied. Additionally, if you have third party WAF, IDS/IPS, and/or NDR solutions deployed in Google Cloud, please consult with your vendors for more guidance related to this vulnerability.

To identify potentially impacted systems we recommend the use of a vulnerability scanner. These tools have reported identifying the vulnerability referenced in National Vulnerability Database (NVD) and will help to locate impacted systems. 

Where possible, we recommend implementing all of the below for a layered defense in depth strategy.

Cloud Armor WAF Log4j 2 Detection and Blocking Rules

Relative to Log4j 2, Cloud Armor helps mitigate threats for applications or services behind external HTTP(S) load balancers. You can enable Cloud Armor through Cloud Console > Network Security, or via API. Cloud Armor’s WAF rules can be configured to detect, or detect and block requests. Cloud Armor customers can now deploy a new preconfigured WAF rule that will help detect and, optionally, block commonly attempted exploits of CVE-2021-44228 and CVE-2021-45046 while you are patching your systems. To learn more about addressing the Apache Log4j 2 vulnerability with Cloud Armor, please read this blog article.

Detecting vulnerable Log4j 2 with On-Demand Scanning

The Java scanning feature of Google Cloud On-Demand Scanning can be used to help identify Linux-based container images that use an impacted version of Log4j.  On-demand scanning is available at no charge for Google Cloud customers until January 1, 2022. Identifying vulnerable images early in the CI/CD process can help prevent them from being deployed in production. To use this functionality on either a locally stored container image or one stored in Artifact Registry or Google Container Registry, follow the instructions in the scanning images for Java vulnerabilities guide. Once results are returned, you can search for the text “CVE-2021-44228” using grep, a text editor, or your own script. Scoring reported in results is based on CVSS Version 2.0. If none of the results contain the CVE, On-Demand Scanning has not identified any affected packages.

Output indicating a vulnerable image

Results returned by On-Demand Scanning are formatted to the open-source Grafeas standard, and can be parsed in the same way as vulnerability scanning in Artifact Registry and Google Container Registry. Thus, any existing tooling that consumes the Grafeas format can be used with On-Demand Scanning. 

Binary Authorization Deploy Rules

For customers using Kubernetes Engine, CloudRun or Anthos clusters on VMware, Binary Authorization (BinAuthz) can enforce deploy policies that prevent packages with specific known vulnerabilities (i.e., Log4j) from being deployed. 

By using the vulnerability scanning features in Google Container Registry / Artifact Registry, customers can add a step in their CI/CD pipeline to only sign images that don't include the Log4j vulnerability, then configure a BinAuthz policy to block all deployments that are not properly signed. Detailed instructions on setting up vulnerability-based deploy enforcement can be found here.

Any violation of BinAuthz policy (in this case, blocked attempts to deploy vulnerable code) can be found in Cloud Audit Logs (instructions) for subsequent remediation.

Chronicle

Threat Hunting and investigation tools can be used to look at historical data and determine if exploitation was attempted - or can be used as vehicles for monitoring active exploitation.  Chronicle is Google’s threat hunting tool that provides extended event collection across cloud and on-prem (such as EDR logs, firewall logs, etc). If you are using Chronicle for log ingestion/SIEM and have historical event data stored (Chronicle retains 12 months of data by default) you can search for historical exploit attempts. Customers should search for events that contain “jndi” and a combination of strings that follow including “ldap”, “rmi”, “ldaps”, or “dns”, generated from HTTP requests, which correspond to possible Log4j 2 exploitation patterns. 

For example - this syntax could be used in a Yara-L rule:

Chronicle customers can also look at external communication events for low-prevalence destinations that could be an indication of impacted systems reaching out for remote code execution. More information can be found in Chronicle documentation here and in a detailed Chronicle blog post.

Cloud IDS Network-based Threat Detection

Cloud IDS has been updated to help detect common types of Log4j exploit attempts. These new detections are on by default for any existing or newly added deployments of Cloud IDS. Positive detections will appear in Cloud IDS alert logs, visible in terse mode in the Cloud IDS UI of Cloud Console, and in verbose mode in Cloud Logging, or by API. Cloud IDS is enabled and configured through Cloud Console > Network Security, or via API. Please see this blog for more information on IDS relative to both CVEs.

Security Command Center Premium

For customers using Security Command Center (SCC) Premium, we have released an active scan that detects DNS calls to known malicious sites associated with Log4j vulnerability abuse. In order to take advantage of this active scan, customers must (1) enable Web Security Scanner, Event Threat Detection, and Cloud DNS logs, and (2) grant Web Security Scanner permissions necessary to conduct scans on potentially vulnerable resources. Details about configuring Web Security Scanner are available here.  

We have also released passive detection rules in Event Threat Detection (ETD) and Security Health Analytics. The detector in Security Health Analytics identifies Dataproc images with subminor image version numbers equal or prior to 1.3.94, 1.4.76, 1.5.51, or 2.0.26. Images with these version numbers were affected by CVE-2021-44228 and CVE-2021-45046 and have since been replaced. Event Threat Detection can help detect inbound exploitation attempts and indicators of compromise as part of Malware: Bad IP and Malware: Bad Domain. These rules are enabled by default.

For customers running on GKE, the Container Threat Detection module in SCC detects execution of added binaries, libraries, and malicious bash scripting, giving defenders additional insights. We recommend enabling ETD and Container Threat Detection across your entire organization.

Cloud Audit Log Admin Activity Logs are enabled by default and used by SCC for threat detection. For detecting anomalous network activity, ETD requires that additional logging be enabled, including Cloud Load Balancing, Cloud DNS, and VPC Flow Logs. This page provides additional information on configuring logging for ETD.

Cloud Logging

You can use the Logs Explorer to help identify potential attacks on your service exploiting the Log4j 2 vulnerability, and to validate that mitigation techniques like Cloud Armor are configured correctly and blocking these attempts as expected.

If you have enabled logging of HTTP(S) Load Balancing for your service, you can check httpRequest fields with user generated content for potential exploits that have string tokens like ${jndi:ldap://.  

There are multiple variations on how this vulnerability is being exploited, and there are many ways to use unicode or escapes to avoid detection. This is an introduction to using a regex query to help detect some of the common exploits.Here is an example to consider:

The above query matches many of the obfuscated variations of the string "${jndi:" in HTTP Load Balancer request logs. You can use similar regular expressions to scan request logs in other services by changing the resource.type. The query may take a long time to complete if you are scanning a large volume of logs. To make your queries run faster, you can make use of indexed fields like resource.type, resource.labels, or logName to narrow your query to a set of specific services or log streams.

To determine if these exploit attempts reached your backend service, check the jsonPayload.statusDetails field in these request log entries to learn whether the request reached a resource. This is applicable for global external HTTP(S) Load Balancers logs. You can then validate whether you properly configured mitigating techniques like access control by Identity-Aware Proxy (IAP) or WAF Log4j 2 security policy rule by Google Cloud Armor as mentioned above.

Some possible values for jsonPayload.statusDetails to look for:

• denied_by_security_policy: request denied by Cloud Armor

• handled_by_identity_aware_proxy: request denied  by Identity-Aware Proxy

• response_sent_by_backend: request allowed and handled by backend service

Detecting matching log entries handled by your application does not always indicate a successful compromise. It may indicate that someone is probing to exploit the vulnerability within your project or workload. There could also be false positives if your application uses patterns like "${jndi:" in the http request fields. 

Cloud Logging query results only include logs that have already been ingested into Cloud Logging and are also within the user specified retention limits. While most Google Cloud services have logs enabled by default, logs that were disabled or excluded will not be included in this search. If you are using an HTTP(S) Load Balancer, logging needs to be enabled for the request logs to be available in Cloud Logging. Similarly, if you have web servers like Apache or NGINX running on a VM, but have not installed the logging agent, those logs will not be accessible within Cloud Logging.

More details on how to use Cloud Logging for this kind of use case can be found here.

Apigee

Apigee provides tools that can help detect and block log4j 2 exploit attempts through your APIs. Detecting potential attacks can be done using custom reports in API Analytics. Edge API Analytics collects metadata on traffic that flows through your API proxies. The report can show patterns in your analytics records that indicate exploit attempts. Matching records don't mean there has been a successful compromise, but may mean that someone is attempting to exploit the vulnerability through your APIs. A detailed explanation on how to configure the report can be found here. Blocking log4j 2 exploit attempts through your Apigee-managed APIs can be done using Raise Fault policy and an appropriate regular expression that matches known attack patterns. The regular expression and the payload location are configurable and you will be able to tune them as more information is revealed over time. A detailed explanation on how to configure this new policy can be found here.

Additional options for preventing vulnerable code in new deployments

Detecting vulnerable code may help reduce the impact of Log4j before it becomes a problem in new production deployments. We have partnered with Code Intelligence to provide fuzzing for Log4j, as part of OSS-Fuzz. Code Intelligence also improved its Jazzer fuzzing engine to make it capable of detecting remote JNDI lookups. Learn more in this blog.

Open source remediation

We encourage the open source community to continue to fix packages impacted by the Log4j vulnerability by enabling automated dependency updates and improving security. Most artifacts that depend on Log4j do so indirectly. The deeper a vulnerability is within a dependency chain, the more steps are required for it to be fixed. Improvements such as these could qualify for financial rewards from the Secure Open Source Rewards program. You can explore your package dependencies and their vulnerabilities by using Open Source Insights. Read more about Google’s analysis of ecosystem impact and steps maintainers can take in this blog.

We will continue to actively monitor this event and will provide updates to this blog post on relevant mitigation steps and detection mechanisms. Please visit our security advisory page for updates on our Google Cloud security assessment.