Use these instructions to troubleshoot issues with your Google Cloud Armor security policies.
General issues
Traffic is allowed despite a deny rule configured in the Google Cloud Armor security policy
To fix this, follow these steps:
Make sure that the Google Cloud Armor security policy is attached to a target backend service. For example, the following command describes all data associated with the backend service
BACKEND
. The results returned should include the name of the Google Cloud Armor security policy associated with this backend service.gcloud compute backend-services describe BACKEND
Review the HTTP(S) logs to determine which policy and rule were matched for your traffic along with the associated action. To view the logs, use Cloud Logging.
The following is a sample log of an allowed request with the interesting fields highlighted. Check for the following fields and make sure that they match the rule that you configured to deny the traffic:
configuredAction
should match the action configured in the rule.name
should match the name of the Google Cloud Armor security policy attached to this backend service.outcome
should matchconfiguredAction
.priority
should match the priority number of the rule.
httpRequest: remoteIp: 104.133.0.95 requestMethod: GET requestSize: '801' requestUrl: http://74.125.67.38/ responseSize: '246' serverIp: 10.132.0.4 status: 200 userAgent: curl/7.35.0 insertId: ajvis5ev4i60 internalId: projectNumber: '895280006100' jsonPayload: '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry enforcedSecurityPolicy: configuredAction: ACCEPT name: mydev-policy-log-test1 outcome: ACCEPT priority: 2147483647 statusDetails: response_sent_by_backend logName: projects/mydev-staging/logs/requests resource: labels: backend_service_name: BACKEND_SERVICE_NAME forwarding_rule_name: FORWARDING_RULE_NAME project_id: PROJECT_ID target_proxy_name: TARGET_HTTP_PROXY_NAME url_map_name: URL_MAP_NAME zone: global type: http_load_balancer severity: INFO timestamp: '2017-04-18T18:57:05.845960288Z'
Review the hierarchy of rules to ensure that the correct rule is matched. It is possible that a higher priority rule with an allow action is matching your traffic. Use the
describe
command on thesecurity-policies
in thegcloud
command-line tool to see the contents of the Google Cloud Armor security policy.For example, the following example shows how a higher priority allow rule (at priority 100) matches traffic coming from the 1.2.3.4 IP address, preventing the lower priority deny rule (at priority 200) from triggering and blocking the traffic.
gcloud compute security-policies describe POLICY_NAME
Output:
creationTimestamp: '2017-04-18T14:47:58.045-07:00 description: '' fingerprint: Yu5spBjdoC0= id: '2560355463394441057' kind: compute#securityPolicy name: POLICY_NAME rules: -action: allow description: allow high priority rule kind: compute#securityPolicyRule match: srcIpRanges: -'1.2.3.4/32' preview: false priority: 100 -action: deny description: deny lower priority rule kind: compute#securityPolicyRule match: srcIpRanges: -'1.2.3.0/24 preview: false priority: 200 -action: deny description: default rule kind: compute#securityPolicyRule match: srcIpRanges: -'*' preview: false priority: 2147483647 selfLink: http://www.googleapis.com/compute/v1/projects/bigclustertestdev0-devconsole/global/securityPolicies/sp
Preconfigured rule returns false positives
XSS and SQLi detection are based on static signature matching on HTTP request headers and other L7 parameters. These regular expression patterns are prone to false positives. You can use the preconfigured rule for XSS and SQLi detection in preview mode and then check the log for any false positives.
If you find a false positive, you can compare the traffic content with the
ModSecurity CRS rules.
If the rule is invalid or not relevant, disable it by using the
evaluatePreconfiguredExpr
expression, and specify the rule's ID in the
exclude ID list
argument.
After reviewing the logs and removing all false positives, disable the preview mode.
To add a preconfigured rule in preview mode:
Create a security policy with the preconfigured expression set in preview mode:
gcloud compute security-policies rules create 1000 --security-policy POLICY_NAME --expression "evaluatePreconfiguredExpr('xss-stable')" --action deny-403 --preview
Review the HTTP(S) logs for HTTP request fields such as
url
andcookie
. For example, therequestUrl
compares positively to the ModSecurity CRS rule ID 941180:httpRequest: remoteIp: 104.133.0.95 requestMethod: GET requestSize: '801' requestUrl: http://74.125.67.38/foo?document.cookie=1010" responseSize: '246' serverIp: 10.132.0.4 status: 200 userAgent: curl/7.35.0 insertId: ajvis5ev4i60 internalId: projectNumber: '895280006100' jsonPayload: '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry enforcedSecurityPolicy: configuredAction: ACCEPT name: POLICY_NAME outcome: ACCEPT priority: 2147483647 preconfiguredExprIds: [ 'owasp-crs-v030001-id941180-xss' ] statusDetails: response_sent_by_backend logName: projects/mydev-staging/logs/requests resource: labels: backend_service_name: BACKEND_SERVICE forwarding_rule_name: mydev-forwarding-rule project_id: mydev-staging target_proxy_name: mydev-target-http-proxy url_map_name: mydev-url-map zone: global type: http_load_balancer severity: INFO timestamp: '2017-04-18T18:57:05.845960288Z'
Exclude the ModSecurity CRS rule ID 941180 by updating the rule in the Google Cloud Armor security policy:
gcloud compute security-policies rules update 1000 \ --security-policy POLICY_NAME \ --expression "evaluatePreconfiguredExpr('xss-stable', ['owasp-crs-v030001-id941180-xss'])" \ --action deny-403 \ --preview
Review the logs again and then disable preview mode to implement the rule.
Clients with denied signatures are not blocked or denied
If you are using Google Cloud Armor with Cloud CDN, security policies are enforced only for requests for dynamic content, cache misses, or other requests that are destined for the CDN origin server. Cache hits are served even if the downstream Google Cloud Armor security policy would prevent that request from reaching the CDN origin server.
Issues with named IP address lists
This section provides information for resolving problems with named IP address lists.
IP addresses within a named IP address list
The IP addresses in the lists always match the IP addresses in the provider websites listed in the Google Cloud Armor named IP address lists guide. If you have questions about the lists, contact the Cloud Support team.
IP addresses within a named IP address list are stale in Google Cloud Armor
Google Cloud Armor synchronizes its lists with IP address list providers daily. It is possible to have stale data that lags a few hours or a day behind the data at a provider. However, if you believe that the stale data lags more than expected, for example, more than a week, contact the Cloud Support team.
Difficulty creating a security policy that references a named IP address list
You might try to create a security policy that references a named IP address list, using a command such as this:
gcloud compute security-policies rules create 750 \ --security-policy my \ --expression "evaluatePreconfiguredExpr('sourceiplist-abc')" \ --action "allow"
If the command fails, the error that you see looks something like this:
ERROR: (gcloud.compute.security-policies.rules.create) Could not fetch resource: - Invalid value for field 'resource.match.expr': '{ "expression": "evaluatePreconfiguredExpr(\u0027sourceiplist-abc\u0027)"}'. Error parsing Cloud Armor rule matcher expression: sourceiplist-abc is not a valid preconfigured expression set.
Make sure that the particular provider is supported, and that the name of the IP
address list is given correctly. You can check this by using the following
gcloud
command to list the current preconfigured expression sets:
gcloud compute security-policies list-preconfigured-expression-sets
Traffic is blocked despite a preconfigured rule for a named IP address allowlist
You might find that traffic is blocked from an IP address that is on a named IP address list:
Make sure that traffic is coming from an IP address that is on a named IP address allowlist.
Check whether there are other security rules with a higher priority that can block the traffic. If you still see the issue, contact the Cloud Support team.
Make sure that the security policy is attached to the correct backend service:
gcloud compute backend-services describe BACKEND_SERVICE
Check the rules that are in the security policy. For example:
gcloud compute security-policies describe POLICY_NAME
The command returns information similar to the following:
--- … name: POLICY_NAME rules: -action: allow description: allow fastly ip addresses kind: compute#securityPolicyRule match: expr: expression: evaluatePreconfiguredExpr('sourceiplist-fastly') preview: false priority: 100 -action: deny(403) description: Default rule, higher priority overrides it kind: compute#securityPolicyRule match: config: srcIpRanges: -'*' versionedExpr: SRC_IPS_V1 preview: false priority: 2147483647 -action: deny(404) description: deny xyz referer kind: compute#securityPolicyRule match: expr: expression: request.headers['Referer'].contains('xyz') preview: false priority: 50 …
The preceding security policy contains three rules: a default deny rule, an allow rule for Fastly's IP addresses, and a deny rule for a referrer that contains
xyz
. Their respective priorities are also listed.Review the logs to determine which rule is matched to your traffic and the associated action. For information about logging, see Viewing logs in the Legacy Logs Viewer for the Cloud Logging API.
The following is an example of a log:
httpRequest: { referer: "http://www.xyz.com/" remoteIp: "23.230.32.10" requestMethod: "HEAD" requestSize: "79" requestUrl: "http://www.abc.com/" responseSize: "258" status: 404 userAgent: "Mozilla/5.0" } … jsonPayload: { @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry" enforcedSecurityPolicy: { configuredAction: "DENY" name: "POLICY_NAME" outcome: "DENY" priority: 50 } statusDetails: "denied_by_security_policy" } …
From the preceding log, the request is coming from
23.230.32.10
, which is covered by Fastly's public IP address list. However, the request is matched with a deny rule of a higher priority of 50. Comparing this with what's in the security policy, the rule corresponds to the deny rule for a referer that containsxyz
. Because the request has a referer ofhttp://www.xyz.com/
, the security rule enforcement is working correctly.
Issues with Security Command Center findings
The Google Cloud Armor card does not appear in the Security Command Center
Enable Google Cloud Armor findings in the Security Command Center interface.
Findings from Google Cloud Armor do not appear in the Security Command Center
If findings from Google Cloud Armor do not appear in the Security Command Center, traffic to the backend services might not meet the criteria for raising a finding.
For questions about traffic volume to your backends, check request statistics in the Cloud Monitoring dashboards under the Network Security Policies.
Findings are too noisy
For assistance with this issue, contact the Cloud Support team.