Troubleshooting Google Cloud Armor problems

Use these instructions to troubleshoot issues with your Google Cloud Armor security policies.

Traffic is allowed despite a deny rule configured in the Google Cloud Armor security policy

  1. Make sure that the Google Cloud Armor security policy is attached to a target backend service. For example, the following command describes all data associated with the backend service my-backend. The results returned should include the name of the Google Cloud Armor security policy associated with this backend service.

    gcloud compute backend-services describe my-backend
    
  2. Review the HTTP(S) logs to determine which policy and rule were matched for your traffic along with the associated action. Use Logging to view the logs.

    Here is a sample log of an allowed request with the interesting fields boldfaced. Check for the following fields and make sure they match the rule you configured to deny the traffic:

    • configuredAction should match the action configured in the rule
    • outcome should match configuredAction above
    • priority should match the priority number of the rule
    • name should match the name of the Google Cloud Armor security policy attached to this backend service
    httpRequest:
     remoteIp: 104.133.0.95
     requestMethod: GET
     requestSize: '801'
     requestUrl: http://74.125.67.38/
     responseSize: '246'
     serverIp: 10.132.0.4
     status: 200
     userAgent: curl/7.35.0
       insertId: ajvis5ev4i60
       internalId:
         projectNumber: '895280006100'
       jsonPayload:
         '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
         **enforcedSecurityPolicy:
           configuredAction: ACCEPT
           name: mydev-policy-log-test1
           outcome: ACCEPT
           priority: 2147483647**
         statusDetails: response_sent_by_backend
       logName: projects/mydev-staging/logs/requests
       resource:
         labels:
           backend_service_name: ''
           forwarding_rule_name: mydev-forwarding-rule
           project_id: mydev-staging
           target_proxy_name: mydev-target-http-proxy
           url_map_name: mydev-url-map
           zone: global
         type: http_load_balancer
       severity: INFO
       timestamp: '2017-04-18T18:57:05.845960288Z'
    
  3. Review the hierarchy of rules to ensure that the correct rule is matched. It is possible that a higher priority rule with an allow action is matching your traffic. Use the describe command on the security-policies in the gcloud command-line tool to see the contents of the Google Cloud Armor security policy.

    For example, the following example shows how a higher-priority allow rule (at priority 100) matches traffic coming from 1.2.3.4 IP address, preventing the lower-priority (at priority 200) deny rule from triggering and blocking the traffic.

    gcloud compute security-policies describe my-policy
    
    creationTimestamp: '2017-04-18T14:47:58.045-07:00
    description: ''
    fingerprint: Yu5spBjdoC0=
    id: '2560355463394441057'
    kind: compute#securityPolicy
    name: my-policy
    rules:
    - action: allow
      description: allow high priority rule
      kind: compute#securityPolicyRule
      match:
        srcIpRanges:
        - '1.2.3.4/32'
      preview: false
      priority: 100
    - action: deny
      description: deny lower priority rule
      kind: compute#securityPolicyRule
      match:
        srcIpRanges:
        - '1.2.3.0/24
      preview: false
      priority: 200
    - action: deny
      description: default rule
      kind: compute#securityPolicyRule
      match:
        srcIpRanges:
        - '*'
      preview: false
      priority: 2147483647
      selfLink: http://www.googleapis.com/compute/v1/projects/bigclustertestdev0-devconsole/global/securityPolicies/sp
    

Pre-configured rule returns false positives

XSS and SQLi detection are based on static signature matching on HTTP request headers and other L7 parameters. These regular expression patterns are prone to false positives. You can use the pre-configured rule for XSS and SQLi detection in preview mode and then check the log for any false positives.

If you find a false positive, you can compare the traffic content with the ModSecurity CRS rules. If the rule is invalid or not relevant, disable it by using the evaluatePreconfiguredExpr expression and specify the rule's ID in the exclude ID list argument.

After reviewing the logs and removing all false positives, disable the preview mode.

To add a pre-configured rule in preview mode:

  1. Create a security policy with the pre-configured expression set in preview mode:

    gcloud compute security-policies rules create 1000
       --security-policy my-policy
       --expression "evaluatePreconfiguredExpr('xss-stable')"
       --action deny-403
       --preview
    
  2. Review the HTTP(S) logs for HTTP request fields such as url and cookie. For example, the requestUrl compares positively to the ModSecurity CRS rule ID 941180:

    httpRequest:
      remoteIp: 104.133.0.95
      requestMethod: GET
      requestSize: '801'
      requestUrl: http://74.125.67.38/foo?document.cookie=1010"
      responseSize: '246'
      serverIp: 10.132.0.4
      status: 200
      userAgent: curl/7.35.0
    insertId: ajvis5ev4i60
    internalId:
      projectNumber: '895280006100'
    jsonPayload:
      '@type': type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
      enforcedSecurityPolicy:
        configuredAction: ACCEPT
        name: my-policy
        outcome: ACCEPT
        priority: 2147483647
        preconfiguredExprIds: [ 'owasp-crs-v030001-id941180-xss' ]
      statusDetails: response_sent_by_backend
    logName: projects/mydev-staging/logs/requests
    resource:
      labels:
        backend_service_name: ''
        forwarding_rule_name: mydev-forwarding-rule
        project_id: mydev-staging
        target_proxy_name: mydev-target-http-proxy
        url_map_name: mydev-url-map
        zone: global
      type: http_load_balancer
    severity: INFO
    timestamp: '2017-04-18T18:57:05.845960288Z'
    
  3. Exclude the ModSecurity CRS rule ID 941180 by updating the rule in the Google Cloud Armor security policy.

    gcloud compute security-policies rules update 1000 \
       --security-policy my-policy \
       --expression "evaluatePreconfiguredExpr('xss-stable', ['owasp-crs-v030001-id941180-xss'])" \
       --action deny-403 \
       --preview
    
  4. Review the logs again and then disable preview mode to implement the rule.

Clients with denied signatures are not blocked or denied

If you are using Google Cloud Armor with Cloud CDN, security policies are enforced only for requests for dynamic content, cache misses, or other requests that are destined for the CDN origin server. Cache hits are served even if the downstream Google Cloud Armor security policy would prevent that request from reaching the CDN origin server.

Issues with named IP address lists

This section provides information on resolving problems with named IP address lists.

IP addresses within a named IP address list

The IP addresses in the lists always match the IP addresses in the provider websites listed in the Google Cloud Armor named IP address lists guide. If you have questions about the lists, contact the Cloud Support team.

IP addresses within a named IP address list are stale in Google Cloud Armor

Google Cloud Armor synchronizes its lists with IP address list providers daily. It is possible to have stale data that lags a few hours or a day behind the data at a provider. However, if you believe that the stale data lags more than expected, for example, more than a week, contact the Cloud Support team.

Difficulty creating a security policy that references a named IP address list

If you try to create a security police that references a named IP list, and the command fails, the error you see looks something like this:

gcloud beta compute security-policies rules create 750 \
    --security-policy my \
    --expression "evaluatePreconfiguredExpr('sourceiplist-abc')" \
    --action "allow"
ERROR: (gcloud.beta.compute.security-policies.rules.create) Could not fetch resource:
 - Invalid value for field 'resource.match.expr': '{  "expression": "evaluatePreconfiguredExpr(\u0027sourceiplist-abc\u0027)"}'. Error parsing Cloud Armor rule matcher expression: sourceiplist-abc is not a valid preconfigured expression set.

Make sure that the particular provider is supported and that the name of the IP address list is given correctly. You can check this by listing the current pre-configured expression sets using this gcloud command:

gcloud beta compute security-policies list-preconfigured-expression-sets

Traffic is blocked despite a pre-configured rule for a named IP address allow list

You might find that traffic is blocked from an IP address that is on a named IP address list. Make sure that traffic is coming from an IP address that is on a named IP address allow list. Second, check whether there are other security rules with a higher priority that can block the traffic. Contact the Cloud Support team if you still see the issue.

First, make sure that the security policy is attached to the correct backend service:

gcloud compute backend-services describe my-backend-service

Next, check the rules that are in the security policy. For example:

 gcloud compute security-policies describe my-policy
---
…
name: my-policy
rules:
- action: allow
  description: allow fastly ip addresses
  kind: compute#securityPolicyRule
  match:
    expr:
      expression: evaluatePreconfiguredExpr('sourceiplist-fastly')
  preview: false
  priority: 100
- action: deny(403)
  description: Default rule, higher priority overrides it
  kind: compute#securityPolicyRule
  match:
    config:
      srcIpRanges:
      - '*'
    versionedExpr: SRC_IPS_V1
  preview: false
  priority: 2147483647
- action: deny(404)
  description: deny xyz referer
  kind: compute#securityPolicyRule
  match:
    expr:
      expression: request.headers['Referer'].contains('xyz')
  preview: false
  priority: 50
…

The above security policy contains three rules: a default deny rule, an allow rule for Fastly's IP addresses, and a deny rule for a referrer that contains 'xyz'. Their respective priorities are listed as well.

Next, review the logs to determine which rule is matched to your traffic and the associated action. For information on logging, see Viewing logs (Classic).

The following is an example of a log:

 httpRequest: {
    referer: "http://www.xyz.com/"
    remoteIp: "23.230.32.10"
    requestMethod: "HEAD"
    requestSize: "79"
    requestUrl: "http://www.abc.com/"
    responseSize: "258"
    status: 404
    userAgent: "Mozilla/5.0"
  }
  …
  jsonPayload: {
    @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
    enforcedSecurityPolicy: {
      configuredAction: "DENY"
      name: "my-policy"
      outcome: "DENY"
      priority: 50
    }
    statusDetails: "denied_by_security_policy"
  }
  …

From the above log, the request is coming from 23.230.32.10, which is covered by Fastly's public IP address list, as viewed at https://api.fastly.com/public-ip-list. However, the request is matched with a deny rule of a higher priority of 50. Comparing this with what's in the security policy, the rule corresponds to the deny rule for a referer that contains xyz. Because the request has a referer of http://www.xyz.com/, the security rule enforcement is working correctly.

Issues with Security Command Center findings

The Google Cloud Armor card does not appear in the Security Command Center

Enable Google Cloud Armor findings in the Security Command Center interface.

Findings from Google Cloud Armor do not appear in Security Command Center

  • If findings from Google Cloud Armor do not appear in Security Command Center, traffic to the backend services might not meet the criteria for raising a finding.
  • For questions on traffic volume to your backends, check request statistics in the Cloud Monitoring dashboards under the Network Security Policy resource.

Findings are too noisy

Contact the Cloud Support team for assistance with this issue.