Customizing match likelihood

Using hotword rules, you can further extend built-in and custom infoType detectors with powerful context rules. A hotword rule instructs Cloud DLP to adjust the likelihood of a finding, depending on whether a hotword occurs near that finding. A hotword rule is a kind of inspection rule, which is specified in rule sets. Each rule is applied to a set of built-in or custom infoTypes.

Anatomy of a hotword rule

An infoType detector can have zero or more hotword rules. In your inspection configuration, you define each HotwordRule object inside the rules array, as follows:

"rules":[
  {
    "hotwordRule":{
      "hotwordRegex":{
        "pattern":"REGEX_PATTERN"
      },
      "proximity":{
        "windowAfter":"NUM_CHARS_TO_CONSIDER_AFTER_FINDING",
        "windowBefore":"NUM_CHARS_TO_CONSIDER_BEFORE_FINDING"
      }
      "likelihoodAdjustment":{
        "fixedLikelihood":"LIKELIHOOD_VALUE"
             -- OR --
        "relativeLikelihood":"LIKELIHOOD_ADJUSTMENT"
      },
    }
  },
  ...
]

Replace the following:

• REGEX_PATTERN: a regular expression (Regex object) that defines what qualifies as a hotword.
• NUM_CHARS_TO_CONSIDER_AFTER_FINDING: a range of characters after the finding. Cloud DLP analyzes this range to determine whether a hotword occurs near the finding.

• NUM_CHARS_TO_CONSIDER_BEFORE_FINDING: a range of characters before the finding. Cloud DLP analyzes this range to determine whether a hotword occurs near the finding.

• LIKELIHOOD_VALUE: a fixed Likelihood level to set the finding to.

• LIKELIHOOD_ADJUSTMENT: a number that indicates how much Cloud DLP must increase or decrease the likelihood of the finding. A positive integer increases the likelihood level, and a negative integer decreases it. For example, if a finding would be POSSIBLE without the detection rule and relativeLikelihood is 1, then the finding is upgraded to LIKELY. If relativeLikelihood is -1, then the finding is downgraded to UNLIKELY. Likelihood can never drop lower than VERY_UNLIKELY or exceed VERY_LIKELY. In these cases, the likelihood level remains the same. For example, if the base likelihood is VERY_LIKELY and the relativeLikelihood is 1, the final likelihood remains to be VERY_LIKELY.

Hotword example: Match medical record numbers

Suppose you want to detect a custom infoType such as a medical record number (MRN) in the form "###-#-#####". Also, you want Cloud DLP to increase the match likelihood of each finding that follows the hotword "MRN".

Example values:

• 123-4-56789 would match as POSSIBLE.
• MRN 123-4-56789 would match as VERY_LIKELY.

The following JSON example and code snippets show you how to configure the hotword rule. This example uses a custom regular expression detector.

In this example, note the following:

• The request defines the C_MRN custom infoType, which is a detector for any string that matches the regular expression [0-9]{3}-[0-9]{1}-[0-9]{5}.
• The regular expression (?i)(mrn|medical)(?-i) defines the hotword. Cloud DLP searches for this hotword within the range of characters defined in the proximity field.
• For each C_MRN finding that has a hotword within the set proximity, Cloud DLP sets the likelihood level to VERY_LIKELY.

Protocol

See the JSON quickstart for more information about using the DLP API with JSON.

Note: The following example requires an OAuth 2.0 access token.

HTTP method and URL:

POST https://dlp.googleapis.com/v2/projects/PROJECT_ID/content:inspect

Replace PROJECT_ID with the project ID.

JSON input:

{
  "item":{
    "value":"Patient's MRN 444-5-22222 and just a number 333-2-33333"
  },
  "inspectConfig":{
    "customInfoTypes":[
      {
        "infoType":{
          "name":"C_MRN"
        },
        "regex":{
          "pattern":"[0-9]{3}-[0-9]{1}-[0-9]{5}"
        },
        "likelihood":"POSSIBLE",
      }
    ],
    "ruleSet":[
        {
        "infoTypes": [{"name" : "C_MRN"}],
        "rules":[
          {
            "hotwordRule":{
              "hotwordRegex":{
                "pattern":"(?i)(mrn|medical)(?-i)"
              },
              "likelihoodAdjustment":{
                "fixedLikelihood":"VERY_LIKELY"
              },
              "proximity":{
                "windowBefore":10
              }
            }
          }
        ]
      }
    ]
  }
}

JSON output (abbreviated):

{
  "result": {
    "findings": [
      {
        "infoType": {
          "name": "C_MRN"
        },
        "likelihood": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "start": "14",
            "end": "25"
          },
          "codepointRange": { ... }
        }
      },
      {
        "infoType": {
          "name": "C_MRN"
        },
        "likelihood": "POSSIBLE",
          "byteRange": {
            "start": "44",
            "end": "55"
          },
          "codepointRange": { ... }
        }
      }
    ]
  }
}

The output shows that Cloud DLP correctly identified the medical record number using the C_MRN custom infoType detector. Further, because of the context matching in the hotword rule, Cloud DLP assigned the first result—which had an MRN within the set proximity—a likelihood of VERY_LIKELY, as configured. The second finding lacked the context, so the likelihood stayed at POSSIBLE.

Java

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

import com.google.cloud.dlp.v2.DlpServiceClient;
import com.google.privacy.dlp.v2.ByteContentItem;
import com.google.privacy.dlp.v2.ByteContentItem.BytesType;
import com.google.privacy.dlp.v2.ContentItem;
import com.google.privacy.dlp.v2.CustomInfoType;
import com.google.privacy.dlp.v2.CustomInfoType.DetectionRule.HotwordRule;
import com.google.privacy.dlp.v2.CustomInfoType.DetectionRule.LikelihoodAdjustment;
import com.google.privacy.dlp.v2.CustomInfoType.DetectionRule.Proximity;
import com.google.privacy.dlp.v2.CustomInfoType.Regex;
import com.google.privacy.dlp.v2.Finding;
import com.google.privacy.dlp.v2.InfoType;
import com.google.privacy.dlp.v2.InspectConfig;
import com.google.privacy.dlp.v2.InspectContentRequest;
import com.google.privacy.dlp.v2.InspectContentResponse;
import com.google.privacy.dlp.v2.InspectionRule;
import com.google.privacy.dlp.v2.InspectionRuleSet;
import com.google.privacy.dlp.v2.Likelihood;
import com.google.privacy.dlp.v2.LocationName;
import com.google.protobuf.ByteString;
import java.io.IOException;

public class InspectWithHotwordRules {

  public static void main(String[] args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String textToInspect = "Patient's MRN 444-5-22222 and just a number 333-2-33333";
    String customRegexPattern = "[1-9]{3}-[1-9]{1}-[1-9]{5}";
    String hotwordRegexPattern = "(?i)(mrn|medical)(?-i)";
    inspectWithHotwordRules(projectId, textToInspect, customRegexPattern, hotwordRegexPattern);
  }

  // Inspects a BigQuery Table
  public static void inspectWithHotwordRules(
      String projectId, String textToInspect, String customRegexPattern, String hotwordRegexPattern)
      throws IOException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (DlpServiceClient dlp = DlpServiceClient.create()) {
      // Specify the type and content to be inspected.
      ByteContentItem byteItem =
          ByteContentItem.newBuilder()
              .setType(BytesType.TEXT_UTF8)
              .setData(ByteString.copyFromUtf8(textToInspect))
              .build();
      ContentItem item = ContentItem.newBuilder().setByteItem(byteItem).build();

      // Specify the regex pattern the inspection will look for.
      Regex regex = Regex.newBuilder().setPattern(customRegexPattern).build();

      // Construct the custom regex detector.
      InfoType infoType = InfoType.newBuilder().setName("C_MRN").build();
      CustomInfoType customInfoType =
          CustomInfoType.newBuilder().setInfoType(infoType).setRegex(regex).build();

      // Specify hotword likelihood adjustment.
      LikelihoodAdjustment likelihoodAdjustment =
          LikelihoodAdjustment.newBuilder().setFixedLikelihood(Likelihood.VERY_LIKELY).build();

      // Specify a window around a finding to apply a detection rule.
      Proximity proximity = Proximity.newBuilder().setWindowBefore(10).build();

      // Construct hotword rule.
      HotwordRule hotwordRule =
          HotwordRule.newBuilder()
              .setHotwordRegex(Regex.newBuilder().setPattern(hotwordRegexPattern).build())
              .setLikelihoodAdjustment(likelihoodAdjustment)
              .setProximity(proximity)
              .build();

      // Construct rule set for the inspect config.
      InspectionRuleSet inspectionRuleSet =
          InspectionRuleSet.newBuilder()
              .addInfoTypes(infoType)
              .addRules(InspectionRule.newBuilder().setHotwordRule(hotwordRule))
              .build();

      // Construct the configuration for the Inspect request.
      InspectConfig config =
          InspectConfig.newBuilder()
              .addCustomInfoTypes(customInfoType)
              .setIncludeQuote(true)
              .setMinLikelihood(Likelihood.POSSIBLE)
              .addRuleSet(inspectionRuleSet)
              .build();

      // Construct the Inspect request to be sent by the client.
      InspectContentRequest request =
          InspectContentRequest.newBuilder()
              .setParent(LocationName.of(projectId, "global").toString())
              .setItem(item)
              .setInspectConfig(config)
              .build();

      // Use the client to send the API request.
      InspectContentResponse response = dlp.inspectContent(request);

      // Parse the response and process results
      System.out.println("Findings: " + response.getResult().getFindingsCount());
      for (Finding f : response.getResult().getFindingsList()) {
        System.out.println("\tQuote: " + f.getQuote());
        System.out.println("\tInfo type: " + f.getInfoType().getName());
        System.out.println("\tLikelihood: " + f.getLikelihood());
      }
    }
  }
}

Python

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

def inspect_with_medical_record_number_w_custom_hotwords(
    project,
    content_string,
):
    """Uses the Data Loss Prevention API to analyze string with medical record
       number custom regex detector, with custom hotwords rules to boost finding
       certainty under some circumstances.

    Args:
        project: The Google Cloud project id to use as a parent resource.
        content_string: The string to inspect.

    Returns:
        None; the response from the API is printed to the terminal.
    """

    # Import the client library.
    import google.cloud.dlp

    # Instantiate a client.
    dlp = google.cloud.dlp_v2.DlpServiceClient()

    # Construct a custom regex detector info type called "C_MRN",
    # with ###-#-##### pattern, where each # represents a digit from 1 to 9.
    # The detector has a detection likelihood of POSSIBLE.
    custom_info_types = [
        {
            "info_type": {"name": "C_MRN"},
            "regex": {"pattern": "[1-9]{3}-[1-9]{1}-[1-9]{5}"},
            "likelihood": google.cloud.dlp_v2.Likelihood.POSSIBLE,
        }
    ]

    # Construct a rule set with hotwords "mrn" and "medical", with a likelohood
    # boost to VERY_LIKELY when hotwords are present within the 10 character-
    # window preceding the PII finding.
    hotword_rule = {
        "hotword_regex": {"pattern": "(?i)(mrn|medical)(?-i)"},
        "likelihood_adjustment": {
            "fixed_likelihood": google.cloud.dlp_v2.Likelihood.VERY_LIKELY
        },
        "proximity": {"window_before": 10},
    }

    rule_set = [
        {"info_types": [{"name": "C_MRN"}], "rules": [{"hotword_rule": hotword_rule}]}
    ]

    # Construct the configuration dictionary with the custom regex info type.
    inspect_config = {
        "custom_info_types": custom_info_types,
        "rule_set": rule_set,
        "include_quote": True,
    }

    # Construct the `item`.
    item = {"value": content_string}

    # Convert the project id into a full resource id.
    parent = f"projects/{project}"

    # Call the API.
    response = dlp.inspect_content(
        request={"parent": parent, "inspect_config": inspect_config, "item": item}
    )

    # Print out the results.
    if response.result.findings:
        for finding in response.result.findings:
            print(f"Quote: {finding.quote}")
            print(f"Info type: {finding.info_type.name}")
            print(f"Likelihood: {finding.likelihood}")
    else:
        print("No findings.")

Hotword example: Set the match likelihood of a table column

This example demonstrates how you can set the match likelihood of an entire column of data. This approach is helpful, for example, if you want to exclude a column of data from inspection results.

Note: The recommended approach for excluding an entire column of data from results is to use an ExcludeByHotword exclusion rule. For more information, see Omit findings in an entire column of data.

Consider the following table. One column contains placeholder Social Security numbers (SSNs), and another contains real SSNs.

Fake Social Security Number	Real Social Security Number
111-11-1111	222-22-2222

To minimize noise in inspection results, you can exclude any findings in the Fake Social Security Number column. Assign a low likelihood level to this column. Then, configure the request such that matches with that likelihood level are excluded from the results.

In this example, note the following:

• The hotword rule is applied to the US_SOCIAL_SECURITY_NUMBER infoType.
• The hotword regular expression (Fake Social Security Number) contains the name of the column that has the placeholder values.
• The windowBefore property is set to 1, which means that the hotword is in a column header, and the findings must be in the column.
• For each US_SOCIAL_SECURITY_NUMBER finding in this column, Cloud DLP sets the likelihood level to VERY_UNLIKELY.
• The minLikelihood property is set to POSSIBLE, which means that any finding that has a likelihood level lower than POSSIBLE is excluded from the inspection results.

See the JSON quickstart for more information about using the DLP API with JSON.

Note: The following example requires an OAuth 2.0 access token.

HTTP method and URL:

POST https://dlp.googleapis.com/v2/projects/PROJECT_ID/content:inspect

Replace PROJECT_ID with the project ID.

JSON input:

{
  "item": {
    "table": {
      "headers": [
        {
          "name": "Fake Social Security Number"
        },
        {
          "name": "Real Social Security Number"
        }
      ],
      "rows": [
        {
          "values": [
            {
              "stringValue": "111-11-1111"
            },
            {
              "stringValue": "222-22-2222"
            }
          ]
        }
      ]
    }
  },
  "inspectConfig": {
    "infoTypes": [
      {
        "name": "US_SOCIAL_SECURITY_NUMBER"
      }
    ],
    "includeQuote": true,
    "ruleSet": [
      {
        "infoTypes": [
          {
            "name": "US_SOCIAL_SECURITY_NUMBER"
          }
        ],
        "rules": [
          {
            "hotwordRule": {
              "hotwordRegex": {
                "pattern": "(Fake Social Security Number)"
              },
              "likelihoodAdjustment": {
                "fixedLikelihood": "VERY_UNLIKELY"
              },
              "proximity": {
                "windowBefore": 1
              }
            }
          }
        ]
      }
    ],
    "minLikelihood": "POSSIBLE"
  }
}

JSON output:

{
  "result": {
    "findings": [
      {
        "quote": "222-22-2222",
        "infoType": {
          "name": "US_SOCIAL_SECURITY_NUMBER"
        },
        "likelihood": "VERY_LIKELY",
        "location": {
          "byteRange": {
            "end": "11"
          },
          "codepointRange": {
            "end": "11"
          },
          "contentLocations": [
            {
              "recordLocation": {
                "fieldId": {
                  "name": "Real Social Security Number"
                },
                "tableLocation": {}
              }
            }
          ]
        },
        "createTime": "TIMESTAMP",
        "findingId": "TIMESTAMP"
      }
    ]
  }
}

The value 111-11-1111, which is in the Fake Social Security Number column, matched the hotword rule, so Cloud DLP assigned to it the VERY_UNLIKELY likelihood level . This level is lower than the minimum likelihood set in the inspection configuration (POSSIBLE), so this finding is excluded from the inspection result.

You can experiment with this example by removing the rule set. Notice that Cloud DLP includes 111-11-1111 in the results.