Sensitive Data Protection uses information types—or infoTypes—to define what it scans for. An infoType is a type of sensitive data, such as a name, email address, telephone number, identification number, credit card number, and so on. An infoType detector is the corresponding detection mechanism that matches on an infoType's matching criteria.
How to use infoTypes
Sensitive Data Protection uses infoType detectors in the configuration for its scans to determine what to inspect for and how to transform findings. InfoType names are also used when displaying or reporting scan results.
For example, if you wanted to look for email addresses in a block of text, you
would specify the EMAIL_ADDRESS
infoType detector in the inspection
configuration. If you wanted to redact email addresses from the text block,
you would specify EMAIL_ADDRESS
in both the inspection configuration and the
de-identification configuration to indicate how to redact or transform that
type.
Further, you could use a combination of built-in and custom infoType detectors
to exclude a subset of email addresses from scan findings. First, create a
custom infoType called INTERNAL_EMAIL_ADDRESS
and configure it to exclude
internal test email addresses. Then, you can set up your scan to include
findings for EMAIL_ADDRESS
, but include an exclusion rule that excludes any
findings that match INTERNAL_EMAIL_ADDRESS
. For more information about
exclusion rules and other features of custom infoType detectors, see Creating
custom infoType detectors.
Sensitive Data Protection provides a set of built-in infoType detectors that you specify by name, each of which is listed in InfoType detector reference. These detectors use a variety of techniques to discover and classify each type. For example, some types will require a pattern match, some may have mathematical checksums, some have special digit restrictions, and others may have specific prefixes or context around the findings.
Examples
When you set up Sensitive Data Protection to scan your content, you include the infoType detectors to use in the scan configuration.
For example, the following JSON and code samples demonstrate a simple scan
request to the DLP API. Notice that the PHONE_NUMBER
detector is specified in
inspectConfig
,
which instructs Sensitive Data Protection to scan the given string for a
phone number.
C#
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
REST
JSON input:
POST https://dlp.googleapis.com/v2/projects/[PROJECT-ID]/content:inspect?key={YOUR_API_KEY}
{
"item":{
"value":"My phone number is (415) 555-0890"
},
"inspectConfig":{
"includeQuote":true,
"minLikelihood":"POSSIBLE",
"infoTypes":{
"name":"PHONE_NUMBER"
}
}
}
When you send the preceding request the specified endpoint, Sensitive Data Protection returns the following:
JSON output:
{
"result":{
"findings":[
{
"quote":"(415) 555-0890",
"infoType":{
"name":"PHONE_NUMBER"
},
"likelihood":"VERY_LIKELY",
"location":{
"byteRange":{
"start":"19",
"end":"33"
},
"codepointRange":{
"start":"19",
"end":"33"
}
},
"createTime":"2018-10-29T23:46:34.535Z"
}
]
}
}
If you need precise control and predictability as to what detectors are run, you should specify particular infoTypes listed in the reference, otherwise Sensitive Data Protection uses a default list, which may change over time. Depending on the amount of content to scan, scanning for default infoTypes can be prohibitively time-consuming or expensive.
For more information on how to use infoType detectors to scan your content, see one of the how-to topics about inspecting, redacting, or de-identifying.
Certainty and testing
Findings are reported with a certainty score called likelihood. The likelihood score indicates how likely a finding matches the corresponding type. For example, a type may return a lower likelihood if it only matches the pattern and return a higher likelihood if it matches the pattern and has positive context around it. For this reason, you may notice that a single finding could match several types at lower likelihood. Also, a finding may not appear or might have lower certainty if it doesn't match properly, or if it has negative context around it. For example, a finding might not reported if it matches the structure for the specified infoType but fails the infoType's checksum. Or a finding could match more than one infoType but have context that boosts one of them, and thus only get reported for that type.
If you are testing various detectors, you may notice that fake or sample data does not get reported because that fake or sample data is not passing enough checks to report.
Kinds of infoType detectors
Sensitive Data Protection includes several kinds of infoType detectors, all of which are summarized here:
- Built-in infoType detectors are built into Sensitive Data Protection. They include detectors for country- or region-specific sensitive data types as well as globally applicable data types.
- Custom infoType detectors are detectors that you create
yourself. There are three kinds of custom infoType detectors:
- Small custom dictionary detectors are simple word lists that Sensitive Data Protection matches on. Use small custom dictionary detectors when you have a list of up to several tens of thousands of words or phrases. Small custom dictionary detectors are preferred if you don't anticipate your word list changing significantly.
- Large custom dictionary detectors are generated by Sensitive Data Protection using large lists of words or phrases stored in either Cloud Storage or BigQuery. Use large custom dictionary detectors when you have a large list of words or phrases—up to tens of millions.
- Regular expressions (regex) detectors enable Sensitive Data Protection to detect matches based on a regular expression pattern.
In addition, Sensitive Data Protection includes the concept of inspection rules, which enable you to fine-tune scan results using the following:
- Exclusion rules enable you to decrease the number of findings returned by adding rules to a built-in or custom infoType detector.
- Hotword rules enable you to increase the quantity or change the likelihood value of findings returned by adding rules to a built-in or custom infoType detector.
Built-in infoType detectors
Built-in infoType detectors are built into Sensitive Data Protection, and
include detectors for country- or region-specific sensitive data types such as
the French Numéro d'Inscription au Répertoire (NIR) (FRANCE_NIR
), UK
driver's license number (UK_DRIVERS_LICENSE_NUMBER
), and US Social Security
number (US_SOCIAL_SECURITY_NUMBER
). They also include globally applicable data
types such as a person name (PERSON_NAME
), telephone numbers (PHONE_NUMBER
),
email addresses (EMAIL_ADDRESS
), and credit card numbers
(CREDIT_CARD_NUMBER
). To detect content that corresponds to infoTypes,
Sensitive Data Protection leverages various techniques including pattern
matching, checksums, machine-learning, context analysis, and others.
The list of built-in infoType detectors is always being updated. For a complete list of currently supported built-in infoType detectors, see InfoType detector reference.
You can also view a complete list of all built-in infoType detectors by
calling Sensitive Data Protection's
infoTypes.list
method.
Built-in infoType detectors are not a 100% accurate detection method. For example, they can't guarantee compliance with regulatory requirements. You must decide what data is sensitive and how to best protect it. Google recommends that you test your settings to make sure your configuration meets your requirements.
Language support
Country-specific infoTypes support the English language and the respective country's languages. Most global infoTypes work with multiple languages. Test Sensitive Data Protection with your data to verify that it meets your requirements.
Custom infoType detectors
There are three kinds of custom infoType detectors:
In addition, Sensitive Data Protection includes inspection rules, which enable you to fine-tune scan results by adding the following to existing detectors:
Small custom dictionary detectors
Use small custom dictionary detectors (also referred to as "regular custom dictionary detectors") to match a short (up to several tens of thousands) list of words or phrases. A small custom dictionary can act as its own unique detector.
Custom dictionary detectors are useful when you want to scan for a list of words or phrases that are not easily matched by a regular expression or a built-in detector. For example, suppose you want to scan for conference rooms that are commonly referred to by their assigned room names rather than their room numbers, such as state or region names, landmarks, fictional characters, and so on. You can make a small custom dictionary detector that contains a list of these room names. Sensitive Data Protection can scan your content for each of the room names and return a match when it encounters one of them in context. Learn more about how Sensitive Data Protection matches dictionary words and phrases in the "Dictionary matching specifics" section of Creating a regular custom dictionary detector.
For more details about how small dictionary custom infoType detectors work, as well as examples in action, see Creating a regular custom dictionary detector.
Large custom dictionary detectors
Use large custom dictionary detectors (also referred to as "stored custom dictionary detectors") when you have more than a few words or phrases to scan for, or if your list of words or phrases changes frequently. Large custom dictionary detectors can match on up to tens of millions of words or phrases.
Large custom dictionary detectors are created differently from both regular expression custom detectors and small custom dictionary detectors. Each large custom dictionary has two components:
- A list of phrases that you create and define. The list is stored as either a text file within Cloud Storage or a column in a BigQuery table.
- The generated dictionary files, which are built by Sensitive Data Protection based on your phrase list. The dictionary files are stored in Cloud Storage, and are comprised of a copy of the source phrase data plus bloom filters, which aid in searching and matching. You can't edit these files directly.
Once you've created a word list and then used Sensitive Data Protection to generate a custom dictionary, you initiate or schedule a scan using a large custom dictionary detector in a similar way as other infoType detectors.
For more details about how large custom dictionary detectors work, as well as examples in action, see Creating a stored custom dictionary detector.
Regular expressions
A regular expression (regex) custom infoType detector allows you to create your
own infoType detectors that enable Sensitive Data Protection to detect matches
based on a regex pattern. For example, suppose that you had medical record
numbers in the form ###-#-#####
. You could define a regex pattern such as the
following:
[1-9]{3}-[1-9]{1}-[1-9]{5}
The Sensitive Data Protection would then match items like this:
123-4-56789
You can also specify a likelihood to assign to each
custom infoType match. That is, when Sensitive Data Protection matches the sequence
you specify, it will assign the likelihood that you have indicated. This is
useful because if your custom regex defines a sequence that is common enough it
could easily match some other random sequence, you would not want
Sensitive Data Protection to label every match as VERY_LIKELY
. Doing so would
erode confidence in scan results and potentially cause the wrong information to
be matched or de-identified.
For more information about regular expression custom infoType detectors, and to see them in action, see Creating a custom regex detector.
Inspection rules
You use inspection rules to refine the results returned by existing infoType detectors—either built-in or custom. Inspection rules can be useful for times when the results that Sensitive Data Protection returns need to be augmented in some way, either by adding to and excluding from the existing infoType detector.
The two types of inspection rules are:
- Exclusion rules
- Hotword rules
For more information about inspection rules, see Modifying infoType detectors to refine scan results.
Exclusion rules
Exclusion rules enable you to decrease the quantity or precision of findings returned by adding rules to a built-in or custom infoType detector. Exclusion rules can help you reduce noise or other unwanted findings from being returned by an infoType detector.
For example, if you scan a database for email addresses, you can add an exclusion rule in the form of a custom regex that instructs Sensitive Data Protection to exclude any findings ending in "@example.com."
For more information about exclusion rules, see Modifying infoType detectors to refine scan results.
Hotword rules
Hotword rules enable you to increase the quantity or accuracy of findings returned by adding rules to a built-in or custom infoType detector. Hotword rules can effectively help you loosen an existing infoType detector's rules.
For example, suppose you want to scan a medical database for patient names. You
can use Sensitive Data Protection's built-in PERSON_NAME
infoType
detector, but that will cause Sensitive Data Protection to match on all
names of people, not just names of patients. To fix this, you can include a
hotword rule in the form of a regex custom infoType that looks for the word
"patient" within a certain character proximity from the first character of
potential matches. You can then assign findings matching this pattern a
likelihood of "very likely," since they correspond to
your special criteria.
For more information about hotword rules, see Modifying infoType detectors to refine scan results.
Examples
To get a better idea of how infoTypes match on findings, look at the following examples of matching on a series of digits to determine whether they constitute a US Social Security number or a US Individual Taxpayer Identification Number. Keep in mind that these examples are for built-in infoType detectors. When you create a custom infoType detector, you specify the criteria that determine the likelihood of a scan match.
Example 1
"SSN 222-22-2222"
Reports a high likelihood score of VERY_LIKELY
for a
US_SOCIAL_SECURITY_NUMBER
because:
- It is in the standard Social Security number format, which raises the certainty.
- It has context nearby ("SSN") that boosts towards
US_SOCIAL_SECURITY_NUMBER
.
Example 2
"999-99-9999"
Reports a low likelihood score of VERY_UNLIKELY
for a
US_SOCIAL_SECURITY_NUMBER
because:
- It is in the standard format, which raises the certainty.
- It starts with a 9, which is not allowed in Social Security numbers, thus lowers the certainty.
- It lacks context, which lowers the certainty.
Example 3
"999-98-9999"
Reports a likelihood score of POSSIBLE
for a
US_INDIVIDUAL_TAXPAYER_IDENTIFICATION_NUMBER
and VERY_UNLIKELY
for
US_SOCIAL_SECURITY_NUMBER
because:
- It has the standard format for both
US_SOCIAL_SECURITY_NUMBER
andUS_INDIVIDUAL_TAXPAYER_IDENTIFICATION_NUMBER
. - It starts with a 9 and has another digit check, which boosts certainty for
US_INDIVIDUAL_TAXPAYER_IDENTIFICATION_NUMBER
. - It lacks any context, which lowers the certainty for both.
What's next
The Sensitive Data Protection team releases new infoType detectors and groups periodically. To learn how to get the latest list of built-in infoTypes, see Listing built-in infoType detectors.