Creating Cloud DLP de-identification templates

You can use templates to create and persist configuration information to use with Cloud Data Loss Prevention. Templates are useful for decoupling configuration such as what you inspect for and how you de-identify it from the implementation of your requests. Templates provide a robust way to re-use configuration and enable consistency across users and data sets.

The Cloud DLP supports two types of templates:

• De-identification templates: Templates for saving configuration information for de-identification jobs, including both infoType and record (structured dataset) transformations.
• Inspection templates: Templates for persisting configuration information for inspection scan jobs, including what predefined or custom detectors to use. For more information about inspection templates, see Creating Cloud DLP inspection templates.

For conceptual information about templates in Cloud DLP, see Templates.

The remainder of this topic shows you how to create de-identification templates for use with the Cloud DLP.

Create a de-identification template

Console

In the Google Cloud console, go to the Create template page.

Go to Create template

The following sections describe how to fill in the Create template page.

Define template

In the Define template section, fill in the fields as follows:

1. For Template type, select De-identify (remove sensitive data).

2. For Data transformation type, select the type of transformation to apply:

   • InfoType. Cloud DLP applies each transformation to only the value that was identified as the specific infoType. This transformation type is useful for unstructured text.

   • Record. Cloud DLP considers the structured format of the data when it applies a transformation. This transformation type is useful for tabular data.

   • Image. Cloud DLP redacts all detected text in an image or only text that matches a specific infoType.

3. For Template ID, enter a unique identifier for the template.

4. For Display name, enter a name for the template.

5. For Description, describe the template.

6. For Resource location, select Global or the location of the data that you want to de-identify.

7. Click Continue.

Configure de-identification

The selection of fields that appear in Configure de-identification is based on your chosen data transformation type.

InfoType

If you selected InfoType as the data transformation type, do the following:

1. In the Transformation field, select a transformation method to apply to findings.

   More fields appear according to your selection. Fill in the fields as needed. For more information, see Transformation reference.

2. In the InfoTypes to transform section, specify whether you want to apply the transformation to all detected infoTypes that are defined in your inspection configuration.

   If you select Specify infoTypes, select the infoTypes that you want to apply the transformation method to.

3. If you want to add transformation rules for infoTypes that aren't covered by your transformation rule, click Add transformation rule. Then, fill in the fields as needed. Repeat this step until you've configured a transformation rule for all infoTypes that you want to transform.

4. Click Create.

Record

If you selected Record as the data transformation type, do the following:

1. For Field(s) or column(s) to transform, type the name of the table column containing the data that you want to transform. Then, press ENTER. Repeat this step for each column containing data that you want to transform.

2. Specify the transformation type:

   • Match on infoType. Cloud DLP treats each cell as unstructured text and applies the transformation to only the infoTypes that it finds in the cell. Cloud DLP doesn't transform any data surrounding an infoType.

     If you select this option, do the following:

     1. Click Add transformation. In the Transformation field, select a transformation method to apply to findings.

     2. Specify whether you want to apply the transformation to all detected infoTypes that are defined in your inspection configuration.

        If you select Specify infoTypes, select the infoTypes that you want to apply the transformation rule to.

     If you want to assign certain transformation rules to certain infoTypes, you can add more transformations as needed.

   • Primitive field transformation. Cloud DLP always transforms the entire cell according to a transformation method that you choose. This option is useful if you want to transform entire columns of data, regardless of whether cells in those columns have detected infoTypes.

     If you select this option, in the Transformation field, select a transformation method to apply to the columns that you specified.

   In both cases, more fields appear according to your selected transformation method. Fill in the fields as needed. For more information, see Transformation reference.

3. Click Create.

Image

If you selected Image as the data transformation type, do the following:

1. Optional: To change the color of the box that covers the redacted text, click the black box in the Redaction color field, and choose a different color.

2. In the InfoTypes to transform section, specify what you want to redact: all detected text, all infoTypes defined in your inspection configuration, or only specific infoTypes.

   If you select Specify infoTypes, select the infoTypes that you want to apply the transformation rule to.

3. If you want to add transformation rules for infoTypes that aren't covered by your transformation rule, click Add transformation rule. Then, fill in the fields as needed. Repeat this step until you've configured a transformation rule for all infoTypes that you want to transform.

4. Click Create.

Protocol

It’s useful to note that a de-identification template is simply a reusable de-identification configuration plus some metadata. In API terms, the DeidentifyTemplate object is effectively a DeidentifyConfig object that includes a few more fields of metadata, such as a display name and a description. Therefore, to create a new de-identification template, the basic steps are:

1. Start with a DeidentifyConfig object.
2. Call or POST the create method of the projects.deidentifyTemplates resource, including in your request a DeidentifyTemplate object that contains a display name, a description, and that DeidentifyConfig object.

The returned DeidentifyTemplate will be usable immediately. You can reference it in other calls or jobs by its name. You can list the existing templates by calling the *.deidentifyTemplates.list method. To view a specific template, call the *.deidentifyTemplates.get method. Note that the limit for the number of templates you can create is 1000.

If you've already had some experience de-identifying sensitive data from text, images, or structured content using Cloud DLP, you've already created a DeidentifyConfig object. One additional step turns it into a DeidentifyTemplate object.

REST example

The following JSON is an example of what you could send to the projects.deidentifyTemplates.create method. This JSON creates a new template with the given display name and description, and scans for matches on the infoTypes EMAIL_ADDRESS and GENERIC_ID. When it finds content matching those infoTypes, it will mask the first three characters with an asterisk (*) character.

Note: The following example requires an OAuth 2.0 access token.

HTTP method and URL

POST https://dlp.googleapis.com/v2/projects/PROJECT_ID/deidentifyTemplates

Replace PROJECT_ID with the project ID.

JSON input

{
  "deidentifyTemplate":{
    "displayName":"Email and id masker",
    "description":"De-identifies emails and ids with a series of asterisks.",
    "deidentifyConfig":{
      "infoTypeTransformations":{
        "transformations":[
          {
            "infoTypes":[
              {
                "name":"GENERIC_ID"
              }
            ],
            "primitiveTransformation":{
              "replaceWithInfoTypeConfig":{

              }
            }
          },
          {
            "infoTypes":[
              {
                "name":"EMAIL_ADDRESS"
              }
            ],
            "primitiveTransformation":{
              "characterMaskConfig":{
                "charactersToIgnore":[
                  {
                    "charactersToSkip":"@"
                  }
                ],
                "maskingCharacter":"*"
              }
            }
          }
        ]
      }
    }
  }
}

JSON output

    {
      "name":"projects/PROJECT_ID/deidentifyTemplates/JOB-ID",
      "displayName":"Email and id masker",
      "description":"De-identifies emails and ids with a series of asterisks.",
      "createTime":"2018-11-30T07:17:59.536022Z",
      "updateTime":"2018-11-30T07:17:59.536022Z",
      "deidentifyConfig":{
        "infoTypeTransformations":{
          "transformations":[
            {
              "infoTypes":[
                {
                  "name":"GENERIC_ID"
                }
              ],
              "primitiveTransformation":{
                "replaceWithInfoTypeConfig":{

                }
              }
            },
            {
              "infoTypes":[
                {
                  "name":"EMAIL_ADDRESS"
                }
              ],
              "primitiveTransformation":{
                "characterMaskConfig":{
                  "maskingCharacter":"*",
                  "charactersToIgnore":[
                    {
                      "charactersToSkip":"@"
                    }
                  ]
                }
              }
            }
          ]
        }
      }
    }

To quickly try this out, you can use the APIs Explorer that's embedded below. For general information about using JSON to send requests to the DLP API, see the JSON quickstart.

Using de-identification templates

Once you’ve created a new template, you can use its name identifier anywhere deidentifyTemplateName is accepted, such as:

• projects.content.deidentify: De-identifies potentially sensitive data in content using the template as its configuration. Be aware that this method can use either an inspection template or a de-identification template.

Listing de-identification templates

To list all de-identification templates that have been created, use one of the *.*.list methods:

• projects.deidentifyTemplates.list

Code examples

Following is sample code in several languages that demonstrates how to use Cloud DLP to list inspection templates. The process is identical to the one for listing de-identification templates.

Important: The code on this page requires that you first set up a Cloud DLP client. For more information about installing and creating a Cloud DLP client, see Cloud DLP client libraries. (Sending JSON to Cloud DLP REST endpoints does not require a client library.)

Java

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

import com.google.cloud.dlp.v2.DlpServiceClient;
import com.google.cloud.dlp.v2.DlpServiceClient.ListInspectTemplatesPagedResponse;
import com.google.privacy.dlp.v2.InfoType;
import com.google.privacy.dlp.v2.InspectConfig;
import com.google.privacy.dlp.v2.InspectTemplate;
import com.google.privacy.dlp.v2.ListInspectTemplatesRequest;
import com.google.privacy.dlp.v2.LocationName;
import java.io.IOException;

class TemplatesList {

  public static void main(String[] args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    listInspectTemplates(projectId);
  }

  // Lists all templates associated with a given project
  public static void listInspectTemplates(String projectId) throws IOException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (DlpServiceClient dlpServiceClient = DlpServiceClient.create()) {

      // Create the request to be sent by the client
      ListInspectTemplatesRequest request =
          ListInspectTemplatesRequest.newBuilder()
              .setParent(LocationName.of(projectId, "global").toString())
              .setPageSize(1)
              .build();

      // Send the request
      ListInspectTemplatesPagedResponse response = dlpServiceClient.listInspectTemplates(request);

      // Parse through and process the response
      System.out.println("Templates found:");
      for (InspectTemplate template : response.getPage().getResponse().getInspectTemplatesList()) {
        System.out.printf("Template name: %s\n", template.getName());
        if (template.getDisplayName() != null) {
          System.out.printf("\tDisplay name: %s \n", template.getDisplayName());
          System.out.printf("\tCreate time: %s \n", template.getCreateTime());
          System.out.printf("\tUpdate time: %s \n", template.getUpdateTime());

          // print inspection config
          InspectConfig inspectConfig = template.getInspectConfig();
          for (InfoType infoType : inspectConfig.getInfoTypesList()) {
            System.out.printf("\tInfoType: %s\n", infoType.getName());
          }
          System.out.printf("\tMin likelihood: %s\n", inspectConfig.getMinLikelihood());
          System.out.printf("\tLimits: %s\n", inspectConfig.getLimits().getMaxFindingsPerRequest());
        }
      }
    }
  }
}

Node.js

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

// Imports the Google Cloud Data Loss Prevention library
const DLP = require('@google-cloud/dlp');

// Instantiates a client
const dlp = new DLP.DlpServiceClient();

// The project ID to run the API call under
// const projectId = 'my-project';

// Helper function to pretty-print dates
const formatDate = date => {
  const msSinceEpoch = parseInt(date.seconds, 10) * 1000;
  return new Date(msSinceEpoch).toLocaleString('en-US');
};

async function listInspectTemplates() {
  // Construct template-listing request
  const request = {
    parent: `projects/${projectId}/locations/global`,
  };

  // Run template-deletion request
  const [templates] = await dlp.listInspectTemplates(request);

  templates.forEach(template => {
    console.log(`Template ${template.name}`);
    if (template.displayName) {
      console.log(`  Display name: ${template.displayName}`);
    }

    console.log(`  Created: ${formatDate(template.createTime)}`);
    console.log(`  Updated: ${formatDate(template.updateTime)}`);

    const inspectConfig = template.inspectConfig;
    const infoTypes = inspectConfig.infoTypes.map(x => x.name);
    console.log('  InfoTypes:', infoTypes.join(' '));
    console.log('  Minimum likelihood:', inspectConfig.minLikelihood);
    console.log('  Include quotes:', inspectConfig.includeQuote);

    const limits = inspectConfig.limits;
    console.log('  Max findings per request:', limits.maxFindingsPerRequest);
  });
}

listInspectTemplates();

Python

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

def list_inspect_templates(project):
    """Lists all Data Loss Prevention API inspect templates.
    Args:
        project: The Google Cloud project id to use as a parent resource.
    Returns:
        None; the response from the API is printed to the terminal.
    """

    # Import the client library
    import google.cloud.dlp

    # Instantiate a client.
    dlp = google.cloud.dlp_v2.DlpServiceClient()

    # Convert the project id into a full resource id.
    parent = f"projects/{project}"

    # Call the API.
    response = dlp.list_inspect_templates(request={"parent": parent})

    for template in response:
        print("Template {}:".format(template.name))
        if template.display_name:
            print("  Display Name: {}".format(template.display_name))
        print("  Created: {}".format(template.create_time))
        print("  Updated: {}".format(template.update_time))

        config = template.inspect_config
        print(
            "  InfoTypes: {}".format(", ".join([it.name for it in config.info_types]))
        )
        print("  Minimum likelihood: {}".format(config.min_likelihood))
        print("  Include quotes: {}".format(config.include_quote))
        print(
            "  Max findings per request: {}".format(
                config.limits.max_findings_per_request
            )
        )

Go

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

import (
	"context"
	"fmt"
	"io"
	"time"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
	"github.com/golang/protobuf/ptypes"
	"google.golang.org/api/iterator"
)

// listInspectTemplates lists the inspect templates in the project.
func listInspectTemplates(w io.Writer, projectID string) error {
	// projectID := "my-project-id"

	ctx := context.Background()

	client, err := dlp.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("dlp.NewClient: %v", err)
	}
	defer client.Close()

	// Create a configured request.
	req := &dlppb.ListInspectTemplatesRequest{
		Parent: fmt.Sprintf("projects/%s/locations/global", projectID),
	}

	// Send the request and iterate over the results.
	it := client.ListInspectTemplates(ctx, req)
	for {
		t, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return fmt.Errorf("Next: %v", err)
		}
		fmt.Fprintf(w, "Inspect template %v\n", t.GetName())
		c, err := ptypes.Timestamp(t.GetCreateTime())
		if err != nil {
			return fmt.Errorf("CreateTime Timestamp: %v", err)
		}
		fmt.Fprintf(w, "  Created: %v\n", c.Format(time.RFC1123))
		u, err := ptypes.Timestamp(t.GetUpdateTime())
		if err != nil {
			return fmt.Errorf("UpdateTime Timestamp: %v", err)
		}
		fmt.Fprintf(w, "  Updated: %v\n", u.Format(time.RFC1123))
		fmt.Fprintf(w, "  Display Name: %q\n", t.GetDisplayName())
		fmt.Fprintf(w, "  Description: %q\n", t.GetDescription())
	}

	return nil
}

PHP

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

use Google\Cloud\Dlp\V2\DlpServiceClient;

/**
 * List DLP inspection configuration templates.
 *
 * @param string $callingProjectId  The project ID to run the API call under
 */
function list_inspect_templates(string $callingProjectId): void
{
    // Instantiate a client.
    $dlp = new DlpServiceClient();

    $parent = "projects/$callingProjectId/locations/global";

    // Run request
    $response = $dlp->listInspectTemplates($parent);

    // Print results
    $templates = $response->iterateAllElements();

    foreach ($templates as $template) {
        printf('Template %s' . PHP_EOL, $template->getName());
        printf('  Created: %s' . PHP_EOL, $template->getCreateTime()->getSeconds());
        printf('  Updated: %s' . PHP_EOL, $template->getUpdateTime()->getSeconds());
        printf('  Display Name: %s' . PHP_EOL, $template->getDisplayName());
        printf('  Description: %s' . PHP_EOL, $template->getDescription());

        $inspectConfig = $template->getInspectConfig();
        if ($inspectConfig === null) {
            print('  No inspect config.' . PHP_EOL);
        } else {
            printf('  Minimum likelihood: %s' . PHP_EOL, $inspectConfig->getMinLikelihood());
            printf('  Include quotes: %s' . PHP_EOL, $inspectConfig->getIncludeQuote());
            $limits = $inspectConfig->getLimits();
            printf('  Max findings per request: %s' . PHP_EOL, $limits->getMaxFindingsPerRequest());
        }
    }
}

C#

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;
using System;
using Google.Api.Gax;

public class InspectTemplateList
{
    public static PagedEnumerable<ListInspectTemplatesResponse, InspectTemplate> List(string projectId)
    {
        var client = DlpServiceClient.Create();

        var response = client.ListInspectTemplates(
            new ListInspectTemplatesRequest
            {
                Parent = new LocationName(projectId, "global").ToString(),
            }
        );

        // Uncomment to list templates
        //PrintTemplates(response);

        return response;
    }

    public static void PrintTemplates(PagedEnumerable<ListInspectTemplatesResponse, InspectTemplate> response)
    {
        foreach (var template in response)
        {
            Console.WriteLine($"Template {template.Name}:");
            Console.WriteLine($"\tDisplay Name: {template.DisplayName}");
            Console.WriteLine($"\tDescription: {template.Description}");
            Console.WriteLine($"\tCreated: {template.CreateTime}");
            Console.WriteLine($"\tUpdated: {template.UpdateTime}");
            Console.WriteLine("Configuration:");
            Console.WriteLine($"\tMin Likelihood: {template.InspectConfig?.MinLikelihood}");
            Console.WriteLine($"\tInclude quotes: {template.InspectConfig?.IncludeQuote}");
            Console.WriteLine($"\tMax findings per request: {template.InspectConfig?.Limits.MaxFindingsPerRequest}");
        }
    }
}

Deleting de-identification templates

To delete a specific de-identification template, use one of the *.*.delete methods:

• projects.deidentifyTemplates.delete

With each *.*.delete method, you include the resource name of the template to be deleted.

Code examples

Following is sample code in several languages that demonstrates how to use Cloud DLP to delete an inspection template. The process is identical to the one for listing de-identification templates.

Important: The code on this page requires that you first set up a Cloud DLP client. For more information about installing and creating a Cloud DLP client, see Cloud DLP client libraries. (Sending JSON to Cloud DLP REST endpoints does not require a client library.)

Java

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

import com.google.cloud.dlp.v2.DlpServiceClient;
import com.google.privacy.dlp.v2.DeleteInspectTemplateRequest;
import java.io.IOException;

class TemplatesDelete {

  public static void main(String[] args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String templateId = "your-template-id";
    deleteInspectTemplate(projectId, templateId);
  }

  // Delete an existing template
  public static void deleteInspectTemplate(String projectId, String templateId) throws IOException {
    // Construct the template name to be deleted
    String templateName = String.format("projects/%s/inspectTemplates/%s", projectId, templateId);

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (DlpServiceClient dlpServiceClient = DlpServiceClient.create()) {

      // Create delete template request to be sent by the client
      DeleteInspectTemplateRequest request =
          DeleteInspectTemplateRequest.newBuilder().setName(templateName).build();

      // Send the request with the client
      dlpServiceClient.deleteInspectTemplate(request);
      System.out.printf("Deleted template: %s\n", templateName);
    }
  }
}

Node.js

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

// Imports the Google Cloud Data Loss Prevention library
const DLP = require('@google-cloud/dlp');

// Instantiates a client
const dlp = new DLP.DlpServiceClient();

// The project ID to run the API call under
// const projectId = 'my-project';

// The name of the template to delete
// Parent project ID is automatically extracted from this parameter
// const templateName = 'projects/YOUR_PROJECT_ID/inspectTemplates/#####'
async function deleteInspectTemplate() {
  // Construct template-deletion request
  const request = {
    name: templateName,
  };

  // Run template-deletion request
  await dlp.deleteInspectTemplate(request);
  console.log(`Successfully deleted template ${templateName}.`);
}

deleteInspectTemplate();

Python

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

def delete_inspect_template(project, template_id):
    """Deletes a Data Loss Prevention API template.
    Args:
        project: The id of the Google Cloud project which owns the template.
        template_id: The id of the template to delete.
    Returns:
        None; the response from the API is printed to the terminal.
    """

    # Import the client library
    import google.cloud.dlp

    # Instantiate a client.
    dlp = google.cloud.dlp_v2.DlpServiceClient()

    # Convert the project id into a full resource id.
    parent = f"projects/{project}"

    # Combine the template id with the parent id.
    template_resource = "{}/inspectTemplates/{}".format(parent, template_id)

    # Call the API.
    dlp.delete_inspect_template(request={"name": template_resource})

    print("Template {} successfully deleted.".format(template_resource))

Go

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

import (
	"context"
	"fmt"
	"io"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
)

// deleteInspectTemplate deletes the given template.
func deleteInspectTemplate(w io.Writer, templateID string) error {
	// projectID := "my-project-id"
	// templateID := "my-template"

	ctx := context.Background()

	client, err := dlp.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("dlp.NewClient: %v", err)
	}
	defer client.Close()

	req := &dlppb.DeleteInspectTemplateRequest{
		Name: templateID,
	}

	if err := client.DeleteInspectTemplate(ctx, req); err != nil {
		return fmt.Errorf("DeleteInspectTemplate: %v", err)
	}
	fmt.Fprintf(w, "Successfully deleted inspect template %v", templateID)
	return nil
}

PHP

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

use Google\Cloud\Dlp\V2\DlpServiceClient;

/**
 * Delete a DLP inspection configuration template.
 *
 * @param string $callingProjectId  The project ID to run the API call under
 * @param string $templateId        The name of the template to delete
 */
function delete_inspect_template(
    string $callingProjectId,
    string $templateId
): void {
    // Instantiate a client.
    $dlp = new DlpServiceClient();

    // Run template deletion request
    $templateName = "projects/$callingProjectId/locations/global/inspectTemplates/$templateId";
    $dlp->deleteInspectTemplate($templateName);

    // Print results
    printf('Successfully deleted template %s' . PHP_EOL, $templateName);
}

C#

To learn how to install and use the client library for Cloud DLP, see Cloud DLP client libraries.

View on GitHub Feedback

using Google.Cloud.Dlp.V2;
using System;

public class InspectTemplateDelete
{
    public static object Delete(string projectId, string templateName)
    {
        var client = DlpServiceClient.Create();

        var request = new DeleteInspectTemplateRequest
        {
            Name = templateName
        };

        client.DeleteInspectTemplate(request);
        Console.WriteLine($"Successfully deleted template {templateName}.");

        return templateName;
    }
}