Mask column data

This document shows you how to implement data masking in order to selectively obscure sensitive data. By implementing data masking, you can provide different levels of visibility to different groups of users. For general information, see Introduction to data masking.

You implement data masking by adding a data policy to a column. To add a data masking policy to a column, you must complete the following steps :

1. Create a taxonomy with at least one policy tag.
2. Optional: Grant the Data Catalog Fine-Grained Reader role to one or more principals on one or more of the policy tags you created.
3. Create up to three data policies for the policy tag, to map masking rules and principals (which represent users or groups) to that tag.
4. Set the policy tag on a column. That maps the data policies associated with the policy tag to the selected column.
5. Assign users who should have access to masked data to the BigQuery Masked Reader role. As a best practice, assign the BigQuery Masked Reader role at the data policy level. Assigning the role at the project level or higher grants users permissions to all data policies under the project, which can lead to issues caused by excess permissions.

You can use the Google Cloud console or the BigQuery Data Policy API to work with data policies.

When you have completed these steps, users running queries against the column get unmasked data, masked data, or an access denied error, depending on the groups that they belong to and the roles that they have been granted. For more information, see How Masked Reader and Fine-Grained Reader roles interact.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Data Catalog and BigQuery Data Policy APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the Data Catalog and BigQuery Data Policy APIs.

   Enable the APIs

8. BigQuery is automatically enabled in new projects, but you might need to activate it in a pre-existing project.

   Enable the BigQuery API.

   Enable the API

9. If you are creating a data policy that references a custom masking routine, create the associated masking UDF so that it is available in the following steps.

Create taxonomies

The user or service account that creates a taxonomy must be granted the Data Catalog Policy Tag Admin role.

Console

1. Open the Policy tag taxonomies page in the Google Cloud console.

   Open the Policy tag taxonomies page

2. Click Create taxonomy.

3. On the New taxonomy page:

   1. For Taxonomy name, enter the name of the taxonomy that you want to create.
   2. For Description, enter a description.
   3. If needed, change the project listed under Project.
   4. If needed, change the location listed under Location.
   5. Under Policy Tags, enter a policy tag name and description.
   6. To add a child policy tag for a policy tag, click Add subtag.
   7. To add a new policy tag at the same level as another policy tag, click + Add policy tag.
   8. Continue adding policy tags and child policy tags as needed for your taxonomy.
   9. When you are done creating policy tags for your hierarchy, click Create.

API

To use existing taxonomies, call taxonomies.import in place of the first two steps of the following procedure.

1. Call taxonomies.create to create a taxonomy.
2. Call taxonomies.policytag.create to create a policy tag.

Work with policy tags

For more information about how to work with policy tags, such as how to view or update them, see Work with policy tags. For best practices, see Best practices for using policy tags in BigQuery.

Create data policies

The user or service account that creates a data policy must have the bigquery.dataPolicies.create, bigquery.dataPolicies.setIamPolicy, and datacatalog.taxonomies.get permissions.

The bigquery.dataPolicies.create and bigquery.dataPolicies.setIamPolicy permissions are included in the BigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles. The datacatalog.taxonomies.get permission is included in the Data Catalog Admin and Data Catalog Viewer roles.

If you are creating a data policy that references a custom masking routine, you also need routine permissions.

In case of custom masking, grant users the BigQuery Admin or BigQuery Data Owner roles to ensure they have the necessary permissions for both routines and data policies.

You can create up to nine data policies for a policy tag. One of these policies is reserved for column-level access control settings.

Console

1. Open the Policy tag taxonomies page in the Google Cloud console.

   Open the Policy tag taxonomies page

2. Click the name of the taxonomy to open.
3. Select a policy tag.
4. Click Manage Data Policies.
5. For Data Policy Name, type a name for the data policy. The data policy name must be unique within the project that data policy resides in.
6. For Masking Rule, choose a predefined masking rule or a custom masking routine. If you are selecting a custom masking routine, ensure that you have both the bigquery.routines.get and the bigquery.routines.list permissions at the project level.
7. For Principal, type the name of one or more users or groups to whom you want to grant masked access to the column. Note that all users and groups you enter here are granted the BigQuery Masked Reader role.
8. Click Submit.

API

1. Call the create method. Pass in a DataPolicy resource That meets the following requirements:

   • The dataPolicyType field is set to DATA_MASKING_POLICY.
   • The dataMaskingPolicy field identifies the data masking rule or routine to use.
   • The dataPolicyId field provides a name for the data policy that is unique within the project that data policy resides in.

2. Call the setIamPolicy method and pass in a Policy. The Policy must identify the principals who are granted access to masked data, and specify roles/bigquerydatapolicy.maskedReader for the role field.

Set policy tags on columns

Set a data policy on a column by attaching the policy tag associated with the data policy to the column.

The user or service account that sets a policy tag needs the datacatalog.taxonomies.get and bigquery.tables.setCategory permissions. datacatalog.taxonomies.get is included in the Data Catalog Policy Tags Admin and Project Viewer roles. bigquery.tables.setCategory is included in the BigQuery Admin (roles/bigquery.admin) and BigQuery Data Owner (roles/bigquery.dataOwner) roles.

To view taxonomies and policy tags across all projects in an organization in Google Cloud console, users need the resourcemanager.organizations.get permission, which is included in the Organization Viewer role.

Note: You can assign only one policy tag per column.

Console

Set the policy tag by modifying a schema using the Google Cloud console.

1. Open the BigQuery page in the Google Cloud console.

   Go to the BigQuery page

2. In the BigQuery Explorer, locate and select the table that you want to update. The table schema for that table opens.

3. Click Edit Schema.

4. In the Current schema screen, select the target column and click Add policy tag.

5. In the Add a policy tag screen, locate and select the policy tag that you want to apply to the column.

6. Click Select. Your screen should look similar to the following:

   

7. Click Save.

bq

1. Write the schema to a local file.

   bq show --schema --format=prettyjson \
      project-id:dataset.table > schema.json
   

   where:

   • project-id is your project ID.
   • dataset is the name of the dataset that contains the table you're updating.
   • table is the name of the table you're updating.

2. Modify schema.json to set a policy tag on a column. For the value of the names field of policyTags, use the policy tag resource name.

   [
    ...
    {
      "name": "ssn",
      "type": "STRING",
      "mode": "REQUIRED",
      "policyTags": {
        "names": ["projects/project-id/locations/location/taxonomies/taxonomy-id/policyTags/policytag-id"]
      }
    },
    ...
   ]
   

3. Update the schema.

   bq update \
      project-id:dataset.table schema.json
   

API

For existing tables, call tables.patch, or for new tables call tables.insert. Use the schema property of the Table object that you pass in to set a policy tag in your schema definition. See the command-line example schema to see how to set a policy tag.

When working with an existing table, the tables.patch method is preferred, because the tables.update method replaces the entire table resource.

Enforce access control

When you create a data policy for a policy tag, access control is automatically enforced. All columns that have that policy tag applied return masked data in response to queries from users who have the Masked Reader role.

To stop enforcement of access control, you must first delete all data policies associated with the policy tags in the taxonomy. For more information, see Enforce access control.

Check IAM permissions on a data policy

To see what permissions you have on a data policy, call the testIamPermissions method.

Update data policies

The user or service account that updates a data policy must have the bigquery.dataPolicies.update permission. If you are updating the policy tag the data policy is associated with, you also require the datacatalog.taxonomies.get permission.

If you are updating the principals associated with the data policy, you require the bigquery.dataPolicies.setIamPolicy permission.

The bigquery.dataPolicies.update and bigquery.dataPolicies.setIamPolicy permissions are included in the BigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles. The datacatalog.taxonomies.get permission is included in the Data Catalog Admin and Data Catalog Viewer roles.

Console

1. Open the Policy tag taxonomies page in the Google Cloud console.

   Open the Policy tag taxonomies page

2. Click the name of the taxonomy to open.
3. Select a policy tag.
4. Click Manage Data Policies.
5. Optionally, change the masking rule.
6. Optional: Add or remove principals.
7. Click Submit.

API

To change the data masking rule, call the patch method and pass in a DataPolicy resource with an updated dataMaskingPolicy field.

To change the principals associated with a data policy, call the setIamPolicy method and pass in a Policy that updates the principals that are granted access to masked data.

Delete data policies

The user or service account that creates a data policy must have the bigquery.dataPolicies.delete permission. This permission is included in the BigQuery Data Policy Admin, BigQuery Admin and BigQuery Data Owner roles.

Console

1. Open the Policy tag taxonomies page in the Google Cloud console.

   Open the Policy tag taxonomies page

2. Click the name of the taxonomy to open.
3. Select a policy tag.
4. Click Manage Data Policies.
5. Click next to the data policy to delete.
6. Click Submit.
7. Click Confirm.

API

To delete a data policy, call the delete method.