Dynamically mask column data

Learn how to use dynamic data masking to selectively obscure sensitive data, in order to provide different levels of visibility to different groups of users. For general information, see Introduction to dynamic data masking.

To add a data masking policy to a column, follow these steps:

  1. Create a taxonomy with at least one policy tag.
  2. Optionally, grant the Data Catalog Fine-Grained Reader role to one or more principals on one or more of the policy tags you created.
  3. Create a data policy for the policy tag, to map a masking rule and one or more principals to that tag.
  4. Set the policy tag on a column. That maps the data policy associated with the policy tag to the selected column.

You can use the Google Cloud console or the BigQuery Data Policy API to work with data policies.

When you have completed these steps, users running queries against the column get unmasked data, masked data, or an access denied error, depending on the roles that they have been granted. For more information, see How Masked Reader and Fine-Grained Reader roles interact.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  4. Enable the Data Catalog and BigQuery Data Policy APIs.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  7. Enable the Data Catalog and BigQuery Data Policy APIs.

    Enable the APIs

  8. BigQuery is automatically enabled in new projects, but you might need to activate it in a preexisting project.

    Enable the BigQuery API.

    Enable the API

Create taxonomies

You use Data Catalog to create a taxonomy of policy tags to use with your data.

The user or service account that creates a taxonomy must be granted the Data Catalog Policy Tag Admin role.

Console

  1. Open the Policy tag taxonomies page in the console.

    Open the Policy tag taxonomies page

  2. Click Create taxonomy.
  3. On the New taxonomy page:

    1. For Taxonomy name, enter the name of the taxonomy that you want to create.
    2. For Description, enter a description.
    3. If needed, change the project listed under Project.
    4. If needed, change the location listed under Location.
    5. Under Policy Tags, enter a policy tag name and description.
    6. To add a child policy tag for a policy tag, click Add subtag.
    7. To add a new policy tag at the same level as another policy tag, click + Add policy tag.
    8. Continue adding policy tags and child policy tags as needed for your taxonomy.
    9. When you are done creating policy tags for your hierarchy, click Create.

API

To use existing taxonomies, call taxonomies.import in place of the first two steps of the following procedure.

  1. Call taxonomies.create to create a taxonomy.
  2. Call taxonomies.policytag.create to create a policy tag.

Working with policy tags

For more information on how to work with policy tags, such as how to view or update them, see Working with policy tags. For best practices, see Best practices for using policy tags in BigQuery.

Create data policies

The user or service account that creates a data policy must have the bigquery.dataPolicies.create, bigquery.dataPolicies.setIamPolicy, and datacatalog.taxonomies.get permissions.

These permissions are included in the BigQuery Admin and BigQuery Data Owner roles.

Console

  1. Open the Data Catalog Taxonomies page in the console.

    Open the Taxonomies page

  2. Click the name of the taxonomy to open.
  3. Select a policy tag.
  4. Click Manage Data Policies.
  5. For Data Policy Name, type a name for the data policy. The data policy name must be unique within the project that data policy resides in.
  6. For Masking Rule, choose one of the masking rules.
  7. For Principal, type the name of one or more users or groups to whom you want to grant masked access to the column. Note that all users and groups you enter here are granted the BigQuery Masked Reader role.
  8. Click Submit.

API

  1. Call the create method. Pass in a DataPolicy resource That meets the following requirements:

    • The dataPolicyType field is set to DATA_MASKING_POLICY.
    • The dataMaskingPolicy field identifies the data masking rule to use.
    • The dataPolicyId field provides a name for the data policy that is unique within the project that data policy resides in.
  2. Call the setIamPolicy method and pass in a Policy. The Policy must identify the principals who are granted access to masked data, and specify roles/bigquerydatapolicy.maskedReader for the role field.

Set data policies on columns

Set a data policy on a column by attaching the policy tag associated with the data policy to the column.

The user or service account that sets a policy tag needs the datacatalog.taxonomies.get and bigquery.tables.setCategory permissions. datacatalog.taxonomies.get is included in the Data Catalog Policy Tags Admin and Project Viewer roles. bigquery.tables.setCategory is included in the Data Catalog Admin and Data Owner roles.

Console

Set the policy tag by modifying a schema using the console.

  1. Open the BigQuery page in the console.

    Go to the BigQuery page

  2. In the BigQuery Explorer, locate and select the table that you want to update. The table schema for that table opens.

  3. Click Edit Schema.

  4. In the Current schema screen, select the target column and click Add policy tag.

  5. In the Add a policy tag screen, locate and select the policy tag that you want to apply to the column.

  6. Click Select. Your screen should look similar to the following:

    Edit schema.

  7. Click Save.

Alternatively, when you create a table using the console, you can set the policy tag when you edit the schema as text.

Manually specify a schema following the Console instructions. Choose option 2 in Step 7, and include any needed policy tags for the columns in the specified schema. For the value of the names field of policyTags, use the policy tag resource name.

Edit as text.

bq

  1. Write the schema to a local file.

    bq show --schema --format=prettyjson \
       project-id:dataset.table > schema.json
    

    where:

    • project-id is your project ID.
    • dataset is the name of the dataset that contains the table you're updating.
    • table is the name of the table you're updating.
  2. Modify schema.json to set a policy tag on a column. For the value of the names field of policyTags, use the policy tag resource name.

    [
     ...
     {
       "name": "ssn",
       "type": "STRING",
       "mode": "REQUIRED",
       "policyTags": {
         "names": ["projects/project-id/locations/location/taxonomies/taxonomy-id/policyTags/policytag-id"]
       }
     },
     ...
    ]
    
  3. Update the schema.

    bq update \
       project-id:dataset.table schema.json
    

API

For existing tables, call tables.patch, or for new tables call tables.insert. Use the schema property of the Table object that you pass in to set a policy tag in your schema definition. See the command-line example schema to see how to set a policy tag.

When working with an existing table, the tables.patch method is preferred, because the tables.update method replaces the entire table resource.

Enforce access control

When you create a data policy for a policy tag, access control is automatically enforced. All columns that have that policy tag applied return masked data in response to queries from users who have the Masked Reader role.

To stop enforcement of access control, you must first delete all data policies associated with the policy tags in the taxonomy. For more information, see Enforce access control.

Check IAM permissions on a data policy

To see what permissions you have on a data policy, call the testIamPermissions method.

Update data policies

The user or service account that updates a data policy must have the bigquery.dataPolicies.update permission. If you are updating the policy tag the data policy is associated with, you also require the datacatalog.taxonomies.get permission.

If you are updating the principals associated with the data policy, you require the bigquery.dataPolicies.setIamPolicy permission.

The bigquery.dataPolicies.update and bigquery.dataPolicies.setIamPolicy permissions are included in the BigQuery Admin and BigQuery Data Owner roles. The datacatalog.taxonomies.get permission is included in the Data Catalog Admin and Data Catalog Viewer roles.

Console

  1. Open the Data Catalog Taxonomies page in the console.

    Open the Taxonomies page

  2. Click the name of the taxonomy to open.
  3. Select a policy tag.
  4. Click Manage Data Policies.
  5. Optionally, change the masking rule.
  6. Optional: Add or remove principals.
  7. Click Submit.

API

To change the data masking rule, call the patch method and pass in a DataPolicy resource with an updated dataMaskingPolicy field.

To change the principals associated with a data policy, call the setIamPolicy method and pass in a Policy that updates the principals that are granted access to masked data.

Delete data policies

The user or service account that creates a data policy must have the bigquery.dataPolicies.delete permission. This permission is included in the BigQuery Admin and BigQuery Data Owner roles.

Console

  1. Open the Data Catalog Taxonomies page in the console.

    Open the Taxonomies page

  2. Click the name of the taxonomy to open.
  3. Select a policy tag.
  4. Click Manage Data Policies.
  5. Click next to the data policy to delete.
  6. Click Submit.
  7. Click Confirm.

API

To delete a data policy, call the delete method.