Use custom organization policies
This page shows you how to use Organization Policy Service custom constraints to restrict specific operations on the following Google Cloud resources:
bigquerydatatransfer.googleapis.com/TransferConfig
To learn more about Organization Policy, see Custom organization policies.
About organization policies and constraints
The Google Cloud Organization Policy Service gives you centralized, programmatic control over your organization's resources. As the organization policy administrator, you can define an organization policy, which is a set of restrictions called constraints that apply to Google Cloud resources and descendants of those resources in the Google Cloud resource hierarchy. You can enforce organization policies at the organization, folder, or project level.
Organization Policy provides predefined constraints for various Google Cloud services. However, if you want more granular, customizable control over the specific fields that are restricted in your organization policies, you can also create custom constraints and use those custom constraints in an organization policy.
Policy inheritance
By default, organization policies are inherited by the descendants of the resources on which you enforce the policy. For example, if you enforce a policy on a folder, Google Cloud enforces the policy on all projects in the folder. To learn more about this behavior and how to change it, refer to Hierarchy evaluation rules.
Benefits
You can use a custom organization policy to allow or deny specific operations on BigQuery Data Transfer Service transfer configurations in order to meet your organization's compliance and security requirements. If a request to create or update a transfer configuration fails to satisfy custom constraints set by your organization policy, then the request fails, and an error is returned to the caller.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
- Ensure that you know your organization ID.
Required roles
To get the permissions that you need to manage organization policies, ask your administrator to grant you the following IAM roles:
-
Organization Policy Administrator (
roles/orgpolicy.policyAdmin
) on the organization resource -
Create or update a BigQuery Data Transfer Service transfer configuration:
BigQuery Admin (
roles/bigquery.admin
) on the project resource
For more information about granting roles, see Manage access to projects, folders, and organizations.
These predefined roles contain the permissions required to manage organization policies. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to manage organization policies:
-
orgpolicy.*
on the organization resource -
Create or update a BigQuery Data Transfer Service transfer configuration:
-
bigquery.transfers.get
on the project resource -
bigquery.transfers.update
on the project resource
-
You might also be able to get these permissions with custom roles or other predefined roles.
Create a custom constraint
A custom constraint is defined in a YAML file by the resources, methods, conditions, and actions that are supported by the service on which you are enforcing the organization policy. Conditions for your custom constraints are defined using Common Expression Language (CEL). For more information about how to build conditions in custom constraints using CEL, see the CEL section of Creating and managing custom constraints.
To create a custom constraint, create a YAML file using the following format:
name: organizations/ORGANIZATION_ID/customConstraints/CONSTRAINT_NAME
resourceTypes:
- RESOURCE_NAME
methodTypes:
- CREATE
- UPDATE
condition: "CONDITION"
actionType: ACTION
displayName: DISPLAY_NAME
description: DESCRIPTION
Replace the following:
ORGANIZATION_ID
: your organization ID, such as123456789
.CONSTRAINT_NAME
: the name you want for your new custom constraint. A custom constraint must start withcustom.
, and can only include uppercase letters, lowercase letters, or numbers, for example,custom.dtsEnableEmailNotification
. The maximum length of this field is 70 characters.RESOURCE_NAME
: the fully qualified name of the Google Cloud resource containing the object and field you want to restrict. For example,bigquerydatatransfer.googleapis.com/TransferConfig
.CONDITION
: a CEL condition that is written against a representation of a supported service resource. This field has a maximum length of 1000 characters. See Supported resources for more information about the resources available to write conditions against. For example,resource.emailPreferences.enableFailureEmail == true
.ACTION
: the action to take if thecondition
is met. Possible values areALLOW
andDENY
.DISPLAY_NAME
: a human-friendly name for the constraint. This field has a maximum length of 200 characters.DESCRIPTION
: a human-friendly description of the constraint to display as an error message when the policy is violated. This field has a maximum length of 2000 characters.
For more information about how to create a custom constraint, see Defining custom constraints.
Set up a custom constraint
After you have created the YAML file for a new custom constraint, you must set it up to make it available for organization policies in your organization. To set up a custom constraint, use thegcloud org-policies set-custom-constraint
command:
gcloud org-policies set-custom-constraint CONSTRAINT_PATH
CONSTRAINT_PATH
with the full path to your
custom constraint file. For example, /home/user/customconstraint.yaml
.
Once completed, your custom constraints are available as organization policies
in your list of Google Cloud organization policies.
To verify that the custom constraint exists, use the
gcloud org-policies list-custom-constraints
command:
gcloud org-policies list-custom-constraints --organization=ORGANIZATION_ID
ORGANIZATION_ID
with the ID of your organization resource.
For more information, see
Viewing organization policies.
Enforce a custom organization policy
You can enforce a boolean constraint by creating an organization policy that references it, and then applying that organization policy to a Google Cloud resource.Console
- In the Google Cloud console, go to the Organization policies page.
- From the project picker, select the project for which you want to set the organization policy.
- From the list on the Organization policies page, select your constraint to view the Policy details page for that constraint.
- To configure the organization policy for this resource, click Manage policy.
- On the Edit policy page, select Override parent's policy.
- Click Add a rule.
- In the Enforcement section, select whether enforcement of this organization policy is on or off.
- Optional: To make the organization policy conditional on a tag, click Add condition. Note that if you add a conditional rule to an organization policy, you must add at least one unconditional rule or the policy cannot be saved. For more information, see Setting an organization policy with tags.
- If this is a custom constraint, you can click Test changes to simulate the effect of this organization policy. For more information, see Test organization policy changes with Policy Simulator.
- To finish and apply the organization policy, click Set policy. The policy requires up to 15 minutes to take effect.
gcloud
To create an organization policy that enforces a boolean constraint, create a policy YAML file that references the constraint:
name: projects/PROJECT_ID/policies/CONSTRAINT_NAME spec: rules: - enforce: true
Replace the following:
-
PROJECT_ID
: the project on which you want to enforce your constraint. -
CONSTRAINT_NAME
: the name you defined for your custom constraint. For example,custom.dtsEnableEmailNotification
.
To enforce the organization policy containing the constraint, run the following command:
gcloud org-policies set-policy POLICY_PATH
Replace POLICY_PATH
with the full path to your organization policy
YAML file. The policy requires up to 15 minutes to take effect.
Test the custom organization policy
The following example creates a custom constraint and policy that require that email notifications are enabled for all new transfer configurations in a specific project.
Before you begin, you should know the following:
- Your organization ID
- A project ID
Create the constraint
Save the following file as
constraint-dts-enable-email.yaml
:name: organizations/ORGANIZATION_ID/customConstraints/custom.dtsEnableEmailNotification resourceTypes: - bigquerydatatransfer.googleapis.com/TransferConfig methodTypes: - CREATE condition: resource.emailPreferences.enableFailureEmail == true actionType: ALLOW displayName: The BigQuery Data Transfer Service always enables email notications description: The BigQuery Data Transfer Service always enables email notications on data transfer failures.
Replace
ORGANIZATION_ID
with your organization ID.This defines a constraint that checks whether email notifications are enabled for a new transfer configuration. If they are not enabled, the operation is denied.
Apply the constraint:
gcloud org-policies set-custom-constraint ~/constraint-dts-enable-email.yaml
Verify that the constraint exists:
gcloud org-policies list-custom-constraints --organization=ORGANIZATION_ID
The output is similar to the following:
CUSTOM_CONSTRAINT ACTION_TYPE METHOD_TYPES RESOURCE_TYPES DISPLAY_NAME custom.dtsEnableEmailNotification ALLOW CREATE bigquerydatatransfer.googleapis.com/TransferConfig The BigQuery Data Transfer Service always enables email notications ...
Create the policy
Save the following file as
policy-dts-enable-email.yaml
:name: projects/PROJECT_ID/policies/custom.dtsEnableEmailNotification spec: rules: - enforce: true
Replace
PROJECT_ID
with your project ID.Apply the policy:
gcloud org-policies set-policy ~/policy-dts-enable-email.yaml
Verify that the policy exists:
gcloud org-policies list --project=PROJECT_ID
The output is similar to the following:
CONSTRAINT LIST_POLICY BOOLEAN_POLICY ETAG custom.dtsEnableEmailNotification - SET CPyxlbgGENDL3tEC-
After you apply the policy, wait for about two minutes for Google Cloud to start enforcing the policy.
Test the policy
In the project, try to create a BigQuery Data Transfer Service transfer configuration with the email notification turned off.
The output is the following:
Operation denied by custom org policy: ["customConstraints/custom.dtsEnableEmailNotification": "The BigQuery Data Transfer Service always enables email notications on data transfer failures."].
Example custom organization policies for common use cases
The following table provides the syntax of some custom constraints for common use cases:
Description | Constraint syntax |
---|---|
Disallow data transfers from Azure Blob Storage |
name: organizations/ORGANIZATION_ID/customConstraints/custom.denyDtsAzureTransfers resourceTypes: - bigquerydatatransfer.googleapis.com/TransferConfig methodTypes: - CREATE condition: resource.dataSourceId == "azure_blob_storage" actionType: DENY displayName: Deny data transfers from Azure Blob Storage to BigQuery description: Disallow creating data transfer configurations from Azure Blob Storage. |
Always enable auto-scheduling |
name: organizations/ORGANIZATION_ID/customConstraints/custom.dtsNoManualSchedule resourceTypes: - bigquerydatatransfer.googleapis.com/TransferConfig methodTypes: - CREATE - UPDATE condition: resource.scheduleOptions.disableAutoScheduling == false actionType: ALLOW displayName: Transfer configurations always enable auto-scheduling description: Always enable auto-scheduling for BigQuery Data Transfer Service transfer configurations. |
Google Ads transfers must have a data refresh window of more than three days |
name: organizations/ORGANIZATION_ID/customConstraints/custom.dtsGoogleAdsConstraint resourceTypes: - bigquerydatatransfer.googleapis.com/TransferConfig methodTypes: - CREATE - UPDATE condition: resource.dataSourceId == "google_ads" && resource.dataRefreshWindowDays < 3 actionType: DENY displayName: Google Ads transfers data refresh window must be greater than three days description: Disallow creating Google Ads data transfer configurations whose data refresh window is less than three days. |
BigQuery Data Transfer Service supported resources
The following table lists the BigQuery Data Transfer Service resources that you can reference in custom constraints.Resource | Field |
---|---|
bigquerydatatransfer.googleapis.com/TransferConfig |
resource.dataRefreshWindowDays
|
resource.dataSourceId
| |
resource.destinationDatasetId
| |
resource.disabled
| |
resource.displayName
| |
resource.emailPreferences.enableFailureEmail
| |
resource.encryptionConfiguration.kmsKeyName
| |
resource.name
| |
resource.notificationPubsubTopic
| |
resource.schedule
| |
resource.scheduleOptions.disableAutoScheduling
| |
resource.scheduleOptions.endTime
| |
resource.scheduleOptions.startTime
| |
resource.scheduleOptionsV2.eventDrivenSchedule.pubsubSubscription
| |
resource.scheduleOptionsV2.timeBasedSchedule.endTime
| |
resource.scheduleOptionsV2.timeBasedSchedule.schedule
| |
resource.scheduleOptionsV2.timeBasedSchedule.startTime
|
What's next
- Learn more about Organization Policy Service.
- Learn more about how to create and manage organization policies.
- See the full list of predefined organization policy constraints.