Manage Dataproc resources using custom constraints

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA products and features might have limited support, and changes to pre-GA products and features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

Google Cloud Organization Policy gives you centralized, programmatic control over your organization's resources. As the organization policy administrator, you can define an organization policy, which is a set of restrictions called constraints that apply to Google Cloud resources and descendants of those resources in the Google Cloud resource hierarchy. You can enforce organization policies at the organization, folder, or project level.

Organization Policy provides predefined constraints for various Google Cloud services. However, if you want more granular, customizable control over the specific fields that are restricted in your organization policies, you can also create custom constraints and use those custom constraints in an organization policy.

Benefits

You can use a custom organization policy to allow or deny specific operations on Dataproc clusters. For example, if a request to create or update a cluster fails to satisfy custom constraint validation as set by your organization policy, the request will fail, and an error will be returned to the caller.

Policy inheritance

By default, organization policies are inherited by the descendants of the resources on which you enforce the policy. For example, if you enforce a policy on a folder, Google Cloud enforces the policy on all projects in the folder. To learn more about this behavior and how to change it, refer to Hierarchy evaluation rules.

Pricing

The Organization Policy Service, including predefined and custom constraints, is offered at no charge.

Limitations

• During preview, if a custom constraint cannot be evaluated, for example if the organization policy server is down, the custom constraint will be ignored, and users will be able to create resources as if the constraint was not set. Note that syntax errors in Common Expression Language (CEL) conditions are not ignored, and will cause constraint creation or enforcement to fail. For more information, see Troubleshooting CEL errors).

Before you begin

1. Set up your project
   1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

   2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

      Go to project selector

   3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

   4. Enable the Dataproc API.

      Enable the API

   5. Install the Google Cloud CLI.

   6. To initialize the gcloud CLI, run the following command:

      gcloud init

   7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

      Go to project selector

   8. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

   9. Enable the Dataproc API.

      Enable the API

   10. Install the Google Cloud CLI.

   11. To initialize the gcloud CLI, run the following command:

       gcloud init

   12. Ensure that you know your organization ID.

Required roles

To get the permissions that you need to manage organization policies, ask your administrator to grant you the following IAM roles:

• Organization policy administrator (roles/orgpolicy.policyAdmin) on the organization resource
• To create or update a Dataproc cluster: Dataproc Admin or Dataproc Editor (roles/dataproc.admin or roles/dataproc.editor) on the project resource

For more information about granting roles, see Manage access.

These predefined roles contain the permissions required to manage organization policies. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to manage organization policies:

• orgpolicy.constraints.list
• orgpolicy.policies.create
• orgpolicy.policies.delete
• orgpolicy.policies.list
• orgpolicy.policies.update
• orgpolicy.policy.get
• orgpolicy.policy.set

You might also be able to get these permissions with custom roles or other predefined roles.

Create a custom constraint

A custom constraint is defined in a YAML file by the resources, methods, conditions, and actions it is applied to. Dataproc supports custom constraints that are applied to the CREATE and UPDATE methods of the CLUSTER resource (see Dataproc constraints on resources and operations).

To create a YAML file for a Dataproc custom constraint:

name: organizations/ORGANIZATION_ID/customConstraints/CONSTRAINT_NAME
resourceTypes:
- dataproc.googleapis.com/Cluster
methodTypes: 
- METHOD
condition: CONDITION
actionType: ACTION
displayName: DISPLAY_NAME
description: DESCRIPTION

Replace the following:

• ORGANIZATION_ID: your organization ID, such as 123456789.

• CONSTRAINT_NAME: the name you want for your new custom constraint. This field has a maximum length of 100 characters, not counting the prefix (for example, organizations/123456789/customConstraints/). A custom constraint must start with custom.. For example, custom.dataprocEnableComponentGateway.

• METHOD: When creating a cluster creation constraint, specify CREATE. When creating a cluster UPDATE constraint, specify both as follows:

  methodTypes:
  - CREATE
  - UPDATE
  

• CONDITION: a CEL condition that is written against a representation of a supported service resource. This field has a maximum length of 1000 characters. See Supported resources for more information about the resources available to write conditions against. For example, resource.config.endpointConfig.enableHttpPortAccess==true.

• ACTION: the action to take if the condition is met. This can be either ALLOW or DENY.

• DISPLAY_NAME: a human-friendly name for the constraint, for example, "Enforce enabling Dataproc Component Gateway". This field has a maximum length of 200 characters.

• DESCRIPTION: a human-friendly description of the constraint to display as an error message when the policy is violated, for example, "Only allow Dataproc cluster creation if the Component Gateway is enabled". This field has a maximum length of 2000 characters.

For more information about how to create a custom constraint, see Defining custom constraints.

Set up a custom constraint

After you have created a new custom constraint using the Google Cloud CLI, you must set it up to make it available for organization policies in your organization. To set up a custom constraint, use the gcloud org-policies set-custom-constraint command:

gcloud org-policies set-custom-constraint CONSTRAINT_PATH

Replace CONSTRAINT_PATH with the full path to your custom constraint file. For example, /home/user/customconstraint.yaml. Once completed, you will find your custom constraints as available constraints in your list of Google Cloud organization policies. To verify that the custom constraint exists, use the gcloud org-policies list-custom-constraints command:

gcloud org-policies list-custom-constraints --organization=ORGANIZATION_ID

Replace ORGANIZATION_ID with the ID of your organization resource. The output is similar to the following:

CONSTRAINT: custom.dataprocEnableComponentGateway
LIST_POLICY: -
BOOLEAN_POLICY: SET
ETAG: COCsm5QGENiXi2E=

For more information, see Viewing organization policies.

Enforce a custom constraint

You can enforce a boolean constraint by creating an organization policy that references it, and applying that organization policy to a Google Cloud resource.

Console

To enforce a boolean constraint:

1. Open the Organization policies page in the Google Cloud console.

   Open Organization policies page

2. Select the Project picker at the top of the page.
3. From the Project picker, select the project for which you want to set the organization policy.
4. Select your constraint from the list on the Organization policies page. The Policy details page for that constraint should appear.
5. To customize the organization policy for this resource, click Manage policy.
6. On the Edit policy page, select Customize.
7. Click Add rule.
8. Under Enforcement, select whether enforcement of this organization policy should be on or off.
9. Optionally, to make the organization policy conditional on a tag, click Add condition. Note that if you add a conditional rule to an organization policy, you must add at least one unconditional rule or the policy cannot be saved. For more details, see Setting an organization policy with tags.
10. To finish and apply the organization policy, click Save. The policy will take up to 15 minutes to take effect.

gcloud

To create an organization policy that enforces a boolean constraint, create a policy JSON file that references the constraint:

      name: projects/PROJECT_ID/policies/CONSTRAINT_NAME
      spec:
        rules:
        - enforce: true
    

Replace the following:

• PROJECT_ID: the project on which you want to enforce your constraint.
• CONSTRAINT_NAME: the name you defined for your custom constraint. For example, custom.dataprocEnableComponentGateway.

To enforce the organization policy containing the constraint, run the following command:

    gcloud org-policies set-policy POLICY_PATH
    

Replace POLICY_PATH with the full path to your organization policy JSON file. The policy will take up to 15 minutes to take effect.

Test the custom constraint

The following cluster creation example assumes a custom constraint has been created and enforced on cluster creation to require enabling the Component Gateway (resource.config.endpointConfig.enableHttpPortAccess==true).

gcloud dataproc clusters create example-cluster \
    --project=PROJECT_ID \
    --zone=COMPUTE_ZONE

Sample output (by default, the Component Gateway is not enabled when a Dataproc cluster is created):

Operation denied by custom org policies: ["customConstraints/custom.dataprocEnableComponentGateway": "Only allow Dataproc cluster creation if the Component Gateway is enabled"]

Dataproc constraints on resources and operations

The following Dataproc custom constraints are available to use when you create or update a Dataproc cluster. Note that when updating a cluster, only the constraints related to editable cluster parameters are supported (see Updating a cluster).

• Compute Engine network configuration (networkUri, internalIpOnly, serviceAccount and metadata)
  • resource.config.gceClusterConfig.networkUri
  • resource.config.gceClusterConfig.internalIpOnly
  • resource.config.gceClusterConfig.serviceAccount
  • resource.config.gceClusterConfig.metadata
• Compute Engine instance group configuration (imageUri and machineTypeUri)
  • resource.config.masterConfig.imageUri
  • resource.config.masterConfig.machineTypeUri
  • resource.config.workerConfig.imageUri
  • resource.config.workerConfig.machineTypeUri
  • resource.config.secondaryWorkerConfig.imageUri
  • resource.config.secondaryWorkerConfig.machineTypeUri
• Initialization actions (executableFile)
  • resource.config.initializationActions.executableFile
• Software config (imageVersion, properties and optionalComponents)
  • resource.config.softwareConfig.imageVersion
  • resource.config.softwareConfig.properties
  • resource.config.softwareConfig.optionalComponents
• Kerberos config (enableKerberos and crossRealmTrustKdc)
  • resource.config.securityConfig.kerberosConfig.enableKerberos
  • resource.config.securityConfig.kerberosConfig.crossRealmTrustKdc
• Component gateway (enableHttpPortAccess)
  • resource.config.endpointConfig.enableHttpPortAccess
• Metastore config (dataprocMetastoreService)
  • resource.config.metastoreConfig.dataprocMetastoreService
• Persistent Disk CMEK (gcePdKmsKeyName)
  • resource.config.encryptionConfig.gcePdKmsKeyName
• Cluster labels
  • resource.labels
• Cluster size
  • resource.config.masterConfig.numInstances
  • resource.config.workerConfig.numInstances
  • resource.config.secondaryWorkerConfig.numInstances
• Autoscaling
  • resource.config.autoscalingConfig.policyUri

Example custom constraints for common use cases

The following table provides the syntax of some custom constraints that you might find useful:

Description	Constraint syntax
Restrict the number of Dataproc worker instances to 10 or fewer when a cluster is created or updated.	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocNoMoreThan10Workers resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE - UPDATE condition: resource.config.workerConfig.numInstances + resource.config.secondaryWorkerConfig.numInstances > 10 actionType: DENY displayName: Total number of worker instances cannot be larger than 10 description: Cluster cannot have more than 10 workers, including primary and secondary workers.
Prevent application master from running on Dataproc cluster preemptible workers.	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocAmPrimaryOnlyEnforced resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE condition: ("dataproc:am.primary_only" in resource.config.softwareConfig.properties) && (resource.config.softwareConfig.properties["dataproc:am.primary_only"]=="true") actionType: ALLOW displayName: Application master cannot run on preemptible workers description: Property "dataproc:am.primary_only" must be "true".
Disallow custom Hive properties on Dataproc clusters	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocNoCustomHiveProperties resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE condition: resource.config.softwareConfig.properties.all(p, !p.startsWith("hive:")) actionType: ALLOW displayName: Cluster cannot have custom Hive properties description: Only allow Dataproc cluster creation if no property starts with Hive prefix "hive:".
Disallow the use of n1-standard-2 machine type on Dataproc master instances	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocMasterMachineType resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE condition: resource.config.masterConfig.machineTypeUri.contains("n1-standard-2") actionType: DENY displayName: Master cannot use the n1-standard-2 machine type description: Prevent Dataproc cluster creation if the master machine type is n1-standard-2.
Enforce the use of a specified initialization action script	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocInitActionScript resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE condition: resource.config.initializationActions.exists(action, action.executableFile=="gs://some/init-action.sh") actionType: ALLOW displayName: Initialization action script "gs://some/init-action.sh" must be used description: Only allow Dataproc cluster creation if the "gs://some/init-action.sh". initialization action script is used.
Enforce the usage of a specified PD encryption key	name: organizations/ORGANIZATION_ID/custom.dataprocPdCmek resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE condition: resource.config.encryptionConfig.gcePdKmsKeyName == "projects/project-id/locations/global/keyRings/key-ring-name/cryptoKeys/key-name" actionType: ALLOW displayName: Cluster PD must be encrypted with "key-name" from "key-ring-name" key-ring description: Only allow Dataproc cluster creation if the PD is encrypted with "key-name" from "key-ring-name" key-ring.
Enforce cluster label restrictions	name: organizations/ORGANIZATION_ID/customConstraints/custom.dataprocEnvLabel resourceTypes: - dataproc.googleapis.com/Cluster methodTypes: - CREATE - UPDATE condition: ("env" in resource.labels) && (resource.labels.env=="test") actionType: DENY displayName: Cluster cannot have the "env=test" label description: Deny Dataproc cluster creation or update if the cluster will be labeled "env=test".

What's next

• See Introduction to the Organization Policy Service to learn more about organization policies.
• Learn more about how to create and manage organization policies.
• See the full list of predefined Organization policy constraints.