Package google.cloud.orchestration.airflow.service.v1beta1

Index

Environments

Managed Apache Airflow Environments.

CreateEnvironment

rpc CreateEnvironment(CreateEnvironmentRequest) returns (Operation)

Create a new environment.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteEnvironment

rpc DeleteEnvironment(DeleteEnvironmentRequest) returns (Operation)

Delete an environment.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetEnvironment

rpc GetEnvironment(GetEnvironmentRequest) returns (Environment)

Get an existing environment.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListEnvironments

rpc ListEnvironments(ListEnvironmentsRequest) returns (ListEnvironmentsResponse)

List environments.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateEnvironment

rpc UpdateEnvironment(UpdateEnvironmentRequest) returns (Operation)

Update an environment.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateEnvironmentRequest

Create a new environment.

Fields
parent

string

The parent must be of the form "projects/{projectId}/locations/{locationId}".

Authorization requires the following Google IAM permission on the specified resource parent:

  • composer.environments.create

environment

Environment

The environment to create.

DeleteEnvironmentRequest

Delete an environment.

Fields
name

string

The environment to delete, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following Google IAM permission on the specified resource name:

  • composer.environments.delete

Environment

An environment for running orchestration tasks.

Fields
name

string

The resource name of the environment, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

config

EnvironmentConfig

Configuration parameters for this environment.

uuid

string

Output only. The UUID (Universally Unique IDentifier) associated with this environment. This value is generated when the environment is created.

state

State

The current state of the environment.

create_time

Timestamp

Output only. The time at which this environment was created.

update_time

Timestamp

Output only. The time at which this environment was last modified.

labels

map<string, string>

Optional. User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions:

  • Keys must conform to regexp: [\p{Ll}\p{Lo}][\p{Ll}\p{Lo}\p{N}_-]{0,62}
  • Values must conform to regexp: [\p{Ll}\p{Lo}\p{N}_-]{0,63}
  • Both keys and values are additionally constrained to be <= 128 bytes in size.

State

State of the environment.

Enums
STATE_UNSPECIFIED The state of the environment is unknown.
CREATING The environment is in the process of being created.
RUNNING The environment is currently running and healthy. It is ready for use.
UPDATING The environment is being updated. It remains usable but cannot receive additional update requests or be deleted at this time.
DELETING The environment is undergoing deletion. It cannot be used.
ERROR The environment has encountered an error and cannot be used.

EnvironmentConfig

Configuration information for an environment.

Fields
gke_cluster

string

Output only. The Kubernetes Engine cluster used to run this environment.

dag_gcs_prefix

string

Output only. The Cloud Storage prefix of the DAGs for this environment. Although Cloud Storage objects reside in a flat namespace, a hierarchical file tree can be simulated using "/"-delimited object name prefixes. DAG objects for this environment reside in a simulated directory with the given prefix.

node_count

int32

The number of nodes in the Kubernetes Engine cluster that will be used to run this environment.

software_config

SoftwareConfig

The configuration settings for software inside the environment.

node_config

NodeConfig

The configuration used for the Kubernetes Engine cluster.

airflow_uri

string

Output only. The URI of the Apache Airflow Web UI hosted within this environment (see Airflow web interface).

GetEnvironmentRequest

Get an environment.

Fields
name

string

The resource name of the environment to get, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following Google IAM permission on the specified resource name:

  • composer.environments.get

ListEnvironmentsRequest

List environments in a project and location.

Fields
parent

string

List environments in the given project and location, in the form: "projects/{projectId}/locations/{locationId}"

Authorization requires the following Google IAM permission on the specified resource parent:

  • composer.environments.list

page_size

int32

The maximum number of environments to return.

page_token

string

The next_page_token value returned from a previous List request, if any.

ListEnvironmentsResponse

The environments in a project and location.

Fields
environments[]

Environment

The list of environments returned by a ListEnvironmentsRequest.

next_page_token

string

The page token used to query for the next page if one exists.

NodeConfig

The configuration information for the Kubernetes Engine nodes running the Apache Airflow software.

Fields
location

string

Optional. The Compute Engine zone in which to deploy the VMs used to run the Apache Airflow software, specified as a relative resource name. For example: "projects/{projectId}/zones/{zoneId}".

This location must belong to the enclosing environment's project and location. If both this field and nodeConfig.machineType are specified, nodeConfig.machineType must belong to this location; if both are unspecified, the service will pick a zone in the Compute Engine region corresponding to the Cloud Composer location, and propagate that choice to both fields. If only one field (location or nodeConfig.machineType) is specified, the location information from the specified field will be propagated to the unspecified field.

machine_type

string

Optional. The Compute Engine machine type used for cluster instances, specified as a relative resource name. For example: "projects/{projectId}/zones/{zoneId}/machineTypes/{machineTypeId}".

The machineType must belong to the enclosing environment's project and location. If both this field and nodeConfig.location are specified, this machineType must belong to the nodeConfig.location; if both are unspecified, the service will pick a zone in the Compute Engine region corresponding to the Cloud Composer location, and propagate that choice to both fields. If exactly one of this field and nodeConfig.location is specified, the location information from the specified field will be propagated to the unspecified field.

If this field is unspecified, the machineTypeId defaults to "n1-standard-1".

network

string

Optional. The Compute Engine network to be used for machine communications, specified as a relative resource name. For example: "projects/{projectId}/global/networks/{networkId}".

If unspecified, the default network in the environment's project is used. If a Custom Subnet Network is provided, nodeConfig.subnetwork must also be provided. For Shared VPC subnetwork requirements, see nodeConfig.subnetwork.

subnetwork

string

Optional. The Compute Engine subnetwork to be used for machine communications, specified as a relative resource name. For example: "projects/{projectId}/regions/{regionId}/subnetworks/{subnetworkId}"

If a subnetwork is provided, nodeConfig.network must also be provided, and the subnetwork must belong to the same project as the network.

For Shared VPC, you must configure the subnetwork with secondary ranges named composer-pods and composer-services to support Alias IPs.

disk_size_gb

int32

Optional. The disk size in GB used for node VMs. Minimum size is 20GB. If unspecified, defaults to 100GB. Cannot be updated.

oauth_scopes[]

string

Optional. The set of Google API scopes to be made available on all node VMs. Defaults to ["https://www.googleapis.com/auth/cloud-platform"] and must be included in the list of specified scopes. Cannot be updated.

service_account

string

Optional. The Google Cloud Platform Service Account to be used by the node VMs. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated.

tags[]

string

Optional. The list of instance tags applied to all node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC1035. Cannot be updated.

OperationMetadata

Metadata describing an operation.

Fields
state

State

Output only. The current operation state.

operation_type

Type

Output only. The type of operation being performed.

resource

string

Output only. The resource being operated on, as a [relative resource name]( /apis/design/resource_names#relative_resource_name).

resource_uuid

string

Output only. The UUID of the resource being operated on.

create_time

Timestamp

Output only. The time the operation was submitted to the server.

end_time

Timestamp

Output only. The time when the operation terminated, regardless of its success. This field is unset if the operation is still ongoing.

State

An enum describing the overall state of an operation.

Enums
STATE_UNSPECIFIED Unused.
PENDING The operation has been created but is not yet started.
RUNNING The operation is underway.
SUCCESSFUL The operation completed successfully.
FAILED The operation is no longer running but did not succeed.

Type

Type of longrunning operation.

Enums
TYPE_UNSPECIFIED Unused.
CREATE A resource creation operation.
DELETE A resource deletion operation.
UPDATE A resource update operation.

SoftwareConfig

Specifies the selection and configuration of software inside the environment.

Fields
image_version

string

Output only. The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-[0-9]+\.[0-9]+(\.[0-9]+)?-airflow-[0-9]+\.[0-9]+(\.[0-9]+.*)?.

The Cloud Composer portion of the version is a semantic version. The portion of the image version following airflow- is an official Apache Airflow repository release name.

See also Release Notes.

airflow_config_overrides

map<string, string>

Optional. Apache Airflow configuration properties to override.

Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation". Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and must not contain an equals sign ("=") or semicolon (";"). Section and property names must not contain a period ("."). Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format.

Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

pypi_packages

map<string, string>

Optional. Custom Python Package Index (PyPI) packages to be installed in the environment.

Keys refer to the lowercase package name such as "numpy" and values are the lowercase extras and version specifier such as "==1.12.0", "[devel,gcp_api]", or "[devel]>=1.8.2, <1.9.2". To specify a package without pinning it to a version specifier, use the empty string as the value.

env_variables

map<string, string>

Optional. Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes.

Environment variable names must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression AIRFLOW__[A-Z0-9_]+__[A-Z0-9_]+), and they cannot match any of the following reserved names:

  • AIRFLOW_HOME
  • C_FORCE_ROOT
  • CONTAINER_NAME
  • DAGS_FOLDER
  • GCP_PROJECT
  • GCS_BUCKET
  • GKE_CLUSTER_NAME
  • SQL_DATABASE
  • SQL_INSTANCE
  • SQL_PASSWORD
  • SQL_PROJECT
  • SQL_REGION
  • SQL_USER

UpdateEnvironmentRequest

Update an environment.

Fields
name

string

The relative resource name of the environment to update, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following Google IAM permission on the specified resource name:

  • composer.environments.update

environment

Environment

A patch environment. Fields specified by the updateMask will be copied from the patch environment into the environment under update.

update_mask

FieldMask

Required. A comma-separated list of paths, relative to Environment, of fields to update. For example, to set the version of scikit-learn to install in the environment to 0.19.0 and to remove an existing installation of argparse, the updateMask parameter would include the following two paths values: "config.softwareConfig.pypiPackages.scikit-learn" and "config.softwareConfig.pypiPackages.argparse". The included patch environment would specify the scikit-learn version as follows:

{
  "config":{
    "softwareConfig":{
      "pypiPackages":{
        "scikit-learn":"==0.19.0"
      }
    }
  }
}

Note that in the above example, any existing PyPI packages other than scikit-learn and argparse will be unaffected.

Only one update type may be included in a single request's updateMask. For example, one cannot update both the PyPI packages and labels in the same request. However, it is possible to update multiple members of a map field simultaneously in the same request. For example, to set the labels "label1" and "label2" while clearing "label3" (assuming it already exists), one can provide the paths "labels.label1", "labels.label2", and "labels.label3" and populate the patch environment as follows:

{
  "labels":{
    "label1":"new-label1-value"
    "label2":"new-label2-value"
  }
}

Note that in the above example, any existing labels that are not included in the updateMask will be unaffected.

It is also possible to replace an entire map field by providing the map field's path in the updateMask. The new value of the field will be that which is provided in the patch environment. For example, to delete all pre-existing user-specified PyPI packages and install botocore at version 1.7.14, the updateMask would contain the path "config.softwareConfig.pypiPackages", and the patch environment would be the following:

{
  "config":{
    "softwareConfig":{
      "pypiPackages":{
        "botocore":"==1.7.14"
      }
    }
  }
}

Note: Only the following fields can be updated:

Mask Purpose
config.softwareConfig.pypiPackages Replace all custom custom PyPI packages. If a replacement package map is not included in environment, all custom PyPI packages are cleared. It is an error to provide both this mask and a mask specifying an individual package.
config.softwareConfig.pypiPackages.packagename Update the custom PyPI package packagename, preserving other packages. To delete the package, include it in updateMask, and omit the mapping for it in environment.config.softwareConfig.pypiPackages. It is an error to provide both a mask of this form and the "config.softwareConfig.pypiPackages" mask.
labels Replace all environment labels. If a replacement labels map is not included in environment, all labels are cleared. It is an error to provide both this mask and a mask specifying one or more individual labels.
labels.labelName Set the label named labelName, while preserving other labels. To delete the label, include it in updateMask and omit its mapping in environment.labels. It is an error to provide both a mask of this form and the "labels" mask.
config.nodeCount Horizontally scale the number of nodes in the environment. An integer greater than or equal to 3 must be provided in the config.nodeCount field.
config.softwareConfig.airflowConfigOverrides Replace all Apache Airflow config overrides. If a replacement config overrides map is not included in environment, all config overrides are cleared. It is an error to provide both this mask and a mask specifying one or more individual config overrides.
config.softwareConfig.airflowConfigOverrides.section-name Override the Apache Airflow config property name in the section named section, preserving other properties. To delete the property override, include it in updateMask and omit its mapping in environment.config.softwareConfig.airflowConfigOverrides. It is an error to provide both a mask of this form and the "config.softwareConfig.airflowConfigOverrides" mask.
config.softwareConfig.envVariables Replace all environment variables. If a replacement environment variable map is not included in environment, all custom environment variables are cleared. It is an error to provide both this mask and a mask specifying one or more individual environment variables.

Was this page helpful? Let us know how we did:

Send feedback about...

Google Cloud Composer