Package google.cloud.orchestration.airflow.service.v1beta1

Index

Environments

Managed Apache Airflow Environments.

CheckUpgrade

rpc CheckUpgrade(CheckUpgradeRequest) returns (Operation)

Check if an upgrade operation on the environment will succeed.

In case of problems detailed info can be found in the returned Operation.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
CreateEnvironment

rpc CreateEnvironment(CreateEnvironmentRequest) returns (Operation)

Create a new environment.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
CreateUserWorkloadsConfigMap

rpc CreateUserWorkloadsConfigMap(CreateUserWorkloadsConfigMapRequest) returns (UserWorkloadsConfigMap)

Creates a user workloads ConfigMap.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
CreateUserWorkloadsSecret

rpc CreateUserWorkloadsSecret(CreateUserWorkloadsSecretRequest) returns (UserWorkloadsSecret)

Creates a user workloads Secret.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
DatabaseFailover

rpc DatabaseFailover(DatabaseFailoverRequest) returns (Operation)

Triggers database failover (only for highly resilient environments).

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
DeleteEnvironment

rpc DeleteEnvironment(DeleteEnvironmentRequest) returns (Operation)

Delete an environment.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
DeleteUserWorkloadsConfigMap

rpc DeleteUserWorkloadsConfigMap(DeleteUserWorkloadsConfigMapRequest) returns (Empty)

Deletes a user workloads ConfigMap.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
DeleteUserWorkloadsSecret

rpc DeleteUserWorkloadsSecret(DeleteUserWorkloadsSecretRequest) returns (Empty)

Deletes a user workloads Secret.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
ExecuteAirflowCommand

rpc ExecuteAirflowCommand(ExecuteAirflowCommandRequest) returns (ExecuteAirflowCommandResponse)

Executes Airflow CLI command.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
FetchDatabaseProperties

rpc FetchDatabaseProperties(FetchDatabasePropertiesRequest) returns (FetchDatabasePropertiesResponse)

Fetches database properties.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
GetEnvironment

rpc GetEnvironment(GetEnvironmentRequest) returns (Environment)

Get an existing environment.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
GetUserWorkloadsConfigMap

rpc GetUserWorkloadsConfigMap(GetUserWorkloadsConfigMapRequest) returns (UserWorkloadsConfigMap)

Gets an existing user workloads ConfigMap.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
GetUserWorkloadsSecret

rpc GetUserWorkloadsSecret(GetUserWorkloadsSecretRequest) returns (UserWorkloadsSecret)

Gets an existing user workloads Secret. Values of the "data" field in the response are cleared.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
ListEnvironments

rpc ListEnvironments(ListEnvironmentsRequest) returns (ListEnvironmentsResponse)

List environments.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
ListUserWorkloadsConfigMaps

rpc ListUserWorkloadsConfigMaps(ListUserWorkloadsConfigMapsRequest) returns (ListUserWorkloadsConfigMapsResponse)

Lists user workloads ConfigMaps.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
ListUserWorkloadsSecrets

rpc ListUserWorkloadsSecrets(ListUserWorkloadsSecretsRequest) returns (ListUserWorkloadsSecretsResponse)

Lists user workloads Secrets.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
ListWorkloads

rpc ListWorkloads(ListWorkloadsRequest) returns (ListWorkloadsResponse)

Lists workloads in a Cloud Composer environment. Workload is a unit that runs a single Composer component.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
LoadSnapshot

rpc LoadSnapshot(LoadSnapshotRequest) returns (Operation)

Loads a snapshot of a Cloud Composer environment.

As a result of this operation, a snapshot of environment's specified in LoadSnapshotRequest is loaded into the environment.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
PollAirflowCommand

rpc PollAirflowCommand(PollAirflowCommandRequest) returns (PollAirflowCommandResponse)

Polls Airflow CLI command execution and fetches logs.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
RestartWebServer

rpc RestartWebServer(RestartWebServerRequest) returns (Operation)

Restart Airflow web server.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
SaveSnapshot

rpc SaveSnapshot(SaveSnapshotRequest) returns (Operation)

Creates a snapshots of a Cloud Composer environment.

As a result of this operation, snapshot of environment's state is stored in a location specified in the SaveSnapshotRequest.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
StopAirflowCommand

rpc StopAirflowCommand(StopAirflowCommandRequest) returns (StopAirflowCommandResponse)

Stops Airflow CLI command execution.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
UpdateEnvironment

rpc UpdateEnvironment(UpdateEnvironmentRequest) returns (Operation)

Update an environment.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
UpdateUserWorkloadsConfigMap

rpc UpdateUserWorkloadsConfigMap(UpdateUserWorkloadsConfigMapRequest) returns (UserWorkloadsConfigMap)

Updates a user workloads ConfigMap.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.
UpdateUserWorkloadsSecret

rpc UpdateUserWorkloadsSecret(UpdateUserWorkloadsSecretRequest) returns (UserWorkloadsSecret)

Updates a user workloads Secret.

This method is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ImageVersions

Readonly service to query available ImageVersions.

ListImageVersions

rpc ListImageVersions(ListImageVersionsRequest) returns (ListImageVersionsResponse)

List ImageVersions for provided location.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

AirflowMetadataRetentionPolicyConfig

The policy for airflow metadata database retention.

Fields
retention_mode

RetentionMode

Optional. Retention can be either enabled or disabled.
retention_days

int32

Optional. How many days data should be retained for.

RetentionMode

Describes retention policy.

Enums
RETENTION_MODE_UNSPECIFIED Default mode doesn't change environment parameters.
RETENTION_MODE_ENABLED Retention policy is enabled.
RETENTION_MODE_DISABLED Retention policy is disabled.

CheckUpgradeRequest

Request to check whether image upgrade will succeed.

Fields
environment

string

The resource name of the environment to check upgrade for, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.update
image_version

string

The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-([0-9]+(\.[0-9]+\.[0-9]+(-preview\.[0-9]+)?)?|latest)-airflow-([0-9]+(\.[0-9]+(\.[0-9]+)?)?). When used as input, the server also checks if the provided version is supported and denies the request for an unsupported version.

The Cloud Composer portion of the image version is a full semantic version, or an alias in the form of major version number or latest. When an alias is provided, the server replaces it with the current Cloud Composer version that satisfies the alias.

The Apache Airflow portion of the image version is a full semantic version that points to one of the supported Apache Airflow versions, or an alias in the form of only major or major.minor versions specified. When an alias is provided, the server replaces it with the latest Apache Airflow version that satisfies the alias and is supported in the given Cloud Composer version.

In all cases, the resolved image version is stored in the same field.

See also version list and versioning overview.

CheckUpgradeResponse

Message containing information about the result of an upgrade check operation.

Fields
build_log_uri

string

Output only. Url for a docker build log of an upgraded image.
contains_pypi_modules_conflict

ConflictResult

Output only. Whether build has succeeded or failed on modules conflicts.
pypi_conflict_build_log_extract

string

Output only. Extract from a docker image build log containing information about pypi modules conflicts.
image_version

string

Composer image for which the build was happening.
pypi_dependencies

map<string, string>

Pypi dependencies specified in the environment configuration, at the time when the build was triggered.

ConflictResult

Whether there were python modules conflict during image build.

Enums
CONFLICT_RESULT_UNSPECIFIED It is unknown whether build had conflicts or not.
CONFLICT There were python packages conflicts.
NO_CONFLICT There were no python packages conflicts.

CloudDataLineageIntegration

Configuration for Cloud Data Lineage integration.

Fields
enabled

bool

Optional. Whether or not Cloud Data Lineage integration is enabled.

CreateEnvironmentRequest

Create a new environment.

Fields
parent

string

The parent must be of the form "projects/{projectId}/locations/{locationId}".

Authorization requires the following IAM permission on the specified resource parent:

  • composer.environments.create
environment

Environment

The environment to create.

CreateUserWorkloadsConfigMapRequest

Create user workloads ConfigMap request.

Fields
parent

string

Required. The environment name to create a ConfigMap for, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.userworkloadsconfigmaps.create
user_workloads_config_map

UserWorkloadsConfigMap

Required. User workloads ConfigMap to create.

CreateUserWorkloadsSecretRequest

Create user workloads Secret request.

Fields
parent

string

Required. The environment name to create a Secret for, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.userworkloadssecrets.create
user_workloads_secret

UserWorkloadsSecret

Required. User workloads Secret to create.

DataRetentionConfig

The configuration setting for Airflow database data retention mechanism.

Fields
airflow_database_retention_days
(deprecated)

int32

Optional. The number of days describing for how long to store event-based records in airflow database. If the retention mechanism is enabled this value must be a positive integer otherwise, value should be set to 0.
task_logs_retention_config

TaskLogsRetentionConfig

Optional. The configuration settings for task logs retention
airflow_metadata_retention_config

AirflowMetadataRetentionPolicyConfig

Optional. The retention policy for airflow metadata database.

DatabaseConfig

The configuration of Cloud SQL instance that is used by the Apache Airflow software.

Fields
machine_type

string

Optional. Cloud SQL machine type used by Airflow database. It has to be one of: db-n1-standard-2, db-n1-standard-4, db-n1-standard-8 or db-n1-standard-16. If not specified, db-n1-standard-2 will be used. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
zone

string

Optional. The Compute Engine zone where the Airflow database is created. If zone is provided, it must be in the region selected for the environment. If zone is not provided, a zone is automatically selected. The zone can only be set during environment creation. Supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.*.

DatabaseFailoverRequest

Request to trigger database failover (only for highly resilient environments).

Fields
environment

string

Target environment: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.update

DatabaseFailoverResponse

This type has no fields.

Response for DatabaseFailoverRequest.

DeleteEnvironmentRequest

Delete an environment.

Fields
name

string

The environment to delete, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.environments.delete

DeleteUserWorkloadsConfigMapRequest

Delete user workloads ConfigMap request.

Fields
name

string

Required. The ConfigMap to delete, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsConfigMaps/{userWorkloadsConfigMapId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.userworkloadsconfigmaps.delete

DeleteUserWorkloadsSecretRequest

Delete user workloads Secret request.

Fields
name

string

Required. The Secret to delete, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsSecrets/{userWorkloadsSecretId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.userworkloadssecrets.delete

EncryptionConfig

The encryption options for the Cloud Composer environment and its dependencies. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

Fields
kms_key_name

string

Optional. Customer-managed Encryption Key available through Google's Key Management Service. Cannot be updated. If not specified, Google-managed key will be used.

Environment

An environment for running orchestration tasks.

Fields
name

string

Identifier. The resource name of the environment, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

EnvironmentId must start with a lowercase letter followed by up to 63 lowercase letters, numbers, or hyphens, and cannot end with a hyphen.
config

EnvironmentConfig

Optional. Configuration parameters for this environment.
uuid

string

Output only. The UUID (Universally Unique IDentifier) associated with this environment. This value is generated when the environment is created.
state

State

The current state of the environment.
create_time

Timestamp

Output only. The time at which this environment was created.
update_time

Timestamp

Output only. The time at which this environment was last modified.
labels

map<string, string>

Optional. User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions:

  • Keys must conform to regexp: [\p{Ll}\p{Lo}][\p{Ll}\p{Lo}\p{N}_-]{0,62}
  • Values must conform to regexp: [\p{Ll}\p{Lo}\p{N}_-]{0,63}
  • Both keys and values are additionally constrained to be <= 128 bytes in size.
satisfies_pzs

bool

Output only. Reserved for future use.
satisfies_pzi

bool

Output only. Reserved for future use.
storage_config

StorageConfig

Optional. Storage configuration for this environment.

State

State of the environment.

Enums
STATE_UNSPECIFIED The state of the environment is unknown.
CREATING The environment is in the process of being created.
RUNNING The environment is currently running and healthy. It is ready for use.
UPDATING The environment is being updated. It remains usable but cannot receive additional update requests or be deleted at this time.
DELETING The environment is undergoing deletion. It cannot be used.
ERROR The environment has encountered an error and cannot be used.

EnvironmentConfig

Configuration information for an environment.

Fields
gke_cluster

string

Output only. The Kubernetes Engine cluster used to run this environment.
dag_gcs_prefix

string

Output only. The Cloud Storage prefix of the DAGs for this environment. Although Cloud Storage objects reside in a flat namespace, a hierarchical file tree can be simulated using "/"-delimited object name prefixes. DAG objects for this environment reside in a simulated directory with the given prefix.
node_count

int32

The number of nodes in the Kubernetes Engine cluster that will be used to run this environment.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
software_config

SoftwareConfig

Optional. The configuration settings for software inside the environment.
node_config

NodeConfig

Optional. The configuration used for the Kubernetes Engine cluster.
private_environment_config

PrivateEnvironmentConfig

Optional. The configuration used for the Private IP Cloud Composer environment.
web_server_network_access_control

WebServerNetworkAccessControl

Optional. The network-level access control policy for the Airflow web server. If unspecified, no network-level access restrictions will be applied.
database_config

DatabaseConfig

Optional. The configuration settings for Cloud SQL instance used internally by Apache Airflow software.
web_server_config

WebServerConfig

Optional. The configuration settings for the Airflow web server App Engine instance.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
airflow_uri

string

Output only. The URI of the Apache Airflow Web UI hosted within this environment (see Airflow web interface).
airflow_byoid_uri

string

Output only. The 'bring your own identity' variant of the URI of the Apache Airflow Web UI hosted within this environment, to be accessed with external identities using workforce identity federation (see Access environments with workforce identity federation).
encryption_config

EncryptionConfig

Optional. The encryption options for the Cloud Composer environment and its dependencies. Cannot be updated.
maintenance_window

MaintenanceWindow

Optional. The maintenance window is the period when Cloud Composer components may undergo maintenance. It is defined so that maintenance is not executed during peak hours or critical time periods.

The system will not be under maintenance for every occurrence of this window, but when maintenance is planned, it will be scheduled during the window.

The maintenance window period must encompass at least 12 hours per week. This may be split into multiple chunks, each with a size of at least 4 hours.

If this value is omitted, the default value for maintenance window is applied. By default, maintenance windows are from 00:00:00 to 04:00:00 (GMT) on Friday, Saturday, and Sunday every week.
workloads_config

WorkloadsConfig

Optional. The workloads configuration settings for the GKE cluster associated with the Cloud Composer environment. The GKE cluster runs Airflow scheduler, web server and workers workloads.

This field is supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
environment_size

EnvironmentSize

Optional. The size of the Cloud Composer environment.

This field is supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
master_authorized_networks_config

MasterAuthorizedNetworksConfig

Optional. The configuration options for GKE cluster master authorized networks. By default master authorized networks feature is: - in case of private environment: enabled with no external networks allowlisted. - in case of public environment: disabled.
recovery_config

RecoveryConfig

Optional. The Recovery settings configuration of an environment.

This field is supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
data_retention_config

DataRetentionConfig

Optional. The configuration setting for Airflow database data retention mechanism.
resilience_mode

ResilienceMode

Optional. Resilience mode of the Cloud Composer Environment.

This field is supported for Cloud Composer environments in versions composer-2.2.0-airflow-*.*.* and newer.

EnvironmentSize

The size of the Cloud Composer environment.

Enums
ENVIRONMENT_SIZE_UNSPECIFIED The size of the environment is unspecified.
ENVIRONMENT_SIZE_SMALL The environment size is small.
ENVIRONMENT_SIZE_MEDIUM The environment size is medium.
ENVIRONMENT_SIZE_LARGE The environment size is large.

ResilienceMode

Resilience mode of the Cloud Composer Environment.

Enums
RESILIENCE_MODE_UNSPECIFIED Default mode doesn't change environment parameters.
HIGH_RESILIENCE Enabled High Resilience mode, including Cloud SQL HA.

ExecuteAirflowCommandRequest

Execute Airflow Command request.

Fields
environment

string

The resource name of the environment in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}".

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.executeAirflowCommand
command

string

Airflow command.
subcommand

string

Airflow subcommand.
parameters[]

string

Parameters for the Airflow command/subcommand as an array of arguments. It may contain positional arguments like ["my-dag-id"], key-value parameters like ["--foo=bar"] or ["--foo","bar"], or other flags like ["-f"].

ExecuteAirflowCommandResponse

Response to ExecuteAirflowCommandRequest.

Fields
execution_id

string

The unique ID of the command execution for polling.
pod

string

The name of the pod where the command is executed.
pod_namespace

string

The namespace of the pod where the command is executed.
error

string

Error message. Empty if there was no error.

FetchDatabasePropertiesRequest

Request to fetch properties of environment's database.

Fields
environment

string

Required. The resource name of the environment, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.get

FetchDatabasePropertiesResponse

Response for FetchDatabasePropertiesRequest.

Fields
primary_gce_zone

string

The Compute Engine zone that the instance is currently serving from.
secondary_gce_zone

string

The Compute Engine zone that the failover instance is currently serving from for a regional Cloud SQL instance.
is_failover_replica_available

bool

The availability status of the failover replica. A false status indicates that the failover replica is out of sync. The primary instance can only fail over to the failover replica when the status is true.

GetEnvironmentRequest

Get an environment.

Fields
name

string

The resource name of the environment to get, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.environments.get

GetUserWorkloadsConfigMapRequest

Get user workloads ConfigMap request.

Fields
name

string

Required. The resource name of the ConfigMap to get, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsConfigMaps/{userWorkloadsConfigMapId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.userworkloadsconfigmaps.get

GetUserWorkloadsSecretRequest

Get user workloads Secret request.

Fields
name

string

Required. The resource name of the Secret to get, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsSecrets/{userWorkloadsSecretId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.userworkloadssecrets.get

IPAllocationPolicy

Configuration for controlling how IPs are allocated in the GKE cluster.

Fields
use_ip_aliases

bool

Optional. Whether or not to enable Alias IPs in the GKE cluster. If true, a VPC-native cluster is created.

This field is only supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*. Environments in newer versions always use VPC-native GKE clusters.
cluster_secondary_range_name

string

Optional. The name of the cluster's secondary range used to allocate IP addresses to pods. Specify either cluster_secondary_range_name or cluster_ipv4_cidr_block but not both.

For Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*, this field is applicable only when use_ip_aliases is true.
services_secondary_range_name

string

Optional. The name of the services' secondary range used to allocate IP addresses to the cluster. Specify either services_secondary_range_name or services_ipv4_cidr_block but not both.

For Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*, this field is applicable only when use_ip_aliases is true.
cluster_ipv4_cidr_block

string

Optional. The IP address range used to allocate IP addresses to pods in the cluster.

For Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*, this field is applicable only when use_ip_aliases is true.

Set to blank to have GKE choose a range with the default size.

Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask.

Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify cluster_secondary_range_name or cluster_ipv4_cidr_block but not both.
services_ipv4_cidr_block

string

Optional. The IP address range of the services IP addresses in this cluster.

For Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*, this field is applicable only when use_ip_aliases is true.

Set to blank to have GKE choose a range with the default size.

Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask.

Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify services_secondary_range_name or services_ipv4_cidr_block but not both.

ImageVersion

Image Version information

Fields
image_version_id

string

The string identifier of the ImageVersion, in the form: "composer-x.y.z-airflow-a.b.c"
is_default

bool

Whether this is the default ImageVersion used by Composer during environment creation if no input ImageVersion is specified.
supported_python_versions[]

string

supported python versions
release_date

Date

The date of the version release.
creation_disabled

bool

Whether it is impossible to create an environment with the image version.
upgrade_disabled

bool

Whether it is impossible to upgrade an environment running with the image version.

ListEnvironmentsRequest

List environments in a project and location.

Fields
parent

string

List environments in the given project and location, in the form: "projects/{projectId}/locations/{locationId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.environments.list
page_size

int32

The maximum number of environments to return.
page_token

string

The next_page_token value returned from a previous List request, if any.

ListEnvironmentsResponse

The environments in a project and location.

Fields
environments[]

Environment

The list of environments returned by a ListEnvironmentsRequest.
next_page_token

string

The page token used to query for the next page if one exists.

ListImageVersionsRequest

List ImageVersions in a project and location.

Fields
parent

string

List ImageVersions in the given project and location, in the form: "projects/{projectId}/locations/{locationId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.imageversions.list
page_size

int32

The maximum number of image_versions to return.
page_token

string

The next_page_token value returned from a previous List request, if any.
include_past_releases

bool

Whether or not image versions from old releases should be included.

ListImageVersionsResponse

The ImageVersions in a project and location.

Fields
image_versions[]

ImageVersion

The list of supported ImageVersions in a location.
next_page_token

string

The page token used to query for the next page if one exists.

ListUserWorkloadsConfigMapsRequest

List user workloads ConfigMaps request.

Fields
parent

string

Required. List ConfigMaps in the given environment, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.userworkloadsconfigmaps.list
page_size

int32

Optional. The maximum number of ConfigMaps to return.
page_token

string

Optional. The next_page_token value returned from a previous List request, if any.

ListUserWorkloadsConfigMapsResponse

The user workloads ConfigMaps for a given environment.

Fields
user_workloads_config_maps[]

UserWorkloadsConfigMap

The list of ConfigMaps returned by a ListUserWorkloadsConfigMapsRequest.
next_page_token

string

The page token used to query for the next page if one exists.

ListUserWorkloadsSecretsRequest

List user workloads Secrets request.

Fields
parent

string

Required. List Secrets in the given environment, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.userworkloadssecrets.list
page_size

int32

Optional. The maximum number of Secrets to return.
page_token

string

Optional. The next_page_token value returned from a previous List request, if any.

ListUserWorkloadsSecretsResponse

The user workloads Secrets for a given environment.

Fields
user_workloads_secrets[]

UserWorkloadsSecret

The list of Secrets returned by a ListUserWorkloadsSecretsRequest.
next_page_token

string

The page token used to query for the next page if one exists.

ListWorkloadsRequest

Request for listing workloads in a Cloud Composer environment.

Fields
parent

string

Required. The environment name to get workloads for, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource parent:

  • composer.environments.get
page_size

int32

Optional. The maximum number of environments to return.
page_token

string

Optional. The next_page_token value returned from a previous List request, if any.
filter

string

Optional. The list filter. Currently only supports equality on the type field. The value of a field specified in the filter expression must be one ComposerWorkloadType enum option. It's possible to get multiple types using "OR" operator, e.g.: "type=SCHEDULER OR type=CELERY_WORKER". If not specified, all items are returned.

ListWorkloadsResponse

Response to ListWorkloadsRequest.

Fields
workloads[]

ComposerWorkload

The list of environment workloads.
next_page_token

string

The page token used to query for the next page if one exists.

ComposerWorkload

Information about a single workload.

Fields
name

string

Name of a workload.
type

ComposerWorkloadType

Type of a workload.
status

ComposerWorkloadStatus

Output only. Status of a workload.

ComposerWorkloadState

Workload states.

Enums
COMPOSER_WORKLOAD_STATE_UNSPECIFIED Not able to determine the status of the workload.
PENDING Workload is in pending state and has not yet started.
OK Workload is running fine.
WARNING Workload is running but there are some non-critical problems.
ERROR Workload is not running due to an error.
SUCCEEDED Workload has finished execution with success.
FAILED Workload has finished execution with failure.

ComposerWorkloadStatus

Workload status.

Fields
state

ComposerWorkloadState

Output only. Workload state.
status_message

string

Output only. Text to provide more descriptive status.
detailed_status_message

string

Output only. Detailed message of the status.

ComposerWorkloadType

Supported workload types.

Enums
COMPOSER_WORKLOAD_TYPE_UNSPECIFIED Not able to determine the type of the workload.
CELERY_WORKER Celery worker.
KUBERNETES_WORKER Kubernetes worker.
KUBERNETES_OPERATOR_POD Workload created by Kubernetes Pod Operator.
SCHEDULER Airflow scheduler.
DAG_PROCESSOR Airflow Dag processor.
TRIGGERER Airflow triggerer.
WEB_SERVER Airflow web server UI.
REDIS Redis.

LoadSnapshotRequest

Request to load a snapshot into a Cloud Composer environment.

Fields
environment

string

The resource name of the target environment in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.update
snapshot_path

string

A Cloud Storage path to a snapshot to load, e.g.: "gs://my-bucket/snapshots/project_location_environment_timestamp".
skip_pypi_packages_installation

bool

Whether or not to skip installing Pypi packages when loading the environment's state.
skip_environment_variables_setting

bool

Whether or not to skip setting environment variables when loading the environment's state.
skip_airflow_overrides_setting

bool

Whether or not to skip setting Airflow overrides when loading the environment's state.
skip_gcs_data_copying

bool

Whether or not to skip copying Cloud Storage data when loading the environment's state.

LoadSnapshotResponse

This type has no fields.

Response to LoadSnapshotRequest.

MaintenanceWindow

The configuration settings for Cloud Composer maintenance window. The following example:

   {
     "startTime":"2019-08-01T01:00:00Z"
     "endTime":"2019-08-01T07:00:00Z"
     "recurrence":"FREQ=WEEKLY;BYDAY=TU,WE"
   }

would define a maintenance window between 01 and 07 hours UTC during each Tuesday and Wednesday.

Fields
start_time

Timestamp

Required. Start time of the first recurrence of the maintenance window.
end_time

Timestamp

Required. Maintenance window end time. It is used only to calculate the duration of the maintenance window. The value for end_time must be in the future, relative to start_time.
recurrence

string

Required. Maintenance window recurrence. Format is a subset of RFC-5545 RRULE. The only allowed values for FREQ field are FREQ=DAILY and FREQ=WEEKLY;BYDAY=... Example values: FREQ=WEEKLY;BYDAY=TU,WE, FREQ=DAILY.

MasterAuthorizedNetworksConfig

Configuration options for the master authorized networks feature. Enabled master authorized networks will disallow all external traffic to access Kubernetes master through HTTPS except traffic from the given CIDR blocks, Google Compute Engine Public IPs and Google Prod IPs.

Fields
enabled

bool

Whether or not master authorized networks feature is enabled.
cidr_blocks[]

CidrBlock

Up to 50 external networks that could access Kubernetes master through HTTPS.

CidrBlock

CIDR block with an optional name.

Fields
display_name

string

User-defined name that identifies the CIDR block.
cidr_block

string

CIDR block that must be specified in CIDR notation.

NetworkingConfig

Configuration options for networking connections in the Composer 2 environment.

Fields
connection_type

ConnectionType

Optional. Indicates the user requested specifc connection type between Tenant and Customer projects. You cannot set networking connection type in public IP environment.

ConnectionType

Represents connection type between Composer environment in Customer Project and the corresponding Tenant project, from a predefined list of available connection modes.

Enums
CONNECTION_TYPE_UNSPECIFIED No specific connection type was requested, so the environment uses the default value corresponding to the rest of its configuration.
VPC_PEERING Requests the use of VPC peerings for connecting the Customer and Tenant projects.
PRIVATE_SERVICE_CONNECT Requests the use of Private Service Connect for connecting the Customer and Tenant projects.

NodeConfig

The configuration information for the Kubernetes Engine nodes running the Apache Airflow software.

Fields
location

string

Optional. The Compute Engine zone in which to deploy the VMs used to run the Apache Airflow software, specified as a relative resource name. For example: "projects/{projectId}/zones/{zoneId}".

This location must belong to the enclosing environment's project and location. If both this field and nodeConfig.machineType are specified, nodeConfig.machineType must belong to this location; if both are unspecified, the service will pick a zone in the Compute Engine region corresponding to the Cloud Composer location, and propagate that choice to both fields. If only one field (location or nodeConfig.machineType) is specified, the location information from the specified field will be propagated to the unspecified field.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
machine_type

string

Optional. The Compute Engine machine type used for cluster instances, specified as a relative resource name. For example: "projects/{projectId}/zones/{zoneId}/machineTypes/{machineTypeId}".

The machineType must belong to the enclosing environment's project and location. If both this field and nodeConfig.location are specified, this machineType must belong to the nodeConfig.location; if both are unspecified, the service will pick a zone in the Compute Engine region corresponding to the Cloud Composer location, and propagate that choice to both fields. If exactly one of this field and nodeConfig.location is specified, the location information from the specified field will be propagated to the unspecified field.

The machineTypeId must not be a shared-core machine type.

If this field is unspecified, the machineTypeId defaults to "n1-standard-1".

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
network

string

Optional. The Compute Engine network to be used for machine communications, specified as a relative resource name. For example: "projects/{projectId}/global/networks/{networkId}".

If unspecified, the default network in the environment's project is used. If a Custom Subnet Network is provided, nodeConfig.subnetwork must also be provided. For Shared VPC subnetwork requirements, see nodeConfig.subnetwork.
subnetwork

string

Optional. The Compute Engine subnetwork to be used for machine communications, specified as a relative resource name. For example: "projects/{projectId}/regions/{regionId}/subnetworks/{subnetworkId}"

If a subnetwork is provided, nodeConfig.network must also be provided, and the subnetwork must belong to the enclosing environment's project and location.
disk_size_gb

int32

Optional. The disk size in GB used for node VMs. Minimum size is 30GB. If unspecified, defaults to 100GB. Cannot be updated.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
oauth_scopes[]

string

Optional. The set of Google API scopes to be made available on all node VMs. If oauth_scopes is empty, defaults to ["https://www.googleapis.com/auth/cloud-platform"]. Cannot be updated.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
service_account

string

Optional. The Google Cloud Platform Service Account to be used by the workloads. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated.
tags[]

string

Optional. The list of instance tags applied to all node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC1035. Cannot be updated.
ip_allocation_policy

IPAllocationPolicy

Optional. The IPAllocationPolicy fields for the GKE cluster.
max_pods_per_node

int32

Optional. The maximum number of pods per node in the Cloud Composer GKE cluster. The value must be between 8 and 110 and it can be set only if the environment is VPC-native. The default value is 32. Values of this field will be propagated both to the default-pool node pool of the newly created GKE cluster, and to the default "Maximum Pods per Node" value which is used for newly created node pools if their value is not explicitly set during node pool creation. For more information, see Optimizing IP address allocation. Cannot be updated.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
enable_ip_masq_agent

bool

Optional. Deploys 'ip-masq-agent' daemon set in the GKE cluster and defines nonMasqueradeCIDRs equals to pod IP range so IP masquerading is used for all destination addresses, except between pods traffic.

See: https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
composer_network_attachment

string

Optional. Network Attachment that Cloud Composer environment is connected to, which provides connectivity with a user's VPC network. Takes precedence over network and subnetwork settings. If not provided, but network and subnetwork are defined during environment, it will be provisioned. If not provided and network and subnetwork are also empty, then connectivity to user's VPC network is disabled. Network attachment must be provided in format projects/{project}/regions/{region}/networkAttachments/{networkAttachment}.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.
composer_internal_ipv4_cidr_block

string

Optional. The IP range in CIDR notation to use internally by Cloud Composer. IP addresses are not reserved - and the same range can be used by multiple Cloud Composer environments. In case of overlap, IPs from this range will not be accessible in the user's VPC network. Cannot be updated. If not specified, the default value of '100.64.128.0/20' is used.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

OperationMetadata

Metadata describing an operation.

Fields
state

State

Output only. The current operation state.
operation_type

Type

Output only. The type of operation being performed.
resource

string

Output only. The resource being operated on, as a [relative resource name]( /apis/design/resource_names#relative_resource_name).
resource_uuid

string

Output only. The UUID of the resource being operated on.
create_time

Timestamp

Output only. The time the operation was submitted to the server.
end_time

Timestamp

Output only. The time when the operation terminated, regardless of its success. This field is unset if the operation is still ongoing.

State

An enum describing the overall state of an operation.

Enums
STATE_UNSPECIFIED Unused.
PENDING The operation has been created but is not yet started.
RUNNING The operation is underway.
SUCCESSFUL The operation completed successfully.
FAILED The operation is no longer running but did not succeed.

Type

Type of longrunning operation.

Enums
TYPE_UNSPECIFIED Unused.
CREATE A resource creation operation.
DELETE A resource deletion operation.
UPDATE A resource update operation.
CHECK A resource check operation.
SAVE_SNAPSHOT Saves snapshot of the resource operation.
LOAD_SNAPSHOT Loads snapshot of the resource operation.
DATABASE_FAILOVER Triggers failover of environment's Cloud SQL instance (only for highly resilient environments).

PollAirflowCommandRequest

Poll Airflow Command request.

Fields
environment

string

The resource name of the environment in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.executeAirflowCommand
execution_id

string

The unique ID of the command execution.
pod

string

The name of the pod where the command is executed.
pod_namespace

string

The namespace of the pod where the command is executed.
next_line_number

int32

Line number from which new logs should be fetched.

PollAirflowCommandResponse

Response to PollAirflowCommandRequest.

Fields
output[]

Line

Output from the command execution. It may not contain the full output and the caller may need to poll for more lines.
output_end

bool

Whether the command execution has finished and there is no more output.
exit_info

ExitInfo

The result exit status of the command.

ExitInfo

Information about how a command ended.

Fields
exit_code

int32

The exit code from the command execution.
error

string

Error message. Empty if there was no error.

Line

Contains information about a single line from logs.

Fields
line_number

int32

Number of the line.
content

string

Text content of the log line.

PrivateClusterConfig

Configuration options for the private GKE cluster in a Cloud Composer environment.

Fields
enable_private_endpoint

bool

Optional. If true, access to the public endpoint of the GKE cluster is denied.
master_ipv4_cidr_block

string

Optional. The CIDR block from which IPv4 range for GKE master will be reserved. If left blank, the default value of '172.16.0.0/23' is used.
master_ipv4_reserved_range

string

Output only. The IP range in CIDR notation to use for the hosted master network. This range is used for assigning internal IP addresses to the cluster master or set of masters and to the internal load balancer virtual IP. This range must not overlap with any other ranges in use within the cluster's network.

PrivateEnvironmentConfig

The configuration information for configuring a Private IP Cloud Composer environment.

Fields
enable_private_environment

bool

Optional. If true, a Private IP Cloud Composer environment is created. If this field is set to true, IPAllocationPolicy.use_ip_aliases must be set to true for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
enable_private_builds_only

bool

Optional. If true, builds performed during operations that install Python packages have only private connectivity to Google services (including Artifact Registry) and VPC network (if either NodeConfig.network and NodeConfig.subnetwork fields or NodeConfig.composer_network_attachment field are specified). If false, the builds also have access to the internet.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.
private_cluster_config

PrivateClusterConfig

Optional. Configuration for the private GKE cluster for a Private IP Cloud Composer environment.
web_server_ipv4_cidr_block

string

Optional. The CIDR block from which IP range for web server will be reserved. Needs to be disjoint from private_cluster_config.master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
cloud_sql_ipv4_cidr_block

string

Optional. The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. Needs to be disjoint from web_server_ipv4_cidr_block
web_server_ipv4_reserved_range

string

Output only. The IP range reserved for the tenant project's App Engine VMs.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
cloud_composer_network_ipv4_cidr_block

string

Optional. The CIDR block from which IP range for Cloud Composer Network in tenant project will be reserved. Needs to be disjoint from private_cluster_config.master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block.

This field is supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
cloud_composer_network_ipv4_reserved_range

string

Output only. The IP range reserved for the tenant project's Cloud Composer network.

This field is supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
enable_privately_used_public_ips

bool

Optional. When enabled, IPs from public (non-RFC1918) ranges can be used for IPAllocationPolicy.cluster_ipv4_cidr_block and IPAllocationPolicy.service_ipv4_cidr_block.
cloud_composer_connection_subnetwork

string

Optional. When specified, the environment will use Private Service Connect instead of VPC peerings to connect to Cloud SQL in the Tenant Project, and the PSC endpoint in the Customer Project will use an IP address from this subnetwork.
networking_config

NetworkingConfig

Optional. Configuration for the network connections configuration in the environment.

RecoveryConfig

The Recovery settings of an environment.

Fields
scheduled_snapshots_config

ScheduledSnapshotsConfig

Optional. The configuration for scheduled snapshot creation mechanism.

RestartWebServerRequest

Restart Airflow web server.

Fields
name

string

The resource name of the environment to restart the web server for, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.environments.update

SaveSnapshotRequest

Request to create a snapshot of a Cloud Composer environment.

Fields
environment

string

The resource name of the source environment in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.update
snapshot_location

string

Location in a Cloud Storage where the snapshot is going to be stored, e.g.: "gs://my-bucket/snapshots".

SaveSnapshotResponse

Response to SaveSnapshotRequest.

Fields
snapshot_path

string

The fully-resolved Cloud Storage path of the created snapshot, e.g.: "gs://my-bucket/snapshots/project_location_environment_timestamp". This field is populated only if the snapshot creation was successful.

ScheduledSnapshotsConfig

The configuration for scheduled snapshot creation mechanism.

Fields
enabled

bool

Optional. Whether scheduled snapshots creation is enabled.
snapshot_location

string

Optional. The Cloud Storage location for storing automatically created snapshots.
snapshot_creation_schedule

string

Optional. The cron expression representing the time when snapshots creation mechanism runs. This field is subject to additional validation around frequency of execution.
time_zone

string

Optional. Time zone that sets the context to interpret snapshot_creation_schedule.

SoftwareConfig

Specifies the selection and configuration of software inside the environment.

Fields
image_version

string

Optional. The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-([0-9]+(\.[0-9]+\.[0-9]+(-preview\.[0-9]+)?)?|latest)-airflow-([0-9]+(\.[0-9]+(\.[0-9]+)?)?). When used as input, the server also checks if the provided version is supported and denies the request for an unsupported version.

The Cloud Composer portion of the image version is a full semantic version, or an alias in the form of major version number or latest. When an alias is provided, the server replaces it with the current Cloud Composer version that satisfies the alias.

The Apache Airflow portion of the image version is a full semantic version that points to one of the supported Apache Airflow versions, or an alias in the form of only major or major.minor versions specified. When an alias is provided, the server replaces it with the latest Apache Airflow version that satisfies the alias and is supported in the given Cloud Composer version.

In all cases, the resolved image version is stored in the same field.

See also version list and versioning overview.
airflow_config_overrides

map<string, string>

Optional. Apache Airflow configuration properties to override.

Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation". Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and must not contain an equals sign ("=") or semicolon (";"). Section and property names must not contain a period ("."). Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format.

Certain Apache Airflow configuration property values are blocked, and cannot be overridden.
pypi_packages

map<string, string>

Optional. Custom Python Package Index (PyPI) packages to be installed in the environment.

Keys refer to the lowercase package name such as "numpy" and values are the lowercase extras and version specifier such as "==1.12.0", "[devel,gcp_api]", or "[devel]>=1.8.2, <1.9.2". To specify a package without pinning it to a version specifier, use the empty string as the value.
env_variables

map<string, string>

Optional. Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes.

Environment variable names must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression AIRFLOW__[A-Z0-9_]+__[A-Z0-9_]+), and they cannot match any of the following reserved names:

  • AIRFLOW_HOME
  • C_FORCE_ROOT
  • CONTAINER_NAME
  • DAGS_FOLDER
  • GCP_PROJECT
  • GCS_BUCKET
  • GKE_CLUSTER_NAME
  • SQL_DATABASE
  • SQL_INSTANCE
  • SQL_PASSWORD
  • SQL_PROJECT
  • SQL_REGION
  • SQL_USER
python_version

string

Optional. The major version of Python used to run the Apache Airflow scheduler, worker, and webserver processes.

Can be set to '2' or '3'. If not specified, the default is '3'. Cannot be updated.

This field is only supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*. Environments in newer versions always use Python major version 3.
scheduler_count

int32

Optional. The number of schedulers for Airflow.

This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-2.*.*.
cloud_data_lineage_integration

CloudDataLineageIntegration

Optional. The configuration for Cloud Data Lineage integration.
web_server_plugins_mode

WebServerPluginsMode

Optional. Whether or not the web server uses custom plugins. If unspecified, the field defaults to PLUGINS_ENABLED.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

WebServerPluginsMode

Web server plugins mode of the Cloud Composer environment.

Enums
WEB_SERVER_PLUGINS_MODE_UNSPECIFIED Default mode.
PLUGINS_DISABLED Web server plugins are not supported.
PLUGINS_ENABLED Web server plugins are supported.

StopAirflowCommandRequest

Stop Airflow Command request.

Fields
environment

string

The resource name of the environment in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}".

Authorization requires the following IAM permission on the specified resource environment:

  • composer.environments.executeAirflowCommand
execution_id

string

The unique ID of the command execution.
pod

string

The name of the pod where the command is executed.
pod_namespace

string

The namespace of the pod where the command is executed.
force

bool

If true, the execution is terminated forcefully (SIGKILL). If false, the execution is stopped gracefully, giving it time for cleanup.

StopAirflowCommandResponse

Response to StopAirflowCommandRequest.

Fields
is_done

bool

Whether the execution is still running.
output[]

string

Output message from stopping execution request.

StorageConfig

The configuration for data storage in the environment.

Fields
bucket

string

Optional. The name of the Cloud Storage bucket used by the environment. No gs:// prefix.

TaskLogsRetentionConfig

The configuration setting for Task Logs.

Fields
storage_mode

TaskLogsStorageMode

Optional. The mode of storage for Airflow workers task logs.

TaskLogsStorageMode

The definition of task_logs_storage_mode.

Enums
TASK_LOGS_STORAGE_MODE_UNSPECIFIED This configuration is not specified by the user.
CLOUD_LOGGING_AND_CLOUD_STORAGE Store task logs in Cloud Logging and in the environment's Cloud Storage bucket.
CLOUD_LOGGING_ONLY Store task logs in Cloud Logging only.

UpdateEnvironmentRequest

Update an environment.

Fields
name

string

The relative resource name of the environment to update, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}"

Authorization requires the following IAM permission on the specified resource name:

  • composer.environments.update
environment

Environment

A patch environment. Fields specified by the updateMask will be copied from the patch environment into the environment under update.
update_mask

FieldMask

Required. A comma-separated list of paths, relative to Environment, of fields to update. For example, to set the version of scikit-learn to install in the environment to 0.19.0 and to remove an existing installation of argparse, the updateMask parameter would include the following two paths values: "config.softwareConfig.pypiPackages.scikit-learn" and "config.softwareConfig.pypiPackages.argparse". The included patch environment would specify the scikit-learn version as follows:

{
  "config":{
    "softwareConfig":{
      "pypiPackages":{
        "scikit-learn":"==0.19.0"
      }
    }
  }
}

Note that in the above example, any existing PyPI packages other than scikit-learn and argparse will be unaffected.

Only one update type may be included in a single request's updateMask. For example, one cannot update both the PyPI packages and labels in the same request. However, it is possible to update multiple members of a map field simultaneously in the same request. For example, to set the labels "label1" and "label2" while clearing "label3" (assuming it already exists), one can provide the paths "labels.label1", "labels.label2", and "labels.label3" and populate the patch environment as follows:

{
  "labels":{
    "label1":"new-label1-value"
    "label2":"new-label2-value"
  }
}

Note that in the above example, any existing labels that are not included in the updateMask will be unaffected.

It is also possible to replace an entire map field by providing the map field's path in the updateMask. The new value of the field will be that which is provided in the patch environment. For example, to delete all pre-existing user-specified PyPI packages and install botocore at version 1.7.14, the updateMask would contain the path "config.softwareConfig.pypiPackages", and the patch environment would be the following:

{
  "config":{
    "softwareConfig":{
      "pypiPackages":{
        "botocore":"==1.7.14"
      }
    }
  }
}

Note: Only the following fields can be updated:

  • config.softwareConfig.pypiPackages
    • Replace all custom custom PyPI packages. If a replacement package map is not included in environment, all custom PyPI packages are cleared. It is an error to provide both this mask and a mask specifying an individual package.
  • config.softwareConfig.pypiPackages.packagename
    • Update the custom PyPI package packagename, preserving other packages. To delete the package, include it in updateMask, and omit the mapping for it in environment.config.softwareConfig.pypiPackages. It is an error to provide both a mask of this form and the config.softwareConfig.pypiPackages mask.
  • labels
    • Replace all environment labels. If a replacement labels map is not included in environment, all labels are cleared. It is an error to provide both this mask and a mask specifying one or more individual labels.
  • labels.labelName
    • Set the label named labelName, while preserving other labels. To delete the label, include it in updateMask and omit its mapping in environment.labels. It is an error to provide both a mask of this form and the labels mask.
  • config.nodeCount
    • Horizontally scale the number of nodes in the environment. An integer greater than or equal to 3 must be provided in the config.nodeCount field. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
  • config.webServerNetworkAccessControl
    • Replace the environment's current WebServerNetworkAccessControl.
  • config.softwareConfig.airflowConfigOverrides
    • Replace all Apache Airflow config overrides. If a replacement config overrides map is not included in environment, all config overrides are cleared. It is an error to provide both this mask and a mask specifying one or more individual config overrides.
  • config.softwareConfig.airflowConfigOverrides.section-name
    • Override the Apache Airflow config property name in the section named section, preserving other properties. To delete the property override, include it in updateMask and omit its mapping in environment.config.softwareConfig.airflowConfigOverrides. It is an error to provide both a mask of this form and the config.softwareConfig.airflowConfigOverrides mask.
  • config.softwareConfig.envVariables
    • Replace all environment variables. If a replacement environment variable map is not included in environment, all custom environment variables are cleared.
  • config.softwareConfig.imageVersion
    • Upgrade the version of the environment in-place. Refer to SoftwareConfig.image_version for information on how to format the new image version. Additionally, the new image version cannot effect a version downgrade, and must match the current image version's Composer and Airflow major versions. Consult the Cloud Composer version list for valid values.
  • config.softwareConfig.schedulerCount
    • Horizontally scale the number of schedulers in Airflow. A positive integer not greater than the number of nodes must be provided in the config.softwareConfig.schedulerCount field. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-2.*.*.
  • config.softwareConfig.cloudDataLineageIntegration
    • Configuration for Cloud Data Lineage integration.
  • config.databaseConfig.machineType
    • Cloud SQL machine type used by Airflow database. It has to be one of: db-n1-standard-2, db-n1-standard-4, db-n1-standard-8 or db-n1-standard-16. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
  • config.webServerConfig.machineType
    • Machine type on which Airflow web server is running. It has to be one of: composer-n1-webserver-2, composer-n1-webserver-4 or composer-n1-webserver-8. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.
  • config.maintenanceWindow
    • Maintenance window during which Cloud Composer components may be under maintenance.
  • config.workloadsConfig
    • The workloads configuration settings for the GKE cluster associated with the Cloud Composer environment. Supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.
  • config.environmentSize
    • The size of the Cloud Composer environment. Supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.

UpdateUserWorkloadsConfigMapRequest

Update user workloads ConfigMap request.

Fields
user_workloads_config_map

UserWorkloadsConfigMap

Optional. User workloads ConfigMap to override.

UpdateUserWorkloadsSecretRequest

Update user workloads Secret request.

Fields
user_workloads_secret

UserWorkloadsSecret

Optional. User workloads Secret to override.

UserWorkloadsConfigMap

User workloads ConfigMap used by Airflow tasks that run with Kubernetes executor or KubernetesPodOperator.

Fields
name

string

Identifier. The resource name of the ConfigMap, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsConfigMaps/{userWorkloadsConfigMapId}"
data

map<string, string>

Optional. The "data" field of Kubernetes ConfigMap, organized in key-value pairs. For details see: https://kubernetes.io/docs/concepts/configuration/configmap/

Example:

{ "example_key": "example_value", "another_key": "another_value" }

UserWorkloadsSecret

User workloads Secret used by Airflow tasks that run with Kubernetes executor or KubernetesPodOperator.

Fields
name

string

Identifier. The resource name of the Secret, in the form: "projects/{projectId}/locations/{locationId}/environments/{environmentId}/userWorkloadsSecrets/{userWorkloadsSecretId}"
data

map<string, string>

Optional. The "data" field of Kubernetes Secret, organized in key-value pairs, which can contain sensitive values such as a password, a token, or a key. The values for all keys have to be base64-encoded strings. For details see: https://kubernetes.io/docs/concepts/configuration/secret/

Example:

{ "example": "ZXhhbXBsZV92YWx1ZQ==", "another-example": "YW5vdGhlcl9leGFtcGxlX3ZhbHVl" }

WebServerConfig

The configuration settings for the Airflow web server App Engine instance. Supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

Fields
machine_type

string

Optional. Machine type on which Airflow web server is running. It has to be one of: composer-n1-webserver-2, composer-n1-webserver-4 or composer-n1-webserver-8. If not specified, composer-n1-webserver-2 will be used. Value custom is returned only in response, if Airflow web server parameters were manually changed to a non-standard values.

WebServerNetworkAccessControl

Network-level access control policy for the Airflow web server.

Fields
allowed_ip_ranges[]

AllowedIpRange

A collection of allowed IP ranges with descriptions.

AllowedIpRange

Allowed IP range with user-provided description.

Fields
value

string

IP address or range, defined using CIDR notation, of requests that this rule applies to. Examples: 192.168.1.1 or 192.168.0.0/16 or 2001:db8::/32 or 2001:0db8:0000:0042:0000:8a2e:0370:7334.

IP range prefixes should be properly truncated. For example, 1.2.3.4/24 should be truncated to 1.2.3.0/24. Similarly, for IPv6, 2001:db8::1/32 should be truncated to 2001:db8::/32.
description

string

Optional. User-provided description. It must contain at most 300 characters.

WorkloadsConfig

The Kubernetes workloads configuration for GKE cluster associated with the Cloud Composer environment. Supported for Cloud Composer environments in versions composer-2.*.*-airflow-*.*.* and newer.

Fields
scheduler

SchedulerResource

Optional. Resources used by Airflow schedulers.
web_server

WebServerResource

Optional. Resources used by Airflow web server.
worker

WorkerResource

Optional. Resources used by Airflow workers.
triggerer

TriggererResource

Optional. Resources used by Airflow triggerers.
dag_processor

DagProcessorResource

Optional. Resources used by Airflow DAG processors.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

DagProcessorResource

Configuration for resources used by Airflow DAG processors.

This field is supported for Cloud Composer environments in versions composer-3.*.*-airflow-*.*.* and newer.

Fields
cpu

float

Optional. CPU request and limit for a single Airflow DAG processor replica.
memory_gb

float

Optional. Memory (GB) request and limit for a single Airflow DAG processor replica.
storage_gb

float

Optional. Storage (GB) request and limit for a single Airflow DAG processor replica.
count

int32

Optional. The number of DAG processors. If not provided or set to 0, a single DAG processor instance will be created.

SchedulerResource

Configuration for resources used by Airflow schedulers.

Fields
cpu

float

Optional. CPU request and limit for a single Airflow scheduler replica.
memory_gb

float

Optional. Memory (GB) request and limit for a single Airflow scheduler replica.
storage_gb

float

Optional. Storage (GB) request and limit for a single Airflow scheduler replica.
count

int32

Optional. The number of schedulers.

TriggererResource

Configuration for resources used by Airflow triggerers.

Fields
count

int32

Optional. The number of triggerers.
cpu

float

Optional. CPU request and limit for a single Airflow triggerer replica.
memory_gb

float

Optional. Memory (GB) request and limit for a single Airflow triggerer replica.

WebServerResource

Configuration for resources used by Airflow web server.

Fields
cpu

float

Optional. CPU request and limit for Airflow web server.
memory_gb

float

Optional. Memory (GB) request and limit for Airflow web server.
storage_gb

float

Optional. Storage (GB) request and limit for Airflow web server.

WorkerResource

Configuration for resources used by Airflow workers.

Fields
cpu

float

Optional. CPU request and limit for a single Airflow worker replica.
memory_gb

float

Optional. Memory (GB) request and limit for a single Airflow worker replica.
storage_gb

float

Optional. Storage (GB) request and limit for a single Airflow worker replica.
min_count

int32

Optional. Minimum number of workers for autoscaling.
max_count

int32

Optional. Maximum number of workers for autoscaling.