gcloud beta eventarc pipelines create

NAME

gcloud beta eventarc pipelines create - create an Eventarc pipeline

SYNOPSIS

gcloud beta eventarc pipelines create (PIPELINE : --location=LOCATION) --destinations=[http_endpoint_uri=URI],[http_endpoint_message_binding_template=HTTP_ENDPOINT_MESSAGE_BINDING_TEMPLATE],[workflow=WORKFLOW],[message_bus=MESSAGE_BUS],[pubsub_topic=PUBSUB_TOPIC],[project=PROJECT],[location=LOCATION],[network_attachment=NETWORK_ATTACHMENT],[google_oidc_authentication_service_account=GOOGLE_OIDC_AUTHENTICATION_SERVICE_ACCOUNT],[google_oidc_authentication_audience=GOOGLE_OIDC_AUTHENTICATION_AUDIENCE],[oauth_token_authentication_service_account=OAUTH_TOKEN_AUTHENTICATION_SERVICE_ACCOUNT],[oauth_token_authentication_scope=OAUTH_TOKEN_AUTHENTICATION_SCOPE],[output_payload_format_json=OUTPUT_PAYLOAD_FORMAT_JSON],[output_payload_format_avro_schema_definition=OUTPUT_PAYLOAD_FORMAT_AVRO_SCHEMA_DEFINITION],[output_payload_format_protobuf_schema_definition=OUTPUT_PAYLOAD_FORMAT_PROTOBUF_SCHEMA_DEFINITION]|[…] [--async] [--crypto-key=CRYPTO_KEY] [--logging-config=LOGGING_CONFIG] [--mediations=[transformation_template=TRANSFORMATION_TEMPLATE|…]] [--input-payload-format-avro-schema-definition=INPUT_PAYLOAD_FORMAT_AVRO_SCHEMA_DEFINITION     | --input-payload-format-json=INPUT_PAYLOAD_FORMAT_JSON     | --input-payload-format-protobuf-schema-definition=INPUT_PAYLOAD_FORMAT_PROTOBUF_SCHEMA_DEFINITION] [--max-retry-attempts=MAX_RETRY_ATTEMPTS --max-retry-delay=MAX_RETRY_DELAY --min-retry-delay=MIN_RETRY_DELAY] [GCLOUD_WIDE_FLAG …]

DESCRIPTION

(BETA) Create an Eventarc pipeline.

EXAMPLES

To create a new pipeline my-pipeline in location us-central1 with its destination targeting HTTP endpoint URI 'https://example-endpoint.com' and network attachment 'my-network-attachment', run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination and a message binding template, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',http_endpoint_message_binding_template='{"headers":
 {"new-header-key":
 "new-header-value"}}',network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with a Cloud Workflow destination my-workflow and network attachment my-network-attachment, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=workflow=my-workflow,network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with a message bus destination my-message-bus and network attachment my-network-attachment, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=message_bus=my-message-bus,network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with a Cloud Pub/Sub Topic destination my-topic and network attachment my-network-attachment, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=pubsub_topic=my-topic,network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with a Cloud Workflow in project example-project and location us-east1, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=workflow=my-workflow,project=example-project,location=us-east1,network_attachment=my-network-attachment

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a service account [e-mail] for OIDC authentication, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,google_oidc_authentication_service_account=[e-mail]

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a service account [e-mail] for OIDC authentication with audience https://example.com, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,google_oidc_authentication_service_account=[e-mail],google_oidc_authentication_audience='https://example.com'

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a service account [e-mail] for OAuth token authentication, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,oauth_token_authentication_service_account=[e-mail]

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a service account [e-mail] for OAuth token authentication with scope https://www.googleapis.com/auth/cloud-platform, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,oauth_token_authentication_service_account=[e-mail],oauth_token_authentication_scope='https://www.googleapis.com/auth/cl\
oud-platform'

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and the JSON input and output payload formats, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,output_payload_format_json= --input-payload-format-json=

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and the Avro input and output payload formats, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,output_payload_format_avro_schema_definition='{"type": "record",
 "name": "my_record", "fields": [{"name": "my_field", "type":
 "string"}]}' \
    --input-payload-format-avro-schema-definition='{"type":
 "record", "name": "my_record", "fields": [{"name": "my_field",
 "type": "string"}]}'

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and the Protobuf input and output payload formats, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,output_payload_format_protobuf_schema_definition='syntax =
 "proto3"; message Location { string home_address = 1; }' \
    --input-payload-format-protobuf-schema-definition='syntax =
 "proto3"; message Location { string home_address = 1; }'

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a transformation mediation, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment --mediations=transformation_template='message.removeFields(["data.credit_card_number","data.ssn"])'

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a INFO level logging configuration, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment --logging_config=INFO

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a custom retry policy, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment --max-retry-attempts=10 --min-retry-delay=2s --max-retry-delay=64s

To create a new pipeline my-pipeline in location us-central1 with an HTTP endpoint URI destination https://example-endpoint.com and a Cloud KMS CryptoKey, run:

gcloud beta eventarc pipelines create my-pipeline --location=us-central1 --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment --crypto-key=projects/PROJECT_ID/locations/KMS_LOCATION/keyRings/KEYRING/cryptoKeys/KEY

POSITIONAL ARGUMENTS

Pipeline resource - The pipeline to create. The arguments in this group can be used to specify the attributes of this resource. (NOTE) Some attributes are not given arguments in this group but can be set in other ways.

To set the project attribute:

• provide the argument pipeline on the command line with a fully specified name;
• provide the argument --project on the command line;
• set the property core/project.

This must be specified.

PIPELINE

ID of the pipeline or fully qualified identifier for the pipeline.

To set the pipeline attribute:

• provide the argument pipeline on the command line.

This positional argument must be specified if any of the other arguments in this group are specified.

--location=LOCATION

The location for the Eventarc pipeline, which should be one of the supported regions. Alternatively, set the [eventarc/location] property.

To set the location attribute:

• provide the argument pipeline on the command line with a fully specified name;
• provide the argument --location on the command line;
• set the property eventarc/location.

REQUIRED FLAGS

--destinations=[http_endpoint_uri=URI],[http_endpoint_message_binding_template=HTTP_ENDPOINT_MESSAGE_BINDING_TEMPLATE],[workflow=WORKFLOW],[message_bus=MESSAGE_BUS],[pubsub_topic=PUBSUB_TOPIC],[project=PROJECT],[location=LOCATION],[network_attachment=NETWORK_ATTACHMENT],[google_oidc_authentication_service_account=GOOGLE_OIDC_AUTHENTICATION_SERVICE_ACCOUNT],[google_oidc_authentication_audience=GOOGLE_OIDC_AUTHENTICATION_AUDIENCE],[oauth_token_authentication_service_account=OAUTH_TOKEN_AUTHENTICATION_SERVICE_ACCOUNT],[oauth_token_authentication_scope=OAUTH_TOKEN_AUTHENTICATION_SCOPE],[output_payload_format_json=OUTPUT_PAYLOAD_FORMAT_JSON],[output_payload_format_avro_schema_definition=OUTPUT_PAYLOAD_FORMAT_AVRO_SCHEMA_DEFINITION],[output_payload_format_protobuf_schema_definition=OUTPUT_PAYLOAD_FORMAT_PROTOBUF_SCHEMA_DEFINITION]|[…]

The pipeline's destinations. This flag can be repeated to add more destinations to the list. Currently, only one destination is supported per pipeline.

A destination is specified in a dict format. The valid choices of keys are

• http_endpoint_uri
• http_endpoint_message_binding_template
• workflow
• message_bus
• pubsub_topic
• project
• location
• network_attachment
• google_oidc_authentication_service_account
• google_oidc_authentication_audience
• oauth_token_authentication_service_account
• oauth_token_authentication_scope
• output_payload_format_json
• output_payload_format_avro_schema_definition
• output_payload_format_protobuf_schema_definition

The detailed explanations for each key are as follows:

1.http_endpoint_uri: the URI of the HTTP endpoint.

The value must be a RFC2396 URI string. Only HTTPS protocol is supported.
The host can be either a static IP addressable from the VPC specified by the network config, or an internal DNS hostname of the service resolvable via Cloud DNS.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment

2.http_endpoint_message_binding_template: the CEL expression used to construct a new HTTP request to be sent to the final destination.

It can be optionally specified alongside with `http_endpoint_uri`.
It represents a configuration used to bind a message to the final HTTP request that will be sent to the destination.
If a binding is not specified, by default the message is treated as a CloudEvent and is mapped to the HTTP request according to the CloudEvent HTTP Protocol Binding Binary Content Mode.
The Pipeline converts the data field of the message to the format provided in `output_payload_format` and maps it to the body field of the result.
It also sets the corresponding Content-Type header to the `output_payload_format` type.
If the `output_payload_format` is unspecified, then the Pipeline will treat the data field of the message as opaque binary data and attach it to the request body as bytes.
In this case, the Content-Type header will be set to the value of the datacontenttype attribute set on the incoming CloudEvent message if present, or the `application/octet-stream` MIME type otherwise.
The Pipeline expects that the content of the message will adhere to the standard CloudEvent format. If not then the outgoing message request may fail with a persistent error.

The result of the CEL expression must be a map of key-value pairs such that:
  1. If a map named `headers` exists on the result of the expression, then its key-value pairs are directly mapped to the HTTP request headers.
    The headers values are constructed from the corresponding value type's canonical representation.
    If the `headers` field does not exist, then the resulting HTTP request will not contain headers.
  2. If a field named `body` exists on the result of the expression, then its value is directly mapped to the body of the request.
    If the value of the `body` field is of type bytes or string, then it is used as the HTTP request body as-is withouth any conversion.
    If the `body` field is of any other type, then it is converted to a JSON string.
    If the `body` field does not exist, then the resulting HTTP request will not contain a body.
  3. Any other fields in the resulting expression will be ignored.

The CEL expression may access the incoming CloudEvent message in its definition, as follows:
  1. The `data` field of the incoming CloudEvent message can be accessed using the `message.data` value.
  2. Each attribute of the incoming CloudEvent message can be accessed using the `message.<key>` value, where <key> is the name of the attribute.

Headers added to the request by previous filters in the chain can be accessed in the CEL expression using the `headers` variable.
The `headers` variable defines a map of key-value pairs corresponding to the HTTP headers added by previous mediation steps and not the headers present on the original incoming request.
For example, the following CEL expression can be used to construct a headers-only HTTP request by adding an additional header to the headers added by previous mediations in the Pipeline:

{"headers": headers.merge({"new-header-key": "new-header-value"})}

Additionally, the following CEL extension functions can be used in this CEL expression:
  1. toBase64Url:
      map.toBase64Url() -> string
      Converts a CelValue to a base64url encoded string.
  2. toJsonString:
      map.toJsonString() -> string
      Converts a CelValue to a JSON string.
  3. merge:
      map1.merge(map2) -> map3
      Merges the passed CEL map with the existing CEL map the function is applied to.
      If the same key exists in both maps, or if the key's value is type map, then both maps are merged; Otherwise, the value from the passed map is used.
  4. toMap:
      list(map).toMap() -> map
      Converts a CEL list of CEL maps to a single CEL map.
  5. toDestinationPayloadFormat:
      message.data.toDestinationPayloadFormat() -> string or bytes
      Converts the message data to the destination payload format specified in `output_payload_format`.
      This function is meant to be applied to the message.data field.
      If the destination payload format is not set, the function will return the message data unchanged.
  6. toCloudEventJsonWithPayloadFormat:
      message.toCloudEventJsonWithPayloadFormat() -> map
      Converts a message to the corresponding structure of JSON format for CloudEvents.
      This function applies toDestinationPayloadFormat() to the message data.
      It also sets the corresponding datacontenttype of the CloudEvent, as indicated by the `output_payload_format` field.
      If `output_payload_format` is not set, it will use the existing datacontenttype on the CloudEvent if present; Otherwise, it leaves the datacontenttype unset.
      This function expects that the content of the message will adhere to the standard CloudEvent format. If it doesn't then this function will fail.
      The result is a CEL map that corresponds to the JSON representation of the CloudEvent. To convert that data to a JSON string it can be chained with the toJsonString() function.

Note that `http_endpoint_message_binding_template` cannot be set when `http_endpoint_uri` is not set.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',http_endpoint_message_binding_template='{"headers":
 {"new-header-key":
 "new-header-value"}}',network_attachment=example-network-attachment

3.workflow: the destination Workflow ID.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=workflow=my-workflow,network_attachment=example-network-attachment

4.message_bus: the destination Message Bus ID.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=message_bus=my-message-bus,network_attachment=example-network-attachment

5.pubsub_topic: the destination Pub/Sub topic ID.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=pubsub_topic=my-topic,network_attachment=example-network-attachment

NOTE: Exactly one of the http_endpoint_uri, workflow, message_bus, or pubsub_topic key must be set.

6.project: the project ID of the destination resource. If project is not set, then the project ID of the Pipeline will be used. NOTE: project cannot be set when http_endpoint_uri is set.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=project=example-project,workflow=my-workflow,network_attachment=example-network-attachment

7.location: the location of the destination resource. If location is not set, then the location of the Pipeline will be used. NOTE: location cannot be set when http_endpoint_uri is set.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=location=us-east1,workflow=my-workflow,network_attachment=example-network-attachment

8.network_attachment: the ID of the Network Attachment that allows access to the consumer VPC.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=network_attachment=my-network-attachment,http_endpoint_uri='https://example.com'

NOTE: network_attachment must be specified for a Pipeline.

9.google_oidc_authentication_service_account: the service account email used to generate the OIDC token. The token can be used to invoke Cloud Run and Cloud Function destinations or HTTP endpoints that support Google OIDC. Note that the principal who calls this API must have iam.serviceAccounts.actAs permission in the service account. See https://cloud.google.com/iam/docs/understanding-service-accounts?hl=en#sa_common for more information.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,google_oidc_authentication_service_account=[e-mail]

10.google_oidc_authentication_audience: the audience claim which identifies the recipient that the JWT is intended for. If unspecified, the destination URI will be used. NOTE: google_oidc_authentication_audience can only be set if google_oidc_authentication_service_account is set.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,google_oidc_authentication_service_account=[e-mail],google_oidc_authentication_audience='https://example.com'

11.oauth_token_authentication_service_account: the service account email used to generate the OAuth token. OAuth authorization should generally only be used when calling Google APIs hosted on *.googleapis.com. Note that the principal who calls this API must have iam.serviceAccounts.actAs permission in the service account. See https://cloud.google.com/iam/docs/understanding-service-accounts?hl=en#sa_common for more information.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,oauth_token_authentication_service_account=[e-mail]

12.oauth_token_authentication_scope: the scope used to generate the OAuth token. If unspecified, "https://www.googleapis.com/auth/cloud-platform" will be used. NOTE: oauth_token_authentication_scope can only be set if oauth_token_authentication_service_account is set.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,oauth_token_authentication_service_account=[e-mail],oauth_token_authentication_scope=https://www.googleapis.com/auth/\
cloud-platform

NOTE: At most one of google_oidc_authentication_service_account or oauth_token_authentication_service_account can be set.

13.output_payload_format_json: indicates that the output payload format is JSON. NOTE: JSON schema is not supported currently. Any value specified by this key will be ignored.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,output_payload_format_json= --input-payload-format-json=

14.output_payload_format_avro_schema_definition: the schema definition of the Avro output payload format.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,output_payload_format_avro_schema_definition='{"type": "record",
 "name": "my_record", "fields": [{"name": "field1", "type":
 "string"}]}' \
    --input-payload-format-avro-schema-definition='{"type":
 "record", "name": "my_record", "fields": [{"name": "field1",
 "type": "string"}]}'

15.output_payload_format_protobuf_schema_definition: the schema definition of the Protobuf output payload format.

Examples:

gcloud eventarc pipelines create example-pipeline --destinations=http_endpoint_uri='https://example.com',network_attachment=example-network-attachment,output_payload_format_protobuf_schema_definition='syntax =
 "proto3"; message Location { string home_address = 1; }' \
    --input-payload-format-protobuf-schema-definition='syntax =
 "proto3"; message Location { string home_address = 1; }'

NOTE: If none of the input_payload_format_json, input_payload_format_avro_schema_definition, or input_payload_format_protobuf_schema_definition is set, then the message data will be treated as an opaque binary and no output format can be set on the Pipeline through the output_payload_format_json, output_payload_format_avro_schema_definition, or output_payload_format_protobuf_schema_definition field. Any Mediations on the Pipeline that involve access to the data field will fail as persistent errors.

OPTIONAL FLAGS

--async

Return immediately, without waiting for the operation in progress to complete.

--crypto-key=CRYPTO_KEY

Fully qualified name of the crypto key to use for customer-managed encryption. If this is unspecified, Google-managed keys will be used for encryption.

--logging-config=LOGGING_CONFIG

The logging config of the pipeline. LOGGING_CONFIG must be one of: NONE, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY.

--mediations=[transformation_template=TRANSFORMATION_TEMPLATE|…]

The different ways to modify the Pipeline. Currently, only one mediation is supported per pipeline.

A mediation is specified in a dict format. Currently, the only valid choice is transformation_template.

This is the template to apply to transform messages.

For complex transformations, shell parameter processing may fail to parse the CEL expressions. Please see gcloud topic flags-file for how to use https://cloud.google.com/sdk/gcloud/reference/topic/flags-file feature of gcloud to pass in CEL expressions.

Examples:

gcloud eventarc pipelines create example-pipeline --mediations=transformation_template='message.removeFields(["data.credit_card_number","data.ssn"])'

At most one of these can be specified:

--input-payload-format-avro-schema-definition=INPUT_PAYLOAD_FORMAT_AVRO_SCHEMA_DEFINITION

The pipeline's input payload Avro schema definition. If this is set, then any messages not matching this format will be treated as persistent errors.

--input-payload-format-json=INPUT_PAYLOAD_FORMAT_JSON

The pipeline's input payload format is JSON. If this is set, then any messages not matching this format will be treated as persistent errors.

--input-payload-format-protobuf-schema-definition=INPUT_PAYLOAD_FORMAT_PROTOBUF_SCHEMA_DEFINITION

The pipeline's input payload Protobuf schema definition. If this is set, then any messages not matching this format will be treated as persistent errors.

The retry policy configuration for the Pipeline. The pipeline exponentially backs off if the destination is non-responsive or returns a retryable error code. The backoff starts with a 1 second delay and doubles the delay after each failed attempt. The delay is capped at 60 seconds. If the max-retry-delay and min-retry-delay are set to the same value, then the duration between retries is constant.

--max-retry-attempts=MAX_RETRY_ATTEMPTS

The maximum number of retry attempts. If not set, the default value is 5.

--max-retry-delay=MAX_RETRY_DELAY

The maximum retry delay in seconds. If not set, the default value is 60.

--min-retry-delay=MIN_RETRY_DELAY

The minimum retry delay in seconds. If not set, the default value is 1.

GCLOUD WIDE FLAGS

These flags are available to all commands: --access-token-file, --account, --billing-project, --configuration, --flags-file, --flatten, --format, --help, --impersonate-service-account, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity.

Run $ gcloud help for details.

NOTES

This command is currently in beta and might change without notice.