This document provides information about publishing messages.
A publisher application creates and sends messages to a topic. Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers.
The general flow for a publisher application is:
- Create a message containing your data.
- Send a request to the Pub/Sub server to publish the message to the specified topic.
Before you begin
Before configuring the publish workflow, ensure you have completed the following tasks:
- Create a topic.
- Optional: Create a subscription.
-
To get the permissions that you need to publish messages to a topic, ask your administrator to grant you the Pub/Sub Publisher (
roles/pubsub.publisher
) IAM role on topic. For more information about granting roles, see Manage access.You might also be able to get the required permissions through custom roles or other predefined roles.
You need additional permissions to create or update topics and subscriptions.
Message format
A message consists of fields with the message data and metadata. Specify at least one of the following in the message:
- The message data
- An ordering key
- Attributes with additional metadata
If you're using the REST API, the message data must be base64-encoded.
The Pub/Sub service adds the following fields to the message:
- A message ID unique to the topic
- A timestamp for when the Pub/Sub service receives the message
Publish messages
You can publish messages with the Google Cloud CLI or the Pub/Sub API. The client libraries can asynchronously publish messages.
Console
To publish a message, follow these steps:
In the Google Cloud console, go to the Pub/Sub topics page.
Click the topic ID.
In the Topic details page under Messages, click Publish message.
In the Message body field, enter the message data.
Optional: Add message attributes.
Click Add an attribute.
Enter a key and a value for the attribute.
Click Publish.
gcloud
To publish a message, use the gcloud pubsub topics publish command:
gcloud pubsub topics publish TOPIC_ID \ --message=MESSAGE_DATA \ [--attribute=KEY="VALUE",...]
Replace the following:
- TOPIC_ID: the ID of the topic
- MESSAGE_DATA: a string with the message data
- KEY: the key of a message attribute
- VALUE: the value for the key of the message attribute
REST
To publish a message, send a POST request like the following:
POST https://pubsub.googleapis.com/v1/projects/PROJECT_ID/topics/TOPIC_ID:publish Content-Type: application/json Authorization: Bearer $(gcloud auth application-default print-access-token)
Replace the following:
- PROJECT_ID: the project ID of the project with the topic
- TOPIC_ID: the ID of the topic
Specify the following fields in the request body:
{ "messages": [ { "attributes": { "KEY": "VALUE", ... }, "data": "MESSAGE_DATA", } ] }
Replace the following:
- KEY: the key of a message attribute
- VALUE: the value for the key of the message attribute
- MESSAGE_DATA: a base64-encoded string with the message data
The message must contain either a non-empty data field or at least one attribute.
If the request is successful, the response is a JSON object with the message ID. The following example is a response with a message ID:
{ "messageIds": [ "19916711285", ] }
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
PHP
Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
After you publish a message, the Pub/Sub service returns the message ID to the publisher.
Use attributes
You can embed custom attributes as metadata in Pub/Sub messages. Attributes can
be text strings or byte strings. You can have at most 100 attributes per message.
Attribute keys should not start with goog
and should not exceed 256 bytes.
Attribute values should not exceed 1024 bytes. The message schema can be
represented as follows:
{ "data": string, "attributes": { string: string, ... }, "messageId": string, "publishTime": string, "orderingKey": string }
The PubsubMessage
JSON schema is published as part of the
REST and
RPC
documentation. You can use custom attriubtes for event timestamps.
gcloud
gcloud pubsub topics publish my-topic --message="hello" \ --attribute="origin=gcloud-sample,username=gcp,eventTime='2021-01-01T12:00:00Z'"
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Use ordering keys
If messages have the same ordering key and you publish the messages to the same region, subscribers can receive the messages in order. Publishing messages with ordering keys might increase latency. To publish messages to the same region, use a regional endpoint.
You can publish messages with ordering keys using the Google Cloud console, the Google Cloud CLI, or the Pub/Sub API.
Console
In the Google Cloud console, go to the Pub/Sub topics page.
Click the topic ID.
In the Topic details page under Messages, click Publish message.
In the Message body field, enter the message data.
In the Message ordering field, enter an ordering key.
Click Publish.
gcloud
To publish a message with an ordering key, use the
gcloud pubsub topics publish
command and the --ordering-key
flag:
gcloud pubsub topics publish TOPIC_ID \ --message=MESSAGE_DATA \ --ordering-key=ORDERING_KEY
Replace the following:
- TOPIC_ID: the ID of the topic
- MESSAGE_DATA: a string with the message data
- ORDERING_KEY: a string with an ordering key
REST
To publish a message with an ordering key, send a POST request like the following:
POST https://pubsub.googleapis.com/v1/projects/PROJECT_ID/topics/TOPIC_ID:publish Content-Type: application/json Authorization: Bearer $(gcloud auth application-default print-access-token)
Replace the following:
- PROJECT_ID: the project ID of the project with the topic
- TOPIC_ID: the ID of the topic
Specify the following fields in the request body:
{ "messages": [ { "attributes": { "KEY": "VALUE", ... }, "data": "MESSAGE_DATA", "ordering_key": "ORDERING_KEY", } ] }
Replace the following:
- KEY: the key of a message attribute
- VALUE: the value for the key of the message attribute
- MESSAGE_DATA: a base64-encoded string with the message data
- ORDERING_KEY: a string with an ordering key
The message must contain either a non-empty data field or at least one attribute.
If the request is successful, the response is a JSON object with the message ID. The following example is a response with a message ID:
{ "messageIds": [ "19916711285", ] }
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
When publishing with an ordering key fails, queued-up messages of the same ordering key in the publisher fail, including future publish requests of this ordering key. You must resume publishing with ordering keys when such failures occur. For an example of resuming the publish operation, see Retry requests with ordering keys.
Use schema
You can publish messages to a topic which is associated with a schema. To learn more, see Create and manage schemas. You must encode the messages in the schema and format that you specified when you created the topic.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
Avro ProtoC#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Avro ProtoGo
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Avro ProtoJava
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Avro ProtoNode.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Avro Protocol BufferPHP
Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.
Avro Protocol BufferPython
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Avro Protocol BufferRuby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Avro Protocol BufferBatch messages in a publish request
You can use the Pub/Sub client library for your publisher to publish messages to a topic. The client library uses the batch feature to publish several messages together in one service call. The batching or grouping of messages helps Pub/Sub to achieve a higher throughput of messages. You can adjust the size of the batch to suit your business needs.
Batch messaging is enabled by default in a client library. Batch messaging creates latency for individual messages. Individual messages must queue up in memory until their corresponding batch is filled. Only then, the messages are published to the topic.
If cost is not a consideration, you can create multiple publisher clients and disable batch messaging. This process minimizes latency and maximizes throughput by scaling horizontally on the number of publishers. However, cost is often a consideration. Sending multiple messages in a single publish request is one way to reach equivalent throughput with fewer publishers. If you are willing to trade off some latency for cost savings, especially if your application deals with a fairly large number of messages within a short time period, you can use batch messaging.
Batch messaging lets you configure the batch size in bytes or number of messages and also the time after which the batch is published. Small batches of messages during peak publishing time can help you control latency.
Configure batch messaging in a client library
You can batch messages based on message request size, number of messages, and time. The default values for the batch messaging variables and the names of the variables might differ across client libraries.
For example, in the Python client library, the following variables control batch messaging:
Python variable | Description | Value |
---|---|---|
max_messages | The number of messages in a batch. | Default=100 |
max_bytes | The maximum size of a batch in MB. | Default=1 MB |
max_latency | The time after which a batch is published, even if the batch is not filled. | Default=10 ms |
You can specify one or all three values in the client library. If any one of the values for the batch messaging variables are met, the client library publishes the next batch of messages.
See the following code samples to learn how to configure batch messaging settings for your publisher.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
PHP
Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Disable batch messaging
To turn off batching in your client library, set the value of
max_messages
to 1.
Batch messaging and ordered delivery
With ordered delivery, failing to acknowledge any message in the batch means that all the messages in the batch, including the ones sent before the message that was not acknowledged, are all redelivered.
Quotas and limits on batch messaging
Before configuring batch messaging, consider the effect of factors such as publish throughput quota and the maximum size of a batch. The high-level client libraries ensure that batch requests are kept within the specified limits.
- 1000 bytes is the minimum request size considered for cost purposes, even if the actual message size might be smaller than 1000 bytes.
- Pub/Sub has a limit of 10-MB size or 1000 number of messages for a single batch publish request.
For more information, see Pub/Sub quotas and limits.
Compress messages
If you are using Pub/Sub to publish messages that amount to a lot of data, you can use gRPC to compress your data to save networking costs before your publisher client sends out the publish request. The Pub/Sub compression for gRPC uses the Gzip algorithm.
The compression ratio for using the gRPC client-side compression feature is different for different publisher clients and dependent on the following factors:
Amount of data. The compression ratio improves when the size of the payload increases from a few hundred bytes to many kilobytes of data. The batch settings of a publish request decides the amount of data that is included in each publish request. We recommend that you turn on batch settings in conjunction with gRPC compression to get the best results.
Type of data. Text-based data such as JSON or XML are more compressible compared to binary data such as images.
If your publisher client is on Google Cloud, you can use
the Sent bytes (instance/network/sent_bytes_count
) metric
to measure the publishing throughput in bytes. If your publisher
client is on a different application, you must use the
client-specific tools for making the measurement.
The code sample in this section shows a sample Java client library code snippet that also includes gRPC compression.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Retry requests
Publishing failures are automatically retried, except for errors that do not warrant retries. This sample code demonstrates creating a publisher with custom retry settings (note that not all client libraries support custom retry settings; see the API Reference documentation for your chosen language):
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Retry settings control how the Pub/Sub client libraries retry publish requests. The client libraries have any of the following retry settings:
- Initial request timeout: the amount of time before a client library stops waiting for the initial publish request to complete.
- Retry delay: the amount of time after a request times out that a client library waits to retry the request.
- Total timeout: the amount of time after a client library stops retrying publish requests.
To retry publish requests, the initial request timeout must be shorter than the total timeout. For example, if you're using exponential backoff, the client libraries compute the request timeout and retry delay as follows:
- After each publish request, the request timeout increases by the request timeout multiplier, up to the maximum request timeout.
- After each retry, the retry delay increases by the retry delay multiplier, up to the maximum retry delay.
Retry requests with ordering keys
When a client library retries a request and the message has an ordering key, the client library repeatedly retries the request, regardless of the retry settings.
If a non-retryable error occurs, the client library doesn't publish the message and stops publishing other messages with the same ordering key. For example, when a publisher sends a message to a topic that doesn't exist, a non-retryable error occurs. To continue publishing messages with the same ordering key, call a method to resume publishing and then start publishing again.
The following sample shows you how to resume publishing messages with the same ordering key.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Flow control
A publisher client may attempt to publish messages faster than that client is capable of sending data to the Pub/Sub service. Clients are limited by many factors, including:
- Machine CPU, RAM, and network capacity
- Network settings, such as the number of outstanding requests and available bandwidth
- The latency of each publish request, largely determined by the network connections between the Pub/Sub service, the client, and Google Cloud
If the publish request rate exceeds these limits, requests accumulate in memory
until they fail with a DEADLINE_EXCEEDED
error. This is especially likely
when tens of thousands of messages are published in a loop, generating thousands
of requests in milliseconds.
You can diagnose this issue by checking the server side metrics in Monitoring. You
won't be able to see the requests that have failed with DEADLINE_EXCEEDED
, only the
successful requests. The rate of successful requests tells you the
throughput capacity of your client machines, providing a baseline for configuring flow
control.
To mitigate flow rate issues, configure your publisher client with flow control to limit the rate of publish requests. You can configure the maximum number of bytes allocated for outstanding requests, and the maximum number of outstanding messages permitted. Set these limits according to the throughput capacity of your client machines.
Publisher flow control is available via the Pub/Sub client libraries in the following languages:
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Concurrency control
Support for concurrency depends on your programming language. Refer to the API Reference documentation for more information.
The following sample illustrates how to control concurrency in a publisher:
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Monitor a publisher
Cloud Monitoring provides a number of metrics to monitor topics.
To monitor a topic and maintain a healthy publisher, see Maintain a healthy publisher.
What's next
To restrict the locations in which Pub/Sub stores message data, see Restricting Pub/Sub resource locations.
To learn more about receiving messages, see Choose a subscription type.