Publish message overview

To publish a message with Pub/Sub, a publisher application creates and sends messages to a topic.

This document provides an overview about the publish workflow, including the concept of topics and messages.

About topics

A Pub/Sub topic is a named resource that represents a feed of messages. When a publisher sends a message, it targets a specific topic. The Pub/Sub service uses this topic name to route the message to all subscriptions attached to the topic. If there are multiple subscribers for a subscription, only one subscriber in the subscription receives the message.

Publishers don't have to know how many subscribers exist. They focus on the topic, ensuring the separation of concerns between message sending and message receiving.

Pub/Sub supports two kinds of topics: a standard topic and an import topic.

Properties of a topic

When you create or update a topic, you can specify the topic properties.

For more information about the topic properties, see Properties of a topic.

About import topics

An import topic lets Pub/Sub ingest streaming data from another source and act as the publisher application that sends the data to the topic. You can enable ingestion on a topic by using the console, Google Cloud CLI, REST calls, or the client libraries. As part of managing the import topic, Google Cloud provides monitoring and scaling of the ingestion pipeline.

Without an import topic, streaming data into Pub/Sub from a data source requires an additional service. This additional service pulls data from the original source and publishes it to Pub/Sub. The additional service can be a streaming engine such as Apache Spark or a custom-written service. You must also configure, deploy, run, scale, and monitor this service.

The following is a list of important information regarding import topics:

  • Similar to a standard topic, you can still manually publish to an import topic.

  • You can only attach a single ingestion source to an import topic.

We recommend import topics for streaming data. If you are considering batch data ingestion into BigQuery instead of streaming data ingestion, you can try BigQuery Data Transfer Service (BQ DTS). If you want to ingest data into Cloud Storage, Storage Transfer Service (STS) is a good option.

Pub/Sub supports the following sources for import topics: Amazon Kinesis Data Streams

Data replication in a topic

A Pub/Sub topic uses three zones to store data. The service supports synchronous replication to at least two zones, and best-effort replication to an additional third zone. Pub/Sub replication is within just one region.

About messages

A Pub/Sub message is the data that moves through the service.

A message consists of fields with the message data and metadata. One of the following must be specified in a message.

  • The message data: This is the core content of the message and can be any text or binary data. It represents the actual information you want to communicate between publishers and subscribers. If you're using the REST API directly, the message data must be base64-encoded. See the example in the REST tab in the Publish messages section.

  • An ordering key: This is an identifier that represents the entity for which messages must be ordered. Messages with the same ordering key are expected to be delivered to a subscriber in the order they were published. An ordering key is only required if you want ordered delivery of your messages. To learn more about ordering keys, see Order message.

  • Attributes: These are optional key-value pairs that provide additional context and information about the message. They can be used for routing, filtering, or enriching the message content. For example, you could add attributes such as timestamps or transaction IDs. To know more about attributes used in publishing messages, see Use attributes to publish a message.

The Pub/Sub service adds the following fields to the message:

  • A message ID unique to the topic
  • A timestamp for when the Pub/Sub service receives the message

For example, here is a message format in JSON:

{
  "data": "This is the core message content.",
  "attributes": {
    "category": "notification",
    "user_id": "12345",
    "priority": "medium"
  },
    "orderingKey": "12345"
}

Publish message workflow

To publish a message with Pub/Sub, a publisher application creates and sends messages to a topic.

  1. Create a message containing your data.
  2. Select any optional publishing attributes.
  3. Send a request to the Pub/Sub server to publish the message to a specified topic.

  4. The Pub/Sub service receives the message and processes it as follows:

    • The message is stored for distribution.

    • The message is replicated across multiple zones for durability and high availability.

    • Pub/Sub identifies subscribers with subscriptions matching the message's topic, and delivers a copy of the message to each.

Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers.

For an overview about the Pub/Sub system, see Overview of the Pub/Sub service.

For an explanation of how Pub/Sub works, see Architectural overview of Pub/Sub.

What's next