Creating and managing Lite topics

This page explains how to create, view, and delete Lite topics.

Lite topics are zonal resources that you can publish messages to. When you create a Lite topic, you must specify a zone to store messages in.

After creating a Lite topic, you can publish messages to the Lite topic, create a Lite subscription to the Lite topic, and receive messages from the Lite subscription.

Creating Lite topics

When you create a Lite topic, you must set its throughput and storage capacity. The number and capacity of partitions determine the capacity of a Lite topic.

Lite topics have the following properties:

  • Number of partitions: The number of partitions in the Lite topic.
  • Storage per partition: The amount of storage, in bytes, for each partition.
  • Scale factor: An integer multiple to increase the default throughput of each partition. The default scale factor is 1. For best results, create a Lite topic with the default scale factor. If traffic increases, update the Lite topic and increase the scale factor.
  • Message retention period: The maximum amount of time a Lite topic stores messages. If you don't specify a message retention period, the Lite topic stores messages until you exceed the storage capacity.

You can create Lite topics with the Cloud Console, the gcloud command-line tool, or the Pub/Sub Lite API.

Console

  1. In the Cloud Console, go to the Lite Topics page.

    Go to the Lite Topics page

  2. Click Create Lite topic.

  3. Select a region and a zone in the region.

  4. In the Name section, enter a Lite topic ID. The Lite topic name includes the Lite topic ID, the zone, and the project number.

  5. In the Throughput section, enter the Number of partitions to provision for the partition.

  6. In the Message storage section, enter the Storage per partition.

  7. Optional: In the Message retention section, select Drop messages after the specified duration even if storage is available and enter a retention period.

  8. Click Create.

gcloud

To create a Lite topic, use the gcloud beta pubsub lite-topics create command:

gcloud beta lite-topics create TOPIC_ID \
  --zone=ZONE \
  --partitions=NUMBER_OF_PARTITIONS \
  --per-partition-bytes=STORAGE_PER_PARTITION \
  [--message-retention-period=MESSAGE_RETENTION_PERIOD]

Replace the following:

  • TOPIC_ID: the ID of the Lite topic

  • ZONE: the name of a zone that Pub/Sub Lite supports

  • NUMBER_OF_PARTITIONS: an integer for the number of partitions in the Lite topic

  • STORAGE_PER_PARTITION: the amount of storage for each partition, such as 30GiB

  • MESSAGE_RETENTION_PERIOD: the amount of time the Lite topic stores messages, such as 1d or 2w

If the request is successful, the command line displays a confirmation:

Created [TOPIC_ID].

Protocol

To create a Lite topic, send a POST request like the following:

POST https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

Specify the following fields in the request body:

{
  "count": NUMBER_OF_PARTITIONS,
  "perPartitionBytes": STORAGE_PER_PARTITION,
}

Replace the following:

  • NUMBER_OF_PARTITIONS: an integer for the number of partitions in the Lite topic

  • STORAGE_PER_PARTITION: the amount of storage for each partition, such as 30GiB

If the request is successful, the response is the Lite topic in JSON format:

{
  "name": projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID,
  "partitionConfig": {
      "count": NUMBER_OF_PARTITIONSs,
      "scale": SCALE_FACTOR,
   },
   "retentionConfig": {
       "perPartitionBytes": STORAGE_PER_PARTITION,
       "period": MESSAGE_RETENTION_PERIOD,
   },
}

Java

import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
import com.google.cloud.pubsublite.CloudRegion;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectNumber;
import com.google.cloud.pubsublite.TopicName;
import com.google.cloud.pubsublite.TopicPath;
import com.google.cloud.pubsublite.TopicPaths;
import com.google.cloud.pubsublite.proto.Topic;
import com.google.cloud.pubsublite.proto.Topic.PartitionConfig;
import com.google.cloud.pubsublite.proto.Topic.RetentionConfig;
import com.google.protobuf.util.Durations;

public class CreateTopicExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String cloudRegion = "your-cloud-region";
    char zoneId = 'b';
    String topicId = "your-topic-id";
    long projectNumber = Long.parseLong("123456789");
    Integer partitions = 1;

    createTopicExample(cloudRegion, zoneId, projectNumber, topicId, partitions);
  }

  public static void createTopicExample(
      String cloudRegion, char zoneId, long projectNumber, String topicId, int partitions)
      throws Exception {

    TopicPath topicPath =
        TopicPaths.newBuilder()
            .setProjectNumber(ProjectNumber.of(projectNumber))
            .setZone(CloudZone.of(CloudRegion.of(cloudRegion), zoneId))
            .setTopicName(TopicName.of(topicId))
            .build();

    Topic topic =
        Topic.newBuilder()
            .setPartitionConfig(
                PartitionConfig.newBuilder()
                    // Set publishing throughput to 1 times the standard partition
                    // throughput of 4 MiB per sec. This must be in the range [1,4]. A
                    // topic with `scale` of 2 and count of 10 is charged for 20 partitions.
                    .setScale(1)
                    .setCount(partitions))
            .setRetentionConfig(
                RetentionConfig.newBuilder()
                    // How long messages are retained.
                    .setPeriod(Durations.fromDays(1))
                    // Set storage per partition to 100 GiB. This must be 30 GiB-10 TiB.
                    // If the number of bytes stored in any of the topic's partitions grows
                    // beyond this value, older messages will be dropped to make room for
                    // newer ones, regardless of the value of `period`.
                    .setPerPartitionBytes(100 * 1024 * 1024 * 1024L))
            .setName(topicPath.value())
            .build();

    AdminClientSettings adminClientSettings =
        AdminClientSettings.newBuilder().setRegion(CloudRegion.of(cloudRegion)).build();

    try (AdminClient adminClient = AdminClient.create(adminClientSettings)) {
      Topic response = adminClient.createTopic(topic).get();
      System.out.println(response.getAllFields() + "created successfully.");
    }
  }
}

For a list of the available zones, see Pub/Sub Lite locations.

After creating the Lite topic, you can't update the number of partitions, but you can change scale the throughput and storage capacity. For details, see Scaling capacity.

Provisioning capacity

Lite topics consist of partitions. A partition supports the following capacity:

  • Publishing throughput: The maximum total rate at which you can publish messages. The default is 4 MiB per sec.
  • Subscribing throughput: The maximum total rate at which messages are forwarded to Lite subscriptions. The default is 8 MiB per sec.
  • Storage: The maximum total size of the messages in the partition. You can specify between 30 GiB-10 TiB of storage.

If you exceed the storage capacity, the Pub/Sub Lite service removes the oldest message from the partition, regardless of the message retention period for the oldest message. The Pub/Sub Lite service removes messages on a best-effort basis.

Updating Lite topics

You can update the message retention period and scale the capacity of a Lite topic. You cannot update the number of partitions or the zone. To scale the capacity of a Lite topic, see Scaling capacity.

You can update a Lite topic with the Cloud Console, the gcloud command-line tool, or the Pub/Sub Lite API.

Console

  1. In the Cloud Console, go to the Lite Topics page.

    Go to the Lite Topics page

  2. Click the Lite topic ID.

  3. Click Edit.

  4. Enter the Storage per partition and the Retention period.

  5. Select a Scale factor.

  6. Enter the Storage per partition.

gcloud

To update a Lite topic, use the gcloud beta pubsub lite-topics update command:

gcloud beta lite-topics update TOPIC_ID \
  --zone=ZONE \
  --scale=SCALE_FACTOR \
  --per-partition-bytes=STORAGE_PER_PARTITION \
  --message-retention-period=MESSAGE_RETENTION_PERIOD

Replace the following:

  • TOPIC_ID: the ID of the Lite topic

  • ZONE: the name of the zone that the Lite topic is in

  • SCALE_FACTOR: an integer multiple to increase the default throughput of each partition

  • STORAGE_PER_PARTITION: the amount of storage for each partition, such as 30GiB

  • MESSAGE_RETENTION_PERIOD: the amount of time the Lite topic stores messages, such as 1d or 2w

If the request is successful, the command line displays the Lite topic:

name: projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
partitionConfig:
  count: NUMBER_OF_PARTITIONS
  scale: SCALE_FACTOR
retentionConfig:
  perPartitionBytes: STORAGE_PER_PARTITION
  period: MESSAGE_RETENTION_PERIOD

Protocol

To update a Lite topic, send a PATCH request like the following:

PATCH https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID?updateMask=partitionConfig.scale,retentionConfig.perPartitionBytes,retentionConfig.period
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

  • REGION: the region of the zone that the Lite topic is in

  • PROJECT_NUMBER: the project number of the project with the Lite topic

  • ZONE: the name of the zone that the Lite topic is in

  • TOPIC_ID: the ID of the Lite topic

Specify the following fields in the request body:

{
  "partitionConfig": {
      "scale": SCALE_FACTOR,
   },
   "retentionConfig": {
       "perPartitionBytes": STORAGE_PER_PARTITION,
       "period": MESSAGE_RETENTION_PERIOD,
   },
}

Replace the following:

  • SCALE_FACTOR: an integer multiple to increase the default throughput of each partition

  • STORAGE_PER_PARTITION: the amount of storage for each partition, such as 30GiB

  • MESSAGE_RETENTION_PERIOD: the amount of time the Lite topic stores messages, such as 1d or 2w

If the request is successful, the response is the Lite topic in JSON format:

{
  "name": projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID,
  "partitionConfig": {
      "count": NUMBER_OF_PARTITIONS,
      "scale": SCALE_FACTOR,
   },
   "retentionConfig": {
       "perPartitionBytes": STORAGE_PER_PARTITION,
       "period": MESSAGE_RETENTION_PERIOD,
   },
}

Java

import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
import com.google.cloud.pubsublite.CloudRegion;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectNumber;
import com.google.cloud.pubsublite.TopicName;
import com.google.cloud.pubsublite.TopicPath;
import com.google.cloud.pubsublite.TopicPaths;
import com.google.cloud.pubsublite.proto.Topic;
import com.google.cloud.pubsublite.proto.Topic.PartitionConfig;
import com.google.cloud.pubsublite.proto.Topic.RetentionConfig;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.Durations;
import java.util.Arrays;

public class UpdateTopicExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String cloudRegion = "your-cloud-region";
    char zoneId = 'b';
    String topicId = "your-topic-id";
    long projectNumber = Long.parseLong("123456789");

    updateTopicExample(cloudRegion, zoneId, projectNumber, topicId);
  }

  public static void updateTopicExample(
      String cloudRegion, char zoneId, long projectNumber, String topicId) throws Exception {

    TopicPath topicPath =
        TopicPaths.newBuilder()
            .setProjectNumber(ProjectNumber.of(projectNumber))
            .setZone(CloudZone.of(CloudRegion.of(cloudRegion), zoneId))
            .setTopicName(TopicName.of(topicId))
            .build();

    Iterable<String> iterablePaths =
        Arrays.asList(
            "partition_config.scale",
            "retention_config.per_partition_bytes",
            "retention_config.period");

    FieldMask MASK = FieldMask.newBuilder().addAllPaths(iterablePaths).build();

    Topic topic =
        Topic.newBuilder()
            .setPartitionConfig(
                PartitionConfig.newBuilder()
                    // Set publishing throughput to 4 times the standard partition
                    // throughput of 4 MiB per sec. This must be in the range [1,4]. A
                    // topic with `scale` of 2 and count of 10 is charged for 20 partitions.
                    .setScale(4)
                    .build())
            .setRetentionConfig(
                RetentionConfig.newBuilder()
                    // Set storage per partition to 200 GiB. This must be 30 GiB-10 TiB.
                    // If the number of bytes stored in any of the topic's partitions grows
                    // beyond this value, older messages will be dropped to make room for
                    // newer ones, regardless of the value of `period`.
                    // Be careful when decreasing storage per partition as it may cause
                    // lost messages.
                    .setPerPartitionBytes(200 * 1024 * 1024 * 1024L)
                    .setPeriod(Durations.fromDays(7)))
            .setName(topicPath.value())
            .build();

    AdminClientSettings adminClientSettings =
        AdminClientSettings.newBuilder().setRegion(CloudRegion.of(cloudRegion)).build();

    try (AdminClient adminClient = AdminClient.create(adminClientSettings)) {
      Topic topicBeforeUpdate = adminClient.getTopic(topicPath).get();
      System.out.println("Before update: " + topicBeforeUpdate.getAllFields());

      Topic topicAfterUpdate = adminClient.updateTopic(topic, MASK).get();
      System.out.println("After update: " + topicAfterUpdate.getAllFields());
    }
  }
}

Scaling capacity

You can increase the throughput capacity of a Lite topic with vertical scaling. To increase the throughput capacity of a Lite topic, scale the capacity of the partitions by an integer multiple.

Scale factor Publishing throughput Subscribing throughput
1 4 MiB per sec 8 MiB per sec
2 8 MiB per sec 16 MiB per sec
3 12 MiB per sec 24 MiB per sec
4 16 MiB per sec 32 MiB per sec

For example, if you update the scale factor to 2, the publishing throughput of each partition is 8 MiB per sec and the subscribing throughput of each partition is 16 MiB per sec.

You can also increase or decrease the amount of storage in a Lite topic. The Lite topic provisions the same amount of storage to each partition. If you increase the storage to 60 GiB, each of the partitions gets 60 GiB of storage.

If you decrease the amount of storage in a Lite topic, the Pub/Sub Lite service removes the oldest messages first.

Getting Lite topic details

You can get details about a Lite topic using the Cloud Console, the gcloud command-line tool, or the Pub/Sub Lite API.

Console

  1. In the Cloud Console, go to the Lite Topics page.

    Go to the Lite Topics page

  2. Click the Lite topic ID.

gcloud

To get details about a Lite topic, use the gcloud beta pubsub lite-topics describe command:

gcloud beta lite-topics describe TOPIC_ID \
--zone=ZONE

Replace the following:

  • TOPIC_ID: the ID of the Lite topic

  • ZONE: the name of the zone that the Lite topic is in

If the request is successful, the command line displays the Lite topic:

name: projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
partitionConfig:
  count: NUMBER_OF_PARTITIONS
  scale: SCALE_FACTOR
retentionConfig:
  perPartitionBytes: STORAGE_PER_PARTITION
  period: MESSAGE_RETENTION_PERIOD

Protocol

To get details about a Lite topic, send a GET request like the following:

GET https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

  • REGION: the region of the zone that the Lite topic is in

  • PROJECT_NUMBER: the project number of the project with the Lite topic

  • ZONE: the name of the zone that the Lite topic is in

  • TOPIC_ID: the ID of the Lite topic

If the request is successful, the response is the Lite topic in JSON format:

{
  "name": projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID,
  "partitionConfig": {
      "count": NUMBER_OF_PARTITIONS,
      "scale": SCALE_FACTOR,
   },
   "retentionConfig": {
       "perPartitionBytes": STORAGE_PER_PARTITION,
       "period": MESSAGE_RETENTION_PERIOD,
   },
}

Java

import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
import com.google.cloud.pubsublite.CloudRegion;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectNumber;
import com.google.cloud.pubsublite.TopicName;
import com.google.cloud.pubsublite.TopicPath;
import com.google.cloud.pubsublite.TopicPaths;
import com.google.cloud.pubsublite.proto.Topic;

public class GetTopicExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String cloudRegion = "your-cloud-region";
    char zoneId = 'b';
    // Choose an existing topic.
    String topicId = "your-topic-id";
    long projectNumber = Long.parseLong("123456789");

    getTopicExample(cloudRegion, zoneId, projectNumber, topicId);
  }

  public static void getTopicExample(
      String cloudRegion, char zoneId, long projectNumber, String topicId) throws Exception {
    TopicPath topicPath =
        TopicPaths.newBuilder()
            .setProjectNumber(ProjectNumber.of(projectNumber))
            .setZone(CloudZone.of(CloudRegion.of(cloudRegion), zoneId))
            .setTopicName(TopicName.of(topicId))
            .build();

    AdminClientSettings adminClientSettings =
        AdminClientSettings.newBuilder().setRegion(CloudRegion.of(cloudRegion)).build();

    try (AdminClient adminClient = AdminClient.create(adminClientSettings)) {
      Topic topic = adminClient.getTopic(topicPath).get();
      long numPartitions = adminClient.getTopicPartitionCount(topicPath).get();
      System.out.println(topic.getAllFields() + "\nhas " + numPartitions + " partition(s).");
    }
  }
}

Listing Lite topics

You can list Lite topics in a project using the Cloud Console, the gcloud command-line tool, or the Pub/Sub Lite API.

Console

To view a list of the Lite topics in a project, go to the Lite Topics page.

gcloud

To list the Lite topics in a project, use the gcloud beta pubsub lite-topics list command:

gcloud beta lite-topics list \
  --zone=ZONE

Replace ZONE with the name of the zone that the Lite topic is in.

If the request is successful, the command line displays the Lite topics:

---
name: projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
partitionConfig:
  count: NUMBER_OF_PARTITIONS
  scale: SCALE_FACTOR
retentionConfig:
  perPartitionBytes: STORAGE_PER_PARTITION
  period: MESSAGE_RETENTION_PERIOND
---
name: projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
partitionConfig:
  count: NUMBER_OF_PARTITIONS
  scale: SCALE_FACTOR
retentionConfig:
  perPartitionBytes: STORAGE_PER_PARTITION
  period: MESSAGE_RETENTION_PERIOND

Protocol

To list the Lite topics in a project, send a GET request like the following:

GET https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/topics
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

  • REGION: the region of the zone that the Lite topic is in

  • PROJECT_NUMBER: the project number of the project with the Lite topic

If the request is successful, the response is a list of Lite topics in JSON format:

{
  "topics": [
      {
          "name": "projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID",
      },
      {
          "name": "projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID",
      }
  ]
}

Java

import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
import com.google.cloud.pubsublite.CloudRegion;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.LocationPath;
import com.google.cloud.pubsublite.LocationPaths;
import com.google.cloud.pubsublite.ProjectNumber;
import com.google.cloud.pubsublite.proto.Topic;
import java.util.List;

public class ListTopicsExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String cloudRegion = "your-cloud-region";
    char zoneId = 'b';
    long projectNumber = Long.parseLong("123456789");

    listTopicsExample(cloudRegion, zoneId, projectNumber);
  }

  public static void listTopicsExample(String cloudRegion, char zoneId, long projectNumber)
      throws Exception {

    AdminClientSettings adminClientSettings =
        AdminClientSettings.newBuilder().setRegion(CloudRegion.of(cloudRegion)).build();

    LocationPath locationPath =
        LocationPaths.newBuilder()
            .setProjectNumber(ProjectNumber.of(projectNumber))
            .setZone(CloudZone.of(CloudRegion.of(cloudRegion), zoneId))
            .build();

    try (AdminClient adminClient = AdminClient.create(adminClientSettings)) {
      List<Topic> topics = adminClient.listTopics(locationPath).get();
      for (Topic topic : topics) {
        System.out.println(topic.getAllFields());
      }
      System.out.println(topics.size() + " topic(s) listed.");
    }
  }
}

Deleting Lite topics

You can delete Lite topics with the Cloud Console, the gcloud command-line tool, or the Pub/Sub Lite API.

Console

  1. In the Cloud Console, go to the Lite Topics page.

    Go to the Lite Topics page

  2. Click the Lite topic ID.

  3. In the Lite topic details page, click Delete.

  4. In the field that appears, enter delete to confirm that you want to delete the Lite topic.

  5. Click Delete.

gcloud

To delete a Lite topic, use the gcloud beta pubsub lite-topics delete command:

  1. Run the delete command:

    gcloud beta lite-topics delete TOPIC_ID \
     --zone=ZONE
    

    Replace the following:

    • TOPIC_ID: the ID of the Lite topic

    • ZONE: the name of the zone that the Lite topic is in

  2. To confirm, type Y.

If the request is successful, the response is the following:

Deleted topic [TOPIC_ID].

Protocol

To delete a Lite topic, send a DELETE request like the following:

DELETE https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/ZONE/topics/TOPIC_ID
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

  • REGION: the region of the zone that the Lite topic is in

  • PROJECT_NUMBER: the project number of the project with the Lite topic

  • ZONE: the name of the zone that the Lite topic is in

  • TOPIC_ID: the ID of the Lite topic

If the request is successful, the response is an empty JSON object.

Java

import com.google.cloud.pubsublite.AdminClient;
import com.google.cloud.pubsublite.AdminClientSettings;
import com.google.cloud.pubsublite.CloudRegion;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectNumber;
import com.google.cloud.pubsublite.TopicName;
import com.google.cloud.pubsublite.TopicPath;
import com.google.cloud.pubsublite.TopicPaths;

public class DeleteTopicExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String cloudRegion = "your-cloud-region";
    char zoneId = 'b';
    // Choose an existing topic.
    String topicId = "your-topic-id";
    long projectNumber = Long.parseLong("123456789");

    deleteTopicExample(cloudRegion, zoneId, projectNumber, topicId);
  }

  public static void deleteTopicExample(
      String cloudRegion, char zoneId, long projectNumber, String topicId) throws Exception {
    TopicPath topicPath =
        TopicPaths.newBuilder()
            .setProjectNumber(ProjectNumber.of(projectNumber))
            .setZone(CloudZone.of(CloudRegion.of(cloudRegion), zoneId))
            .setTopicName(TopicName.of(topicId))
            .build();

    AdminClientSettings adminClientSettings =
        AdminClientSettings.newBuilder().setRegion(CloudRegion.of(cloudRegion)).build();

    try (AdminClient adminClient = AdminClient.create(adminClientSettings)) {
      adminClient.deleteTopic(topicPath).get();
      System.out.println(topicPath.value() + " deleted successfully.");
    }
  }

If you delete a Lite topic, you can't publish messages to it. The Lite subscriptions to the Lite topic still exist, but you can't receive messages from the Lite subscriptions.