This document shows you how to associate schemas for Pub/Sub topics.
Before you begin
- Understand how Pub/Sub schemas work.
- Create a schema.
Required roles and permissions
To get the permissions that you need to associate and manage schemas,
ask your administrator to grant you the
Pub/Sub Editor (roles/pubsub.editor
) IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to associate and manage schemas. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to associate and manage schemas:
-
Create schema:
pubsub.schemas.create
-
Attach schema to topic:
pubsub.schemas.attach
-
Commit a schema revision:
pubsub.schemas.commit
-
Delete a schema or a schema revision:
pubsub.schemas.delete
-
Get a schema or schema revisions:
pubsub.schemas.get
-
List schemas:
pubsub.schemas.list
-
List schema revisions:
pubsub.schemas.listRevisions
-
Rollback a schema:
pubsub.schemas.rollback
-
Validate a message:
pubsub.schemas.validate
-
Get the IAM policy for a schema:
pubsub.schemas.getIamPolicy
-
Configure the IAM policy for a schema:
pubsub.schemas.setIamPolicy
You might also be able to get these permissions with custom roles or other predefined roles.
You can grant roles and permissions to principals such as users, groups, domains, or service accounts. You can create a schema in one project and attach it to a topic located in a different project. Ensure that you have the required permissions for each project.
Guidelines to associate a schema with a topic
You can associate a schema with a topic when you create or edit a topic. Here are the guidelines for associating a schema with a topic:
You can associate a schema with one or more topics.
After a schema is associated with a topic, every message that the topic receives from the publishers must follow that schema.
When you associate a schema with a topic, you must also specify the encoding of the messages to be published as
BINARY
orJSON
. If using JSON with an Avro schema, pay close attention to the encoding rules for unions.If a schema associated with a topic has revisions, messages must match the encoding and validate against a revision within the available range. If they don't validate, the message fails to publish.
Revisions are tried in reverse chronological order that is based on creation time. To create a schema revision, see Commit a schema revision.
Validation logic for a message schema
When you associate a schema with a topic and if the schema has revisions, you can specify a subset range of revisions to use. If you don't specify a range, the entire range is used for validation.
If you don't specify a revision as First revision allowed, then the oldest existing revision for the schema is used for validation. If you don't specify a revision as Last revision allowed, then the newest existing revision for the schema is used.
Let's take the example of Schema S
that is attached to topic T
.
Schema S
has the revision IDs A
,B
, C
, and D
created in order,
where A
is the first or oldest revision. None of the schemas are identical
to each other, or rollbacks of an existing schema.
If you only set the First revision allowed field as
B
, messages conforming only to schemaA
are rejected, while messages conforming to schemasB
,C
, andD
are accepted.If you only set the Last revision allowed field as
C
, messages that conform to schemasA
,B
, andC
are accepted, and messages that conform only to schemaD
are rejected.If you set both fields First revision allowed as
B
and Last revision allowed asC
, messages that conform to schemasB
andC
are accepted.You can also set the first and last revision to the same revision ID. In this case, only messages that conform to that revision are accepted.
Create and associate a schema when you create a topic
You can create a topic with a schema using the Google Cloud console, the gcloud CLI, the Pub/Sub API, or the Cloud Client Libraries.
Console
In the Google Cloud console, go to the Pub/Sub topics page.
Click Create topic.
In the Topic ID field, enter an ID for your topic.
To name a topic, see the guidelines.
Check the box for Use a schema.
Retain the default settings for the remaining fields.
You can create a schema or use an existing schema.
If you are creating a schema, follow these steps: `
- For Select a Pub/Sub schema, select Create a new schema.
The Create schema page is displayed in a secondary tab.
Follow the steps in Create a schema.
Go back to the Create topic tab and click Refresh.
Search for your schema in the Select a Pub/Sub schema field.
Select the message encoding as JSON or Binary.
The schema you just created has a revision ID. You can create additional schema revisions as discussed in Commit a schema revision.
If you are associating an already-created schema, follow these steps:
For Select a Pub/Sub schema, select an existing schema.
Select the message encoding as JSON or Binary.
Optional: If the selected schema has revisions, for Revision Range, use the drop-down menus for First revision allowed and Last revision allowed.
You can specify both fields, specify only one, or retain the default settings based on your requirements.
Retain the default settings for the remaining fields.
Click Create to save the topic and assign it to the selected schema.
gcloud
To create a topic assigned with a previously created schema, run the
gcloud pubsub topics create
command:
gcloud pubsub topics create TOPIC_ID \ --message-encoding=ENCODING_TYPE \ --schema=SCHEMA_ID \ --first-revision-id=FIRST_REVISION_ID \ --last-revision-id=LAST_REVISION_ID \
Where:
- TOPIC_ID is the ID for the topic that you're creating.
- ENCODING_TYPE is the encoding of messages validated against the
schema. This value must be set to either
JSON
orBINARY
. - SCHEMA_ID is the ID of an existing schema.
- FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both --first-revision-id
and --last-revision-id
are optional.
You can also assign a schema from a different Google Cloud project:
gcloud pubsub topics create TOPIC_ID \ --message-encoding=ENCODING_TYPE \ --schema=SCHEMA_ID \ --schema-project=SCHEMA_PROJECT \ --project=TOPIC_PROJECT
Where:
- SCHEMA_PROJECT is the project ID of the Google Cloud project for the schema.
- TOPIC_PROJECT is the project ID of the Google Cloud project for the topic.
REST
To create a topic, use the projects.topics.create
method:
Request:
The request must be authenticated with an access token in the
Authorization
header. To obtain an access token for the current
Application Default Credentials: gcloud auth application-default print-access-token
.
PUT https://pubsub.googleapis.com/v1/projects/PROJECT_ID/topics/TOPIC_ID Authorization: Bearer ACCESS_TOKEN
Request body:
{ "schemaSettings": { "schema": "SCHEMA_NAME", "encoding": "ENCODING_TYPE" "firstRevisionId": "FIRST_REVISION_ID" "lastRevisionId": "LAST_REVISION_ID" } }
Where:
- PROJECT_ID is your project ID.
- TOPIC_ID is your topic ID.
- SCHEMA_NAME is the name of the schema that messages published should be validated against. The format is:
projects/PROJECT_ID/schemas/SCHEMA_ID
. - ENCODING_TYPE is the encoding of messages validated against the schema. This must be set to either
JSON
orBINARY
. - FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both firstRevisionId
and lastRevisionId
are optional.
Response:
{ "name": "projects/PROJECT_ID/topics/TOPIC_ID", "schemaSettings": { "schema": "SCHEMA_NAME", "encoding": "ENCODING_TYPE" "firstRevisionId": "FIRST_REVISION_ID" "lastRevisionId": "LAST_REVISION_ID" } }
Both firstRevisionId
and lastRevisionId
are omitted if not provided
in the request.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
C#
Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
Node.js
Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.
PHP
Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
Ruby
Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.
Edit a schema associated with a topic
You can edit a topic to attach a schema, remove a schema, or update the revision range used for validating messages. In general, if you have changes planned for the schema in use, you can commit a new revision and update the range of revisions used for the topic.
You can edit a schema associated with a topic using the Google Cloud console, the gcloud CLI, the Pub/Sub API, or the Cloud Client Libraries.
Console
In the Google Cloud console, go to the Pub/Sub topics page.
Click the Topic ID of a topic.
In the topic details page, click Edit.
You can make the following changes to the schema.
It might take a few minutes for the changes to take place.
If you want to remove the schema from the topic, in the Edit topic page, clear the Use a schema checkbox.
If you want to change the schema, in the Schema section, select the name of a schema.
Update the other fields as required.
- If you want to update the revision range, for Revision Range, use the drop-down menus for First revision allowed and Last revision allowed.
You can specify both fields, specify only one, or retain the default settings based on your requirements.
Click Update to save the changes.
gcloud
gcloud pubsub topics update TOPIC_ID \ --message-encoding=ENCODING_TYPE \ --schema=SCHEMA_NAME \ --first-revision-id=FIRST_REVISION_ID \ --last-revision-id=LAST_REVISION_ID \
Where:
- TOPIC_ID is the ID for the topic that you're creating.
- ENCODING_TYPE is the encoding of messages validated against the
schema. This value must be set to either
JSON
orBINARY
. - SCHEMA_NAME is the name of an existing schema.
- FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both --first-revision-id
and --last-revision-id
are optional.
REST
To update a topic, use the projects.topics.patch
method:
Request:
The request must be authenticated with an access token in the
Authorization
header. To obtain an access token for the current
Application Default Credentials: gcloud auth application-default print-access-token
.
PATCH https://pubsub.googleapis.com/v1/projects/PROJECT_ID/topics/TOPIC_ID Authorization: Bearer ACCESS_TOKEN
Request body:
{ "schemaSettings": { "schema": "SCHEMA_NAME", "encoding": "ENCODING_TYPE" "firstRevisionId": "FIRST_REVISION_ID" "lastRevisionId": "LAST_REVISION_ID" "update_mask": } }
Where:
- PROJECT_ID is your project ID.
- TOPIC_ID is your topic ID.
- SCHEMA_NAME is the name of the schema that messages published should be validated against. The format is:
projects/PROJECT_ID/schemas/SCHEMA_ID
. - ENCODING_TYPE is the encoding of messages validated against the schema. This must be set to either
JSON
orBINARY
. - FIRST_REVISION_ID is the ID of oldest revision to validate against.
- LAST_REVISION_ID is the ID of the newest revision to validate against.
Both firstRevisionId
and lastRevisionId
are optional.
Response:
{ "name": "projects/PROJECT_ID/topics/TOPIC_ID", "schemaSettings": { "schema": "SCHEMA_NAME", "encoding": "ENCODING_TYPE" "firstRevisionId": "FIRST_REVISION_ID" "lastRevisionId": "LAST_REVISION_ID" } }
Both firstRevisionId
and lastRevisionId
they are not set after the
update.
C++
Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.
Go
Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.
Java
Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.
0What's next
- Commit a schema revision
- Publish messages to a topic with a schema
- Validate a schema definition
- Validate a message for a schema