Creating and managing schemas

This page shows you how to create and manage schemas for Pub/Sub topics.

A schema is a format that messages must follow, creating a contract between publisher and subscriber that Pub/Sub will enforce. They also facilitate inter-team consumption of data streams in your organization by creating a central authority for message types and permissions.

A Pub/Sub schema defines the names and data types of fields in a message. You can create schemas as standalone versioned resources, associate schemas with multiple Pub/Sub topics, and use them to validate the structure of published messages.

Creating schemas

You can create a schema and assign it one or more topics, or create a schema during topic creation. After a schema is assigned to a topic, every message that the topic receives from publishers must follow that schema.

You can create a schema using the Cloud Console, the gcloud tool, and the Pub/Sub API:

Console

To create a schema, follow these steps:

  1. In the Cloud Console, go to the Pub/Sub schemas page.

    Go to the schemas page

  2. Click Create schema.

  3. In the Schema ID field, enter an ID for your schema.

  4. For Schema type, select either Avro or Protocol Buffer. Learn more about schema types below.

  5. In the Schema definition field, enter the Avro of Protocol Buffer definition for your schema.

  6. Click Create to save the schema.

gcloud

gcloud pubsub schemas create SCHEMA_ID \
        --type=SCHEMA_TYPE \
        --definition=SCHEMA_DEFINITION

Where:

  • SCHEMA_TYPE is either AVRO or PROTOCOL_BUFFER.
  • SCHEMA_DEFINITION is a string containing the definition of the schema, formatted according to the chosen schema type.

REST

To create a schema, send a POST request like the following:

POST https://pubsub.googleapis.com/v1/projects/PROJECT_ID/schemas/SCHEMA_ID
Authorization: Bearer $(gcloud auth application-default print-access-token)
Content-Type: application/json --data @response-body.json

Specify the following fields in the request body:

{
  "definition": SCHEMA_DEFINITION
  "type": SCHEMA_TYPE
}

Where:

  • SCHEMA_TYPE is either AVRO or PROTOCOL_BUFFER.
  • SCHEMA_DEFINITION is a string containing the definition of the schema, formatted according to the chosen schema type.

The response body should contain a JSON representation of a schema resource. For example:

{
  "name": SCHEMA_NAME,
  "type": SCHEMA_TYPE,
  "definition": SCHEMA_DEFINITION
}

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

Avro
namespace pubsub = ::google::cloud::pubsub;
[](pubsub::SchemaAdminClient client, std::string const& project_id,
   std::string const& schema_id) {
  auto constexpr kDefinition = R"js({
    "type": "record",
    "name": "State",
    "namespace": "utilities",
    "doc": "A list of states in the United States of America.",
    "fields": [
      {
        "name": "name",
        "type": "string",
        "doc": "The common name of the state."
      },
      {
        "name": "post_abbr",
        "type": "string",
        "doc": "The postal code abbreviation of the state."
      }
    ]
  })js";
  auto schema = client.CreateAvroSchema(pubsub::Schema(project_id, schema_id),
                                        kDefinition);
  if (schema.status().code() == google::cloud::StatusCode::kAlreadyExists) {
    std::cout << "The schema already exists\n";
    return;
  }
  if (!schema) throw std::runtime_error(schema.status().message());

  std::cout << "Schema successfully created: " << schema->DebugString()
            << "\n";
}
Proto
namespace pubsub = ::google::cloud::pubsub;
[](pubsub::SchemaAdminClient client, std::string const& project_id,
   std::string const& schema_id) {
  auto constexpr kDefinition = R"pfile(
      syntax = "proto3";
      package google.cloud.pubsub.samples;

      message State {
        string name = 1;
        string post_abbr = 2;
      }
      )pfile";
  auto schema = client.CreateProtobufSchema(
      pubsub::Schema(project_id, schema_id), kDefinition);
  if (schema.status().code() == google::cloud::StatusCode::kAlreadyExists) {
    std::cout << "The schema already exists\n";
    return;
  }
  if (!schema) return;  // TODO(#4792) - protobuf schema support in emulator
  std::cout << "Schema successfully created: " << schema->DebugString()
            << "\n";
}

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.

Avro

using Google.Cloud.PubSub.V1;
using Grpc.Core;
using System;
using System.IO;

public class CreateAvroSchemaSample
{
    public Schema CreateAvroSchema(string projectId, string schemaId, string pathToDefinition)
    {
        SchemaServiceClient schemaService = SchemaServiceClient.Create();
        var schemaName = SchemaName.FromProjectSchema(projectId, schemaId);
        string schemaDefinition = File.ReadAllText(pathToDefinition);
        Schema schema = new Schema
        {
            Name = schemaName.ToString(),
            Type = Schema.Types.Type.Avro,
            Definition = schemaDefinition
        };
        CreateSchemaRequest createSchemaRequest = new CreateSchemaRequest
        {
            Parent = "projects/" + projectId,
            SchemaId = schemaId,
            Schema = schema
        };

        try
        {
            schema = schemaService.CreateSchema(createSchemaRequest);
            Console.WriteLine($"Schema {schema.Name} created.");
        }
        catch (RpcException e) when (e.Status.StatusCode == StatusCode.AlreadyExists)
        {
            Console.WriteLine($"Schema {schemaName} already exists.");
        }
        return schema;
    }
}
Proto

using Google.Cloud.PubSub.V1;
using Grpc.Core;
using System;
using System.IO;

public class CreateProtoSchemaSample
{
    public Schema CreateProtoSchema(string projectId, string schemaId, string pathToDefinition)
    {
        SchemaServiceClient schemaService = SchemaServiceClient.Create();
        var schemaName = SchemaName.FromProjectSchema(projectId, schemaId);
        string schemaDefinition = File.ReadAllText(pathToDefinition);
        Schema schema = new Schema
        {
            Name = schemaName.ToString(),
            Type = Schema.Types.Type.ProtocolBuffer,
            Definition = schemaDefinition
        };
        CreateSchemaRequest createSchemaRequest = new CreateSchemaRequest
        {
            Parent = "projects/" + projectId,
            SchemaId = schemaId,
            Schema = schema
        };

        try
        {
            schema = schemaService.CreateSchema(createSchemaRequest);
            Console.WriteLine($"Schema {schema.Name} created.");
        }
        catch (RpcException e) when (e.Status.StatusCode == StatusCode.AlreadyExists)
        {
            Console.WriteLine($"Schema {schemaName} already exists.");
        }
        return schema;
    }
}

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

Avro
import (
	"context"
	"fmt"
	"io"
	"io/ioutil"

	"cloud.google.com/go/pubsub"
)

// createAvroSchema creates a schema resource from a JSON-formatted Avro schema file.
func createAvroSchema(w io.Writer, projectID, schemaID, avscFile string) error {
	// projectID := "my-project-id"
	// schemaID := "my-schema"
	// avscFile = "path/to/an/avro/schema/file(.avsc)/formatted/in/json"
	ctx := context.Background()
	client, err := pubsub.NewSchemaClient(ctx, projectID)
	if err != nil {
		return fmt.Errorf("pubsub.NewSchemaClient: %v", err)
	}
	defer client.Close()

	avscSource, err := ioutil.ReadFile(avscFile)
	if err != nil {
		return fmt.Errorf("error reading from file: %s", avscFile)
	}

	config := pubsub.SchemaConfig{
		Type:       pubsub.SchemaAvro,
		Definition: string(avscSource),
	}
	s, err := client.CreateSchema(ctx, schemaID, config)
	if err != nil {
		return fmt.Errorf("CreateSchema: %v", err)
	}
	fmt.Fprintf(w, "Schema created: %#v\n", s)
	return nil
}
Proto
import (
	"context"
	"fmt"
	"io"
	"io/ioutil"

	"cloud.google.com/go/pubsub"
)

// createProtoSchema creates a schema resource from a schema proto file.
func createProtoSchema(w io.Writer, projectID, schemaID, protoFile string) error {
	// projectID := "my-project-id"
	// schemaID := "my-schema"
	// protoFile = "path/to/a/proto/schema/file(.proto)/formatted/in/protocol/buffers"
	ctx := context.Background()
	client, err := pubsub.NewSchemaClient(ctx, projectID)
	if err != nil {
		return fmt.Errorf("pubsub.NewSchemaClient: %v", err)
	}
	defer client.Close()

	protoSource, err := ioutil.ReadFile(protoFile)
	if err != nil {
		return fmt.Errorf("error reading from file: %s", protoFile)
	}

	config := pubsub.SchemaConfig{
		Type:       pubsub.SchemaProtocolBuffer,
		Definition: string(protoSource),
	}
	s, err := client.CreateSchema(ctx, schemaID, config)
	if err != nil {
		return fmt.Errorf("CreateSchema: %v", err)
	}
	fmt.Fprintf(w, "Schema created: %#v\n", s)
	return nil
}

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.

Avro

import com.google.api.gax.rpc.AlreadyExistsException;
import com.google.cloud.pubsub.v1.SchemaServiceClient;
import com.google.pubsub.v1.ProjectName;
import com.google.pubsub.v1.Schema;
import com.google.pubsub.v1.SchemaName;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class CreateAvroSchemaExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String schemaId = "your-schema-id";
    String avscFile = "path/to/an/avro/schema/file/(.avsc)/formatted/in/json";

    createAvroSchemaExample(projectId, schemaId, avscFile);
  }

  public static void createAvroSchemaExample(String projectId, String schemaId, String avscFile)
      throws IOException {

    ProjectName projectName = ProjectName.of(projectId);
    SchemaName schemaName = SchemaName.of(projectId, schemaId);

    // Read an Avro schema file formatted in JSON as a string.
    String avscSource = new String(Files.readAllBytes(Paths.get(avscFile)));

    try (SchemaServiceClient schemaServiceClient = SchemaServiceClient.create()) {

      Schema schema =
          schemaServiceClient.createSchema(
              projectName,
              Schema.newBuilder()
                  .setName(schemaName.toString())
                  .setType(Schema.Type.AVRO)
                  .setDefinition(avscSource)
                  .build(),
              schemaId);

      System.out.println("Created a schema using an Avro schema:\n" + schema);
    } catch (AlreadyExistsException e) {
      System.out.println(schemaName + "already exists.");
    }
  }
}
Protocol Buffer

import com.google.api.gax.rpc.AlreadyExistsException;
import com.google.cloud.pubsub.v1.SchemaServiceClient;
import com.google.pubsub.v1.ProjectName;
import com.google.pubsub.v1.Schema;
import com.google.pubsub.v1.SchemaName;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class CreateProtoSchemaExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String schemaId = "your-schema-id";
    String protoFile = "path/to/a/proto/file/(.proto)/formatted/in/protocol/buffers";

    createProtoSchemaExample(projectId, schemaId, protoFile);
  }

  public static void createProtoSchemaExample(String projectId, String schemaId, String protoFile)
      throws IOException {

    ProjectName projectName = ProjectName.of(projectId);
    SchemaName schemaName = SchemaName.of(projectId, schemaId);

    // Read a proto file as a string.
    String protoSource = new String(Files.readAllBytes(Paths.get(protoFile)));

    try (SchemaServiceClient schemaServiceClient = SchemaServiceClient.create()) {

      Schema schema =
          schemaServiceClient.createSchema(
              projectName,
              Schema.newBuilder()
                  .setName(schemaName.toString())
                  .setType(Schema.Type.PROTOCOL_BUFFER)
                  .setDefinition(protoSource)
                  .build(),
              schemaId);

      System.out.println("Created a schema using a protobuf schema:\n" + schema);
    } catch (AlreadyExistsException e) {
      System.out.println(schemaName + "already exists.");
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.

Avro
/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
// const schemaName = 'YOUR_SCHEMA_NAME';
// const avscFile = 'path/to/an/avro/schema/file/(.avsc)/formatted/in/json';

// Imports the Google Cloud client library
const {PubSub, SchemaTypes} = require('@google-cloud/pubsub');
const fs = require('fs');

// Creates a client; cache this for further use
const pubSubClient = new PubSub();

async function createAvroSchema(schemaName, avscFile) {
  const definition = fs.readFileSync(avscFile).toString();
  const schema = await pubSubClient.createSchema(
    schemaName,
    SchemaTypes.Avro,
    definition
  );

  const name = await schema.getName();
  console.log(`Schema ${name} created.`);
}
Protocol Buffer
/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
// const schemaName = 'YOUR_SCHEMA_NAME';
// const protoFile = 'path/to/a/proto/schema/file/(.proto)/formatted/in/protcol/buffers';

// Imports the Google Cloud client library
const {PubSub, SchemaTypes} = require('@google-cloud/pubsub');
const fs = require('fs');

// Creates a client; cache this for further use
const pubSubClient = new PubSub();

async function createProtoSchema(schemaName, protoFile) {
  const definition = fs.readFileSync(protoFile).toString();
  const schema = await pubSubClient.createSchema(
    schemaName,
    SchemaTypes.ProtocolBuffer,
    definition
  );

  const fullName = await schema.getName();
  console.log(`Schema ${fullName} created.`);
}

PHP

Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.

Avro
use Google\Cloud\PubSub\PubSubClient;
use Google\Cloud\PubSub\V1\Schema\Type;

/**
 * Create a Schema with an AVRO definition.
 *
 * @param string $projectId
 * @param string $schemaId
 * @param string $avscFile
 */
function create_avro_schema($projectId, $schemaId, $avscFile)
{
    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);

    $definition = file_get_contents($avscFile);
    $schema = $pubsub->createSchema($schemaId, Type::AVRO, $definition);

    printf('Schema %s created.', $schema->name());
}
Protocol Buffer
use Google\Cloud\PubSub\PubSubClient;
use Google\Cloud\PubSub\V1\Schema\Type;

/**
 * Create a Schema with an Protocol Buffer definition.
 *
 * @param string $projectId
 * @param string $schemaId
 * @param string $protoFile
 */
function create_proto_schema($projectId, $schemaId, $protoFile)
{
    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);

    $definition = file_get_contents($protoFile);
    $schema = $pubsub->createSchema($schemaId, Type::PROTOCOL_BUFFER, $definition);

    printf('Schema %s created.', $schema->name());
}

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

Avro
from google.api_core.exceptions import AlreadyExists
from google.cloud.pubsub import SchemaServiceClient
from google.pubsub_v1.types import Schema

# TODO(developer): Replace these variables before running the sample.
# project_id = "your-project-id"
# schema_id = "your-schema-id"
# avsc_file = "path/to/an/avro/schema/file/(.avsc)/formatted/in/json"

project_path = f"projects/{project_id}"

# Read a JSON-formatted Avro schema file as a string.
with open(avsc_file, "rb") as f:
    avsc_source = f.read().decode("utf-8")

schema_client = SchemaServiceClient()
schema_path = schema_client.schema_path(project_id, schema_id)
schema = Schema(name=schema_path, type_=Schema.Type.AVRO, definition=avsc_source)

try:
    result = schema_client.create_schema(
        request={"parent": project_path, "schema": schema, "schema_id": schema_id}
    )
    print(f"Created a schema using an Avro schema file:\n{result}")
except AlreadyExists:
    print(f"{schema_id} already exists.")
Protocol Buffer
from google.api_core.exceptions import AlreadyExists
from google.cloud.pubsub import SchemaServiceClient
from google.pubsub_v1.types import Schema

# TODO(developer): Replace these variables before running the sample.
# project_id = "your-project-id"
# schema_id = "your-schema-id"
# proto_file = "path/to/a/proto/file/(.proto)/formatted/in/protocol/buffers"

project_path = f"projects/{project_id}"

# Read a protobuf schema file as a string.
with open(proto_file, "rb") as f:
    proto_source = f.read().decode("utf-8")

schema_client = SchemaServiceClient()
schema_path = schema_client.schema_path(project_id, schema_id)
schema = Schema(
    name=schema_path, type_=Schema.Type.PROTOCOL_BUFFER, definition=proto_source
)

try:
    result = schema_client.create_schema(
        request={"parent": project_path, "schema": schema, "schema_id": schema_id}
    )
    print(f"Created a schema using a protobuf schema file:\n{result}")
except AlreadyExists:
    print(f"{schema_id} already exists.")

Ruby

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.

Avro
# schema_id = "your-schema-id"
# avsc_file = "path/to/an/avro/schema/file/(.avsc)/formatted/in/json"
require "google/cloud/pubsub"

pubsub = Google::Cloud::Pubsub.new

definition = File.read avsc_file
schema = pubsub.create_schema schema_id, :avro, definition

puts "Schema #{schema.name} created."
Protocol Buffer
# schema_id = "your-schema-id"
# proto_file = "path/to/a/proto/file/(.proto)/formatted/in/protocol/buffers"
require "google/cloud/pubsub"

pubsub = Google::Cloud::Pubsub.new

definition = File.read proto_file
schema = pubsub.create_schema schema_id, :protocol_buffer, definition

puts "Schema #{schema.name} created."

You must assign the schema to a topic in order to use the schema. A single schema can be associated with any number of topics.

Schema types

You can create a schema using the following frameworks:

For example, the following schema defines a message with a string field, a float field, and boolean field:

Avro

{
 "type" : "record",
 "name" : "Avro",
 "fields" : [
   {
     "name" : "StringField",
     "type" : "string"
   },
   {
     "name" : "FloatField",
     "type" : "float"
   },
   {
     "name" : "BooleanField",
     "type" : "boolean"
   },
 ]
}

Protocol Buffer

syntax = "proto3";
message ProtocolBuffer {
  string string_field = 1;
  float float_field = 2;
  bool boolean_field = 3;
}

For a schema to work with Pub/Sub, only one top-level type may be defined. Import statements to reference other types are also not supported.

Getting schema details

You can get the details of a schema by its name using the Cloud Console, the gcloud tool, and the Pub/Sub API:

Console

  1. In the Cloud Console, go to the Pub/Sub schemas page.

    Go to the schemas page

  2. Select the schema you want to view from the list.

gcloud

gcloud pubsub schemas describe SCHEMA_NAME

REST

To get the details of schema, send a GET request like the following:

GET https://pubsub.googleapis.com/v1/SCHEMA_NAME

If successful, the response body will contain an instance of the Schema class.

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

namespace pubsub = ::google::cloud::pubsub;
[](pubsub::SchemaAdminClient client, std::string const& project_id,
   std::string const& schema_id) {
  auto schema = client.GetSchema(pubsub::Schema(project_id, schema_id),
                                 google::pubsub::v1::FULL);
  if (!schema) throw std::runtime_error(schema.status().message());

  std::cout << "The schema exists and its metadata is: "
            << schema->DebugString() << "\n";
}

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.


using Google.Cloud.PubSub.V1;

public class GetSchemaSample
{
    public Schema GetSchema(string projectId, string schemaId)
    {
        SchemaServiceClient schemaService = SchemaServiceClient.Create();
        SchemaName schemaName = SchemaName.FromProjectSchema(projectId, schemaId);
        GetSchemaRequest request = new GetSchemaRequest
        {
            Name = schemaName.ToString(),
            View = SchemaView.Full
        };

        return schemaService.GetSchema(request);
    }
}

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

import (
	"context"
	"fmt"
	"io"

	"cloud.google.com/go/pubsub"
)

func getSchema(w io.Writer, projectID, schemaID string) error {
	// projectID := "my-project-id"
	// schemaID := "my-schema"
	ctx := context.Background()
	client, err := pubsub.NewSchemaClient(ctx, projectID)
	if err != nil {
		return fmt.Errorf("pubsub.NewSchemaClient: %v", err)
	}
	defer client.Close()

	// Retrieve the full schema view. If you don't want to retrive the
	// definition, pass in pubsub.SchemaViewBasic which retrieves
	// just the name and type of the schema.
	s, err := client.Schema(ctx, schemaID, pubsub.SchemaViewFull)
	if err != nil {
		return fmt.Errorf("client.Schema: %v", err)
	}
	fmt.Fprintf(w, "Got schema: %#v\n", s)
	return nil
}

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.


import com.google.api.gax.rpc.NotFoundException;
import com.google.cloud.pubsub.v1.SchemaServiceClient;
import com.google.pubsub.v1.Schema;
import com.google.pubsub.v1.SchemaName;
import java.io.IOException;

public class GetSchemaExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String schemaId = "your-schema-id";

    getSchemaExample(projectId, schemaId);
  }

  public static void getSchemaExample(String projectId, String schemaId) throws IOException {
    SchemaName schemaName = SchemaName.of(projectId, schemaId);

    try (SchemaServiceClient schemaServiceClient = SchemaServiceClient.create()) {

      Schema schema = schemaServiceClient.getSchema(schemaName);

      System.out.println("Got a schema:\n" + schema);

    } catch (NotFoundException e) {
      System.out.println(schemaName + "not found.");
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.

/**
 * TODO(developer): Uncomment this variable before running the sample.
 */
// const schemaName = 'YOUR_SCHEMA_NAME';

// Imports the Google Cloud client library
const {PubSub} = require('@google-cloud/pubsub');

// Creates a client; cache this for further use
const pubSubClient = new PubSub();

async function getSchema(schemaName) {
  const schema = pubSubClient.schema(schemaName);
  const info = await schema.get();
  const fullName = await schema.getName();
  console.log(`Schema ${fullName} info: ${JSON.stringify(info, null, 4)}.`);
}

PHP

Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.

use Google\Cloud\PubSub\PubSubClient;

/**
 * Get a schema.
 *
 * @param string $projectId
 * @param string $schemaId
 */
function get_schema($projectId, $schemaId)
{
    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);

    $schema = $pubsub->schema($schemaId);
    $schema->info();

    printf('Schema %s retrieved', $schema->name());
}

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

from google.api_core.exceptions import NotFound
from google.cloud.pubsub import SchemaServiceClient

# TODO(developer): Replace these variables before running the sample.
# project_id = "your-project-id"
# schema_id = "your-schema-id"

schema_client = SchemaServiceClient()
schema_path = schema_client.schema_path(project_id, schema_id)

try:
    result = schema_client.get_schema(request={"name": schema_path})
    print(f"Got a schema:\n{result}")
except NotFound:
    print(f"{schema_id} not found.")

Ruby

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.

# schema_id = "your-schema-id"
require "google/cloud/pubsub"

pubsub = Google::Cloud::Pubsub.new

schema = pubsub.schema schema_id

puts "Schema #{schema.name} retrieved."

Listing schemas

You can list the schemas in a Google Cloud project using the Cloud Console, the gcloud tool, and the Pub/Sub API:

Console

In the Cloud Console, go to the Pub/Sub schemas page. This page contains a list of all schemas in the current project.

Go to the schemas page

gcloud

gcloud pubsub schemas list

REST

To list the schemas in a project, send a GET request like the following:

GET https://pubsub.googleapis.com/v1/projects/PROJECT_ID/schemas

If successful, the response body will contain a JSON object containing all schemas in the project.

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

namespace pubsub = ::google::cloud::pubsub;
[](pubsub::SchemaAdminClient client, std::string const& project_id) {
  for (auto const& schema :
       client.ListSchemas(project_id, google::pubsub::v1::FULL)) {
    if (!schema) throw std::runtime_error(schema.status().message());
    std::cout << "Schema: " << schema->DebugString() << "\n";
  }
}

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.


using Google.Api.Gax.ResourceNames;
using Google.Cloud.PubSub.V1;
using System.Collections.Generic;

public class ListSchemasSample
{
    public IEnumerable<Schema> ListSchemas(string projectId)
    {
        SchemaServiceClient schemaService = SchemaServiceClient.Create();
        ProjectName projectName = ProjectName.FromProject(projectId);
        var schemas = schemaService.ListSchemas(projectName);
        return schemas;
    }
}

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

import (
	"context"
	"fmt"
	"io"

	"cloud.google.com/go/pubsub"
	"google.golang.org/api/iterator"
)

func listSchemas(w io.Writer, projectID string) ([]*pubsub.SchemaConfig, error) {
	// projectID := "my-project-id"
	ctx := context.Background()
	client, err := pubsub.NewSchemaClient(ctx, projectID)
	if err != nil {
		return nil, fmt.Errorf("pubsub.NewSchemaClient: %v", err)
	}
	defer client.Close()

	var schemas []*pubsub.SchemaConfig

	schemaIter := client.Schemas(ctx, pubsub.SchemaViewFull)
	for {
		sc, err := schemaIter.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return nil, fmt.Errorf("schemaIter.Next: %v", err)
		}
		fmt.Fprintf(w, "Got schema: %#v\n", sc)
		schemas = append(schemas, sc)
	}

	fmt.Fprintf(w, "Got %d schemas", len(schemas))
	return schemas, nil
}

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.

import com.google.cloud.pubsub.v1.SchemaServiceClient;
import com.google.pubsub.v1.ProjectName;
import com.google.pubsub.v1.Schema;
import java.io.IOException;

public class ListSchemasExample {
  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";

    listSchemasExample(projectId);
  }

  public static void listSchemasExample(String projectId) throws IOException {
    ProjectName projectName = ProjectName.of(projectId);

    try (SchemaServiceClient schemaServiceClient = SchemaServiceClient.create()) {
      for (Schema schema : schemaServiceClient.listSchemas(projectName).iterateAll()) {
        System.out.println(schema);
      }
      System.out.println("Listed schemas.");
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.


// Imports the Google Cloud client library
const {PubSub} = require('@google-cloud/pubsub');

// Creates a client; cache this for further use
const pubSubClient = new PubSub();

async function listSchemas() {
  for await (const s of pubSubClient.listSchemas()) {
    console.log(await s.name);
  }
  console.log('Listed schemas.');
}

PHP

Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.

use Google\Cloud\PubSub\PubSubClient;

/**
 * List schemas in the project.
 *
 * @param string $projectId
 */
function list_schemas($projectId)
{
    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);

    $schemas = $pubsub->schemas();
    foreach ($schemas as $schema) {
        printf('Schema name: %s' . PHP_EOL, $schema->name());
    }
}

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

from google.cloud.pubsub import SchemaServiceClient

# TODO(developer): Replace these variables before running the sample.
# project_id = "your-project-id"

project_path = f"projects/{project_id}"
schema_client = SchemaServiceClient()

for schema in schema_client.list_schemas(request={"parent": project_path}):
    print(schema)

print("Listed schemas.")

Ruby

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.

require "google/cloud/pubsub"

pubsub = Google::Cloud::Pubsub.new

schemas = pubsub.schemas

puts "Schemas in project:"
schemas.each do |schema|
  puts schema.name
end

Deleting schemas

You can delete a schema using the Cloud Console, the gcloud tool, and the Pub/Sub API:

Console

  1. In the Cloud Console, go to the Pub/Sub schemas page.

    Go to the schemas page

  2. Select the schema you want to delete from the list.

  3. Click Delete.

gcloud

gcloud pubsub schemas delete SCHEMA_NAME

REST

To delete a schema send a DELETE request like the following:

DELETE https://pubsub.googleapis.com/v1/SCHEMA_NAME

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

namespace pubsub = ::google::cloud::pubsub;
[](pubsub::SchemaAdminClient client, std::string const& project_id,
   std::string const& schema_id) {
  auto status = client.DeleteSchema(pubsub::Schema(project_id, schema_id));
  // Note that kNotFound is a possible result when the library retries.
  if (status.code() == google::cloud::StatusCode::kNotFound) {
    std::cout << "The schema was not found\n";
    return;
  }
  if (!status.ok()) throw std::runtime_error(status.message());

  std::cout << "Schema successfully deleted\n";
}

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.


using Google.Cloud.PubSub.V1;

public class DeleteSchemaSample
{
    public void DeleteSchema(string projectId, string schemaId)
    {
        SchemaServiceClient schemaService = SchemaServiceClient.Create();
        SchemaName schemaName = SchemaName.FromProjectSchema(projectId, schemaId);
        schemaService.DeleteSchema(schemaName);
    }
}

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

import (
	"context"
	"fmt"
	"io"

	"cloud.google.com/go/pubsub"
)

func deleteSchema(w io.Writer, projectID, schemaID string) error {
	// projectID := "my-project-id"
	// schemaID := "my-schema"
	ctx := context.Background()
	client, err := pubsub.NewSchemaClient(ctx, projectID)
	if err != nil {
		return fmt.Errorf("pubsub.NewSchemaClient: %v", err)
	}
	defer client.Close()

	if err := client.DeleteSchema(ctx, schemaID); err != nil {
		return fmt.Errorf("client.DeleteSchema: %v", err)
	}
	fmt.Fprintf(w, "Deleted schema: %s", schemaID)
	return nil
}

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.


import com.google.api.gax.rpc.NotFoundException;
import com.google.cloud.pubsub.v1.SchemaServiceClient;
import com.google.pubsub.v1.SchemaName;
import java.io.IOException;

public class DeleteSchemaExample {

  public static void main(String... args) throws Exception {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String schemaId = "your-schema-id";

    deleteSchemaExample(projectId, schemaId);
  }

  public static void deleteSchemaExample(String projectId, String schemaId) throws IOException {
    SchemaName schemaName = SchemaName.of(projectId, schemaId);

    try (SchemaServiceClient schemaServiceClient = SchemaServiceClient.create()) {

      schemaServiceClient.deleteSchema(schemaName);

      System.out.println("Deleted a schema:" + schemaName);

    } catch (NotFoundException e) {
      System.out.println(schemaName + "not found.");
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Node.js API reference documentation.

/**
 * TODO(developer): Uncomment this variable before running the sample.
 */
// const schemaName = 'YOUR_SCHEMA_NAME';

// Imports the Google Cloud client library
const {PubSub} = require('@google-cloud/pubsub');

// Creates a client; cache this for further use
const pubSubClient = new PubSub();

async function deleteSchema(schemaName) {
  const schema = pubSubClient.schema(schemaName);
  const name = await schema.getName();
  await schema.delete();
  console.log(`Schema ${name} deleted.`);
}

PHP

Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub PHP API reference documentation.

use Google\Cloud\PubSub\PubSubClient;

/**
 * Delete a schema.
 *
 * @param string $projectId
 * @param string $schemaId
 */
function delete_schema($projectId, $schemaId)
{
    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);

    $schema = $pubsub->schema($schemaId);

    if ($schema->exists()) {
        $schema->delete();

        printf('Schema %s deleted.', $schema->name());
    } else {
        printf('Schema %s does not exist.', $schema->name());
    }
}

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

from google.api_core.exceptions import NotFound
from google.cloud.pubsub import SchemaServiceClient

# TODO(developer): Replace these variables before running the sample.
# project_id = "your-project-id"
# schema_id = "your-schema-id"

schema_client = SchemaServiceClient()
schema_path = schema_client.schema_path(project_id, schema_id)

try:
    schema_client.delete_schema(request={"name": schema_path})
    print(f"Deleted a schema:\n{schema_path}")
except NotFound:
    print(f"{schema_id} not found.")

Ruby

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Ruby API reference documentation.

# schema_id = "your-schema-id"
require "google/cloud/pubsub"

pubsub = Google::Cloud::Pubsub.new

schema = pubsub.schema schema_id
schema.delete

puts "Schema #{schema_id} deleted."

Validating message schemas

You can validate that messages adhere to a certain schema, before or after a schema resource is created. This can be useful to ensure that the messages you intend to send through a topic associated with a schema actually match before applying the schema (which cannot be undone).

To validate a message against a schema that has not yet been created, pass in the schema type and definition:

gcloud

gcloud pubsub schemas validate-message \
        --type=SCHEMA_TYPE \
        --definition=SCHEMA_DEFINITION \
        --message-encoding=MESSAGE_ENCODING \
        --message=MESSAGE

Where:

  • SCHEMA_TYPE is either AVRO or PROTOCOL_BUFFER.
  • SCHEMA_DEFINITION is a string containing the definition of the schema, formatted according to the chosen schema type.
  • MESSAGE_ENCODING is either JSON or BINARY.
  • MESSAGE is the message to validate. To be valid, it must be encoded according to the specified MESSAGE_ENCODING, and adhere to the specified SCHEMA_DEFINITION.

REST

Request:

POST https://pubsub.googleapis.com/v1/projects/PROJECT_ID/schemas/:validateMessage
Authorization: Bearer $(gcloud auth application-default print-access-token)

Specify the following fields in the request body:

{
  "schema": {
    "definition": SCHEMA_DEFINITION
    "type": SCHEMA_TYPE
  }
  "encoding": MESSAGE_ENCODING
  "message": MESSAGE
}

Where:

  • SCHEMA_TYPE is either AVRO or PROTOCOL_BUFFER.
  • SCHEMA_DEFINITION is a string containing the definition of the schema, formatted according to the chosen schema type.
  • MESSAGE_ENCODING is either JSON or BINARY.
  • MESSAGE is the based64-encoded message to validate. To be valid, it must be encoded according to the specified MESSAGE_ENCODING, and adhere to the specified SCHEMA_DEFINITION.

If the request is successful, the response is an empty JSON object.

To validate a message against an existing schema, pass in the schema name:

gcloud

gcloud pubsub schemas validate-message \
        --message-encoding=MESSAGE_ENCODING \
        --message=MESSAGE \
        --schema-name=SCHEMA_NAME

Where:

  • SCHEMA_NAME is the name of an existing schema.
  • MESSAGE_ENCODING is either JSON or BINARY.
  • MESSAGE is the message to validate. To be valid, it must be encoded according to the specified MESSAGE_ENCODING, and adhere to the schema definition of SCHEMA_NAME.

REST

Request:

POST https://pubsub.googleapis.com/v1/projects/PROJECT_ID/schemas/:validateMessage
Authorization: Bearer $(gcloud auth application-default print-access-token)

Specify the following fields in the request body:

{
  "schema": {
    "name": SCHEMA_NAME
  }
  "encoding": MESSAGE_ENCODING
  "message": MESSAGE
}

Where:

  • SCHEMA_NAME is the name of an existing schema.
  • MESSAGE_ENCODING is either JSON or BINARY.
  • MESSAGE is the based64-encoded message to validate. To be valid, it must be encoded according to the specified MESSAGE_ENCODING, and adhere to the specified SCHEMA_DEFINITION.

If the request is successful, the response is an empty JSON object.