Filtering messages

This page explains how to create subscriptions with filters.

When you receive messages from a subscription with a filter, you only receive the messages that match the filter. The Pub/Sub service automatically acknowledges the messages that don't match the filter. You can filter messages by their attributes.

When you receive messages from a subscription with a filter, you don't incur egress fees for the messages that Pub/Sub automatically acknowledges. You incur message delivery fees and seek-related storage fees for these messages.

Creating subscriptions with filters

Pull and push subscriptions can have filters. All subscribers can receive messages from subscriptions with filters, including subscribers that use the StreamingPull API.

You can create a subscription with a filter using the Cloud Console, the gcloud command-line tool, or the Pub/Sub API.

Console

To create a pull subscription with a filter, follow these steps:

  1. In the Cloud Console, go to the Subscriptions page.

    Go to the Subscriptions page

  2. Click Create subscription.

  3. Enter the Subscription ID.

  4. Choose or create a topic from the drop-down menu. The subscription receives messages from the topic.

  5. In the Subscription filter section, enter the filter expression.

  6. Click Create.

To create a push subscription with a filter, follow these steps:

  1. In the Cloud Console, go to the Subscriptions page.

    Go to the Subscriptions page

  2. Click Create subscription.

  3. Enter the Subscription ID.

  4. Choose or create a topic from the drop-down menu. The subscription receives messages from the topic.

  5. In the Delivery type section, click Push.

  6. In the Endpoint URL field, enter the URL of the push endpoint.

  7. In the Subscription filter section, enter the filter expression.

  8. Click Create.

gcloud

To create a pull subscription with a filter, use the gcloud pubsub subscriptions create command with the --message-filter flag:

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --message-filter='FILTER'

Replace the following:

  • SUBSCRIPTION_ID: the ID of the subscription to create
  • TOPIC_ID: the ID of the topic to attach to the subscription
  • FILTER: an expression in the filtering syntax

To create a push subscription with a filter, use the gcloud pubsub subscriptions create command with the --push-endpoint and --message-filter flags:

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=PUSH_ENDPOINT \
  --message-filter='FILTER'

Replace the following:

  • SUBSCRIPTION_ID: the ID of the subscription to create
  • TOPIC_ID: the ID of the topic to attach to the subscription
  • PUSH_ENDPOINT: the URL of the server that the push subscriber runs on
  • FILTER: an expression in the filtering syntax

REST

To create a subscription with a filter, use the projects.subscriptions.create method.

PUT https://pubsub.googleapis.com/v1/projects/PROJECT_ID/subscriptions/SUBSCRIPTION_ID
Authorization: Bearer $(gcloud auth print-access-token)

Replace the following:

  • PROJECT_ID: the project ID for the project to create the subscription in
  • SUBSCRIPTION_ID: the ID of the subscription to create

To create a pull subscription with a filter, specify the filter in the request body:

{
  "topic": TOPIC_ID,
  "filter": FILTER
}

Replace the following:

  • TOPIC_ID: the ID of the topic to attach to the subscription
  • FILTER: an expression in the filtering syntax

To create a push subscription with a filter, specify the push endpoint and the filter in the request body:

{
  "topic": TOPIC_ID,
  "pushConfig": {
    "pushEndpoint": PUSH_ENDPOINT
  },
  "filter": FILTER
}

Replace the following:

  • TOPIC_ID: the ID of the topic to attach to the subscription
  • PUSH_ENDPOINT: the URL of the server that the push subscriber runs on
  • FILTER: an expression in the filtering syntax

The maximum length of a filter is 256 bytes. After creating the subscription, you can't modify the filter. The backlog metrics might include messages that don't match the filter.

Filtering syntax

To filter messages, write an expression that operates on attributes. You can write an expression that matches the key or value of the attributes. The attributes identifier selects the attributes in the message.

For example, the filters in the following table select the domain attribute:

Filter Description
attributes:domain Messages with the domain attribute
NOT attributes:domain Messages without the domain attribute
attributes.domain = "com" Messages with the domain attribute and the value of com
attributes.domain != "com" Messages without the domain attribute and the value of com
hasPrefix(attributes.domain, "co") Messages with the domain attribute and a value that starts with co
NOT hasPrefix(attributes.domain, "co") Messages without the domain attribute and a value that starts with co

Comparison operators

You can filter attributes with the following comparison operators:

  • :
  • =
  • !=

The : operator matches a key in a list of attributes.

attributes:KEY

The equality operators match keys and values. The value must be a string literal.

attributes.KEY = "VALUE"

An expression with an equality operator must begin with a key, and the equality operator must compare a key and a value.

  • Valid: Filter compares a key and a value

    attributes.domain = "com"
    
  • Invalid: The left-hand side of the filter is a value

    "com" = attributes.domain
    
  • Invalid: Filter compares two keys

    attributes.domain = attributes.website
    

The key and value are case-sensitive and must match the attribute exactly. If a key contains characters other than hyphens, underscores, or alphanumeric characters, use a string literal.

attributes."iana.org/language_tag" = "en"

To use backslashes, quotation marks, and non-printing characters in a filter, escape the characters within a string literal. You can also use Unicode, hexadecimal, and octal escape sequences within a string literal.

  • Valid: Filter escapes characters within a string literal

    attributes:"\u307F\u3093\u306A"
    
  • Invalid: Filter escapes characters without a string literal

    attributes:\u307F\u3093\u306A
    

Boolean operators

You can use boolean operators in a filter. For example, the following filter is for messages with the iana.org/language_tag attribute, but without the domain attribute and the com value.

attributes:"iana.org/language_tag" AND NOT attributes.domain = "com"

The NOT operator has the highest precedence. To combine the AND and OR operators, use parentheses and complete expressions.

  • Valid: AND and OR operators with parentheses

    attributes:"iana.org/language_tag" AND (attributes.domain = "net" OR attributes.domain = "org")
    
  • Invalid: AND and OR operators without parentheses

    attributes:"iana.org/language_tag" AND attributes.domain = "net" OR attributes.domain = "org"
    
  • Invalid: AND and OR operators combine incomplete expressions

    attributes.domain = "com" AND ("net" OR "org")
    

You can also use the unary minus operator instead of the NOT operator.

attributes.domain = "com" AND -attributes:"iana.org/language_tag"

Functions

The hasPrefix function filters for attributes with values that start with a substring.

hasPrefix(attributes.KEY, "SUBSTRING")

Replace the following:

  • KEY: the name of the attribute
  • SUBSTRING: a substring of the value