Stream sharing with Pub/Sub

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Note: To provide feedback or ask questions that are related to this preview feature, contact [e-mail].

You can share Pub/Sub topics through Analytics Hub to curate and distribute a library of streaming data across multiple internal and external organizational boundaries. Real-time streaming data is shared using Analytics Hub exchanges and listings, allowing you to logically categorize and group large sets of Pub/Sub topics and provision access at scale.

You can share streaming data to do the following:

• Financial services:
  • Share rapidly changing instrument prices, quotes, and orders in real-time
  • Detect money laundering and payment fraud
  • Support trading risk computations
• Retail and consumer packaged goods (CPG):
  • Help stores manage inventory in real-time
  • Personalize marketing and customer support
  • Adjust prices dynamically
  • Monitor social media channels
  • Optimize physical store layouts
• Healthcare:
  • Power predictive algorithms to monitor patients and analyze risk in real time
  • Monitor vitals with wearable medical devices
  • Automatic acquisition, structuring, storage, and processing of data from patients, healthcare professionals, and facility administration
• Telecommunications:
  • Review network health and predict failures
  • Discover user behavior patterns to better locate devices and antennas

Roles

The roles for sharing Pub/Sub topics are analogous to sharing BigQuery datasets:

• Admin: a user that manages permissions, visibility, and membership of exchanges and listings. Use the Analytics Hub Admin role.
• Publisher (of a shared topic): a user that creates, manages, and grants access to listings of the shared Pub/Sub topics. Use the Analytics Hub Publisher or Analytics Hub Listing Admin role. The publisher referred here is a user and not the publisher application that creates and sends the messages to the topic.
• Subscriber (of a shared topic): a user that subscribes to the shared topic listing. Subscribers can configure how topic messages will be delivered. Use the Analytics Hub Subscriber role or the Analytics Hub Subscription Owner role. Depending on how the message data is delivered, additional roles may be required (e.g. Pub/Sub service account permissions for writing messages to BigQuery).

Architecture

The following diagram describes how publishers and subscribers to Pub/Sub resources interact with Analytics Hub:

Figure 1. Analytics Hub Publisher and Subscriber workflow with Pub/Sub topics.

Shared topics

A shared topic is the unit of sharing a Pub/Sub topic in Analytics Hub. As a publisher, you create or use an existing Pub/Sub topic to distribute message data to your subscribers. Analytics Hub does not replicate the source Pub/Sub topic.

Listings

A listing is created when a publisher adds a shared topic to an exchange. It contains a reference to the shared topic.

Exchanges

An exchange is a logical grouping of listings that reference shared topics.

Linked Pub/Sub subscriptions

When subscribing to a listing with a shared topic, a linked Pub/Sub subscription is created in the subscriber project. A Pub/Sub subscription is also represented in the Publisher project as a Pub/Sub subscription and an Analytics Hub Listing subscription.

Limitations

Stream sharing with Pub/Sub has the following limitations:

• A shared topic can have a maximum of 10,000 Pub/Sub subscriptions. This limit includes linked Pub/Sub subscriptions and Pub/Sub subscriptions created outside of Analytics Hub, for example, subscriptions created directly from Pub/Sub.
• Shared topics are indexed in Data Catalog, but you cannot filter specifically for its resource type.
• Usage metrics are not captured for linked Pub/Sub subscriptions in Analytics Hub. You can view usage metrics in Pub/Sub.

Enable the Analytics Hub API

To enable the Analytics Hub API, do the following:

Console

Go to the API library and enable the Analytics Hub API for your Google Cloud project.

Enable the Analytics Hub API

gcloud

Run the gcloud services enable command:

gcloud services enable analyticshub.googleapis.com

Enable the Pub/Sub API

Go to the API library and enable the Pub/Sub API for your Google Cloud project.

Enable the Pub/Sub API

Publisher workflows

As an Analytics Hub publisher, you can do the following:

• Add shared topics to an exchange by creating a listing
• Update a listing
• Delete a listing
• Share a listing
• Manage subscriptions to your listing
• Remove subscribers from your listing

Additional publisher permissions

To perform publisher tasks, you must have the Analytics Hub Publisher role (roles/analyticshub.publisher) on an exchange or listing. To view all data exchanges across projects in an organization that you have access to, you must have the resourcemanager.organizations.get permission.

In addition, you need the pubsub.topics.setIamPolicy permission on the Pub/Sub topic you want to list. To share topics with a schema attached, you must grant Analytics Hub subscribers pubsub.schemas.get permission on the schema attached to the topic. This allows Analytics Hub Subscribers to properly parse messages from the Pub/Sub subscription.

Create a listing (to add a shared topic)

To add a shared topic to a listing, select one of the following options:

Console

1. Go to the Analytics Hub page.

   Go to Analytics Hub

   A page appears that lists all data exchanges that you can access.

2. Click the data exchange name in which you want to create the listing.

3. Click Create listing.

4. On the Create listing page, select "Pub/Sub Topic" in the Resource type drop-down.

5. Select an existing Pub/Sub topic in the Shared topic drop-down, or click Create a topic.

6. On the Listing details page, in Display name, enter the name of the listing.

7. Enter the following optional details:

   • Category: select up to two categories that best represent your listing. Subscribers can filter listings based on these categories.

   • Data affinity: region(s) used by the publisher for publishing the data. This information is useful for subscribers to minimize or avoid Pub/Sub network egress costs by reading the data from the same region. For more information about egress costs, see Data transfer costs.

   • Icon: an icon for your listing. PNG and JPEG file formats are supported. Icons must have a file size of less than 512 KiB and dimensions of no more than 512 x 512 pixels.

   • Description: a brief description about your listing. Subscribers can search for listings based on the description.

   • Public Discoverability: enable public discoverability of your listing in the Analytics Hub catalog. You must also set permissions for the listing. Click Actions of the exchange and Set permissions. Give allUsers or allAuthenticatedusers the Analytics Hub Viewer role. Click Allow Public Access.

   • Documentation > Markdown: additional information such as links to any relevant documentation and any additional information that can help subscribers to use your topic.

8. On the Listing contact information page, enter the following optional details:

   • Primary contact: enter an email or a URL of the primary contact for the listing.

   • Request access contact: enter an email or URL of the intake form for subscribers to contact you.

   • Provider: expand the Provider section and specify details in the following fields:

     • Provider name: the name of the topic provider.
     • Provider primary contact: an email or a URL of the topic provider's primary contact.

     Subscribers can filter listings based on the data providers.

   • Publisher: expand the Publisher section and specify details in the following fields:

     • Publisher name: the name of the publisher who's creating the listing.
     • Publisher primary contact: an email or a URL of the topic publisher's primary contact.

9. Review the Listing preview page.

10. Click Publish.

API

Use the projects.locations.dataExchanges.listings.create method.

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings?listingId=LISTING_ID

Replace the following:

• PROJECT_ID: the project ID that contains the data exchange in which you want to create the listing.
• LOCATION: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.
• DATAEXCHANGE_ID: the data exchange ID.
• LISTING_ID: the listing ID.

In the body of the request, provide the listing details. If the request is successful, then the response body contains details of the listing.

For more information about the tasks that you can perform on listings using APIs, see projects.locations.dataExchanges.listings methods.

Update a listing

To update a listing, select one of the following options:

Console

1. Go to the Analytics Hub page.

   Go to Analytics Hub

2. Click the data exchange name that contains the listing.

3. Click the listing that you want to update.

4. Click Edit listing.

5. Modify values in the fields. You can modify all values except the shared topic of the listing.

6. To save changes, click Publish.

API

Use the projects.locations.dataExchanges.listings.patch method.

PATCH https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID?updateMask=UPDATEMASK

Replace the following:

• PROJECT_ID: the project ID that contains the data exchange in which you want to create the listing.
• LOCATION: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.
• DATAEXCHANGE_ID: the data exchange ID.
• LISTING_ID: the listing ID.
• UPDATEMASK: the list of fields that you want to update. For updating multiple values, use a comma-separated list.

In the body of the request, specify the updated values.

For more information about the tasks that you can perform on listings using APIs, see projects.locations.dataExchanges.listings methods.

Delete a listing

Listings with shared Pub/Sub topics cannot be deleted with active subscriptions. Revoke all active subscriptions before attempting to delete a shared topic listing. Once a listing is deleted, you cannot undo it. To delete a listing, select one of the following options:

Console

1. Go to the Analytics Hub page.

   Go to Analytics Hub

2. Click the data exchange name that contains the listing.

3. Click the listing that you want to delete.

4. Click Delete.

5. In the Delete listing? dialog, confirm deletion by typing delete.

6. Click Delete.

API

Use the projects.locations.dataExchanges.listings.delete method.

DELETE https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID

Replace the following:

• PROJECT_ID: the project ID that contains the data exchange in which you want to create the listing.
• LOCATION: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.
• DATAEXCHANGE_ID: the data exchange ID.
• LISTING_ID: the listing ID.

For more information about the tasks that you can perform on listings using APIs, see projects.locations.dataExchanges.listings methods.

Share a listing

If you want to give users access to a private listing, you must set IAM policy for an individual or a group for that listing. For a commercial listing, your data exchange must be public. Listings in a public data exchange appears in Analytics Hub for all Google Cloud users (allAuthenticatedUsers). To enable users to request access for commercial listings, you must grant users the Analytics Hub Viewer role.

To give users access to view or subscribe to your listing, do the following steps:

1. Go to the Analytics Hub page.

   Go to Analytics Hub

2. Click the data exchange name that contains the listing.

3. Click the listing for which you want to add subscribers.

4. Click Set permissions.

5. To add principals, click Add principal.

6. In the New principals field, add the following details based on the type of listing:

   • For a private listing, enter emails of the identity to whom you want to grant access.
   • For a public listing, add allAuthenticatedUsers.

7. For Select a role, hold the pointer over Analytics Hub, and then based on the type of listing, select one of the following roles:

   • For a commercial listing, select the Analytics Hub Viewer role. This role lets users view the listing and request access.
   • For a private or non-commercial public listing, select the Analytics Hub Subscriber role. This role lets users subscribe to your listing.

8. Click Save.

For more information, see Analytics Hub Subscriber and Viewer roles.

Manage subscriptions

To allow a user to subscribe to a listing with a shared topic, grant them the Analytics Hub Subscriber (roles/analyticshub.subscriber) and Analytics Hub Subscription Owner (roles/analyticshub.subscriptionOwner) roles on a specific listing:

1. Go to the Analytics Hub page.

   Go to Analytics Hub

2. Click the data exchange name that contains the listing for which you want to manage the subscriptions.

3. Click the listing for which you want to list all subscribers.

4. Click Set permissions.

5. To add principals, click Add principal.

6. In the New principals field, enter the usernames or emails of the subscribers that you want to add.

7. For Select a role, select Analytics Hub > Analytics Hub Subscriber.

8. Click Add another role.

9. For Select a role, select Analytics Hub > Analytics Hub Subscription Owner.

10. Click Save.

You can delete and update subscribers at any time by clicking Set Permissions.

Revoke a subscription

To remove a subscription to a shared topic listing from Analytics Hub, select one of the following options:

Console

1. To list all subscribers of a listing, follow the Google Cloud console instructions in View all subscriptions.

2. Click the Subscriptions tab to view all the subscriptions to the data exchange.

3. Select the checkbox next to the subscriptions you want to remove, or select all subscriptions.

4. In the Revoke subscription? dialog, enter revoke to confirm.

5. Click Revoke.

API

To remove a subscription, use the projects.locations.subscriptions.revoke method.

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/subscriptions/SUBSCRIPTION_ID:revoke

Replace the following:

• PROJECT_ID: the project ID of the subscription that you want to remove.
• LOCATION: the location of the subscription.
• SUBSCRIPTION_ID: the ID of the subscription that you want to remove.

By revoking a subscription from Analytics Hub, a subscriber no longer receives message data from the shared topic. The Pub/Sub subscription is detached from the shared topic. If a subscription is deleted directly from Pub/Sub, the Analytics Hub subscription will persist and need to be cleaned up.

Subscriber workflows

As an Analytics Hub subscriber, you can view and subscribe to a listing. Subscribing to a listing of a shared topic creates one linked Pub/Sub subscription in the subscriber's project. The Pub/Sub subscription will be visible in the publisher's project.

Additional subscriber permissions

You must have the Analytics Hub Subscriber (roles/analyticshub.subscriber) role granted at the project, exchange, or listing-level to perform subscriber tasks.

Subscribe to a shared topic listing

To subscribe to a listing with a shared topic, select one of the following options:

Console

1. To view a list of listings that you have access to, follow the steps in View listings.

2. Browse through the listings and click a listing that you want to subscribe to. A dialog containing the details of the listing appears.

3. Click Subscribe to open the Create subscription dialog.

4. If you don't have the Analytics Hub API enabled in your project, an error message appears with a link to enable the API. Click Enable Analytics Hub API.

5. In the Create subscription dialog, specify the following details:

   • Subscription ID: specify the name of the subscription you are creating.
   • Delivery type: select the mechanism on how the message data will be delivered.
   • Message retention duration: set the message retention duration.
   • Expiration period: set when the subscription will expire after inactivity, if ever.
   • Acknowledgement deadline: set the acknowledgement deadline time.
   • Subscription filter: set filter syntax on messages.
   • Exactly once delivery: enable exactly once delivery.
   • Message ordering: enable ordering of messages with an order key.
   • Dead lettering: enable dead lettering.
   • Retry policy: set a retry policy.

   For more information about Pub/Sub subscription properties, see subscription properties.

6. To save your changes, click Create. The linked Pub/Sub subscription is created in your project.

API

Use the projects.locations.dataExchanges.listings.subscribe method.

POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID:subscribe

Replace the following:

• PROJECT_ID: the project ID of the listing that you want to subscribe to.
• LOCATION: the location for your listing that you want to subscribe to.
• DATAEXCHANGE_ID: the data exchange ID of the listing that you want to subscribe to.
• LISTING_ID: the listing ID that you want to subscribe to.

In the body of the request, specify the Pub/Sub subscription where you want to create the linked Pub/Sub subscription. If successful, the response body is empty.

For more information about Pub/Sub subscriptions, see subscription overview.

Pricing

There is no additional charge for Pub/Sub topic publishers to list and share topics through Analytics Hub. Publishers are charged for the total number of bytes written (publish throughput) to the shared topic and network egress (if applicable). Subscribers are charged for the total number of bytes read (subscribe throughput) from the linked subscription and network egress (if applicable). See Pub/Sub pricing for additional details.

VPC Service Controls

For shared topics from projects that have VPC Service Controls perimeters, you can set the appropriate ingress and egress rules required to grant access to publishers and subscribers.