Stream sharing with Pub/Sub
You can share Pub/Sub topics through Analytics Hub to curate and distribute a library of streaming data across multiple internal and external organizational boundaries. Real-time streaming data is shared using Analytics Hub exchanges and listings, allowing you to logically categorize and group large sets of Pub/Sub topics and provision access at scale.
You can share streaming data to do the following:
- Financial services:
- Share rapidly changing instrument prices, quotes, and orders in real-time
- Detect money laundering and payment fraud
- Support trading risk computations
- Retail and consumer packaged goods (CPG):
- Help stores manage inventory in real-time
- Personalize marketing and customer support
- Adjust prices dynamically
- Monitor social media channels
- Optimize physical store layouts
- Healthcare:
- Power predictive algorithms to monitor patients and analyze risk in real time
- Monitor vitals with wearable medical devices
- Automatic acquisition, structuring, storage, and processing of data from patients, healthcare professionals, and facility administration
- Telecommunications:
- Review network health and predict failures
- Discover user behavior patterns to better locate devices and antennas
Roles
The roles for sharing Pub/Sub topics are analogous to sharing BigQuery datasets:
- Admin: a user that manages permissions, visibility, and membership of exchanges and listings. Use the Analytics Hub Admin role.
- Publisher (of a shared topic): a user that creates, manages, and grants access to listings of the shared Pub/Sub topics. Use the Analytics Hub Publisher or Analytics Hub Listing Admin role. The publisher referred here is a user and not the publisher application that creates and sends the messages to the topic.
- Subscriber (of a shared topic): a user that subscribes to the shared topic listing. Subscribers can configure how topic messages will be delivered. Use the Analytics Hub Subscriber role or the Analytics Hub Subscription Owner role. Depending on how the message data is delivered, additional roles may be required (e.g. Pub/Sub service account permissions for writing messages to BigQuery).
Architecture
The following diagram describes how publishers and subscribers to Pub/Sub resources interact with Analytics Hub:
Shared topics
A shared topic is the unit of sharing a Pub/Sub topic in Analytics Hub. As a publisher, you create or use an existing Pub/Sub topic to distribute message data to your subscribers. Analytics Hub does not replicate the source Pub/Sub topic.
Listings
A listing is created when a publisher adds a shared topic to an exchange. It contains a reference to the shared topic.
Exchanges
An exchange is a logical grouping of listings that reference shared topics.
Linked Pub/Sub subscriptions
When subscribing to a listing with a shared topic, a linked Pub/Sub subscription is created in the subscriber project. A Pub/Sub subscription is also represented in the Publisher project as a Pub/Sub subscription and an Analytics Hub Listing subscription.
Limitations
Stream sharing with Pub/Sub has the following limitations:
- A shared topic can have a maximum of 10,000 Pub/Sub subscriptions. This limit includes linked Pub/Sub subscriptions and Pub/Sub subscriptions created outside of Analytics Hub, for example, subscriptions created directly from Pub/Sub.
- Shared topics are indexed in Data Catalog, but you cannot filter specifically for its resource type.
- Usage metrics are not captured for linked Pub/Sub subscriptions in Analytics Hub. You can view usage metrics in Pub/Sub.
Enable the Analytics Hub API
To enable the Analytics Hub API, do the following:
Console
Go to the API library and enable the Analytics Hub API for your Google Cloud project.
gcloud
Run the gcloud services enable
command:
gcloud services enable analyticshub.googleapis.com
Enable the Pub/Sub API
Go to the API library and enable the Pub/Sub API for your Google Cloud project.
Publisher workflows
As an Analytics Hub publisher, you can do the following:
- Add shared topics to an exchange by creating a listing
- Update a listing
- Delete a listing
- Share a listing
- Manage subscriptions to your listing
- Remove subscribers from your listing
Additional publisher permissions
To perform publisher tasks, you must have the Analytics Hub Publisher role (roles/analyticshub.publisher
) on an exchange or listing. To view all data exchanges across projects in an organization that you have access to, you must have the resourcemanager.organizations.get permission.
In addition, you need the pubsub.topics.setIamPolicy
permission on the Pub/Sub topic you want to list. To share topics with a schema attached, you must grant Analytics Hub subscribers pubsub.schemas.get
permission on the schema attached to the topic. This allows Analytics Hub Subscribers to properly parse messages from the Pub/Sub subscription.
Create a listing (to add a shared topic)
To add a shared topic to a listing, select one of the following options:
Console
Go to the Analytics Hub page.
A page appears that lists all data exchanges that you can access.
Click the data exchange name in which you want to create the listing.
Click
Create listing.On the Create listing page, select "Pub/Sub Topic" in the Resource type drop-down.
Select an existing Pub/Sub topic in the Shared topic drop-down, or click Create a topic.
On the Listing details page, in Display name, enter the name of the listing.
Enter the following optional details:
Category: select up to two categories that best represent your listing. Subscribers can filter listings based on these categories.
Data affinity: region(s) used by the publisher for publishing the data. This information is useful for subscribers to minimize or avoid Pub/Sub network egress costs by reading the data from the same region. For more information about egress costs, see Data transfer costs.
Icon: an icon for your listing. PNG and JPEG file formats are supported. Icons must have a file size of less than 512 KiB and dimensions of no more than 512 x 512 pixels.
Description: a brief description about your listing. Subscribers can search for listings based on the description.
Public Discoverability: enable public discoverability of your listing in the Analytics Hub catalog. You must also set permissions for the listing. Click Actions of the exchange and Set permissions. Give
allUsers
orallAuthenticatedusers
the Analytics Hub Viewer role. Click Allow Public Access.Documentation > Markdown: additional information such as links to any relevant documentation and any additional information that can help subscribers to use your topic.
On the Listing contact information page, enter the following optional details:
Primary contact: enter an email or a URL of the primary contact for the listing.
Request access contact: enter an email or URL of the intake form for subscribers to contact you.
Provider: expand the Provider section and specify details in the following fields:
- Provider name: the name of the topic provider.
- Provider primary contact: an email or a URL of the topic provider's primary contact.
Subscribers can filter listings based on the data providers.
Publisher: expand the Publisher section and specify details in the following fields:
- Publisher name: the name of the publisher who's creating the listing.
- Publisher primary contact: an email or a URL of the topic publisher's primary contact.
Review the Listing preview page.
Click Publish.
API
Use the projects.locations.dataExchanges.listings.create
method.
POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings?listingId=LISTING_ID
Replace the following:
PROJECT_ID
: the project ID that contains the data exchange in which you want to create the listing.LOCATION
: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.DATAEXCHANGE_ID
: the data exchange ID.LISTING_ID
: the listing ID.
In the body of the request, provide the listing details. If the request is successful, then the response body contains details of the listing.
For more information about the tasks that you can perform on listings using
APIs, see projects.locations.dataExchanges.listings
methods.
Update a listing
To update a listing, select one of the following options:
Console
Go to the Analytics Hub page.
Click the data exchange name that contains the listing.
Click the listing that you want to update.
Click
Edit listing.Modify values in the fields. You can modify all values except the shared topic of the listing.
To save changes, click Publish.
API
Use the projects.locations.dataExchanges.listings.patch
method.
PATCH https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID?updateMask=UPDATEMASK
Replace the following:
PROJECT_ID
: the project ID that contains the data exchange in which you want to create the listing.LOCATION
: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.DATAEXCHANGE_ID
: the data exchange ID.LISTING_ID
: the listing ID.UPDATEMASK
: the list of fields that you want to update. For updating multiple values, use a comma-separated list.
In the body of the request, specify the updated values.
For more information about the tasks that you can perform on listings using
APIs, see projects.locations.dataExchanges.listings
methods.
Delete a listing
Listings with shared Pub/Sub topics cannot be deleted with active subscriptions. Revoke all active subscriptions before attempting to delete a shared topic listing. Once a listing is deleted, you cannot undo it. To delete a listing, select one of the following options:
Console
Go to the Analytics Hub page.
Click the data exchange name that contains the listing.
Click the listing that you want to delete.
Click
Delete.In the Delete listing? dialog, confirm deletion by typing delete.
Click Delete.
API
Use the projects.locations.dataExchanges.listings.delete
method.
DELETE https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID
Replace the following:
PROJECT_ID
: the project ID that contains the data exchange in which you want to create the listing.LOCATION
: the location for your data exchange. For more information about locations that support Analytics Hub, see Supported regions.DATAEXCHANGE_ID
: the data exchange ID.LISTING_ID
: the listing ID.
For more information about the tasks that you can perform on listings using
APIs, see projects.locations.dataExchanges.listings
methods.
Share a listing
If you want to give users access to a private listing, you must set IAM policy for an individual or a group for that listing. For a commercial listing, your data exchange must be public. Listings in a public data exchange appears in Analytics Hub for all Google Cloud users (allAuthenticatedUsers). To enable users to request access for commercial listings, you must grant users the Analytics Hub Viewer role.
To give users access to view or subscribe to your listing, do the following steps:
Go to the Analytics Hub page.
Click the data exchange name that contains the listing.
Click the listing for which you want to add subscribers.
Click
Set permissions.To add principals, click
Add principal.In the New principals field, add the following details based on the type of listing:
- For a private listing, enter emails of the identity to whom you want to grant access.
- For a public listing, add
allAuthenticatedUsers
.
For Select a role, hold the pointer over Analytics Hub, and then based on the type of listing, select one of the following roles:
- For a commercial listing, select the Analytics Hub Viewer role. This role lets users view the listing and request access.
- For a private or non-commercial public listing, select the Analytics Hub Subscriber role. This role lets users subscribe to your listing.
Click Save.
For more information, see Analytics Hub Subscriber and Viewer roles.
Manage subscriptions
To allow a user to subscribe to a listing with a shared topic, grant them the Analytics Hub Subscriber (roles/analyticshub.subscriber
) and Analytics Hub Subscription Owner (roles/analyticshub.subscriptionOwner
) roles on a specific listing:
Go to the Analytics Hub page.
Click the data exchange name that contains the listing for which you want to manage the subscriptions.
Click the listing for which you want to list all subscribers.
Click
Set permissions.To add principals, click
Add principal.In the New principals field, enter the usernames or emails of the subscribers that you want to add.
For Select a role, select Analytics Hub > Analytics Hub Subscriber.
Click
Add another role.For Select a role, select Analytics Hub > Analytics Hub Subscription Owner.
Click Save.
You can delete and update subscribers at any time by clicking Set Permissions.
Revoke a subscription
To remove a subscription to a shared topic listing from Analytics Hub, select one of the following options:
Console
To list all subscribers of a listing, follow the Google Cloud console instructions in View all subscriptions.
Click the Subscriptions tab to view all the subscriptions to the data exchange.
Select the checkbox next to the subscriptions you want to remove, or select all subscriptions.
In the Revoke subscription? dialog, enter revoke to confirm.
Click Revoke.
API
To remove a subscription, use the projects.locations.subscriptions.revoke
method.
POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/subscriptions/SUBSCRIPTION_ID:revoke
Replace the following:
PROJECT_ID
: the project ID of the subscription that you want to remove.LOCATION
: the location of the subscription.SUBSCRIPTION_ID
: the ID of the subscription that you want to remove.
By revoking a subscription from Analytics Hub, a subscriber no longer receives message data from the shared topic. The Pub/Sub subscription is detached from the shared topic. If a subscription is deleted directly from Pub/Sub, the Analytics Hub subscription will persist and need to be cleaned up.
Subscriber workflows
As an Analytics Hub subscriber, you can view and subscribe to a listing. Subscribing to a listing of a shared topic creates one linked Pub/Sub subscription in the subscriber's project. The Pub/Sub subscription will be visible in the publisher's project.
Additional subscriber permissions
You must have the Analytics Hub Subscriber (roles/analyticshub.subscriber
) role granted at the project, exchange, or listing-level to perform subscriber tasks.
Subscribe to a shared topic listing
To subscribe to a listing with a shared topic, select one of the following options:
Console
To view a list of listings that you have access to, follow the steps in View listings.
Browse through the listings and click a listing that you want to subscribe to. A dialog containing the details of the listing appears.
Click Subscribe to open the Create subscription dialog.
If you don't have the Analytics Hub API enabled in your project, an error message appears with a link to enable the API. Click Enable Analytics Hub API.
In the Create subscription dialog, specify the following details:
- Subscription ID: specify the name of the subscription you are creating.
- Delivery type: select the mechanism on how the message data will be delivered.
- Message retention duration: set the message retention duration.
- Expiration period: set when the subscription will expire after inactivity, if ever.
- Acknowledgement deadline: set the acknowledgement deadline time.
- Subscription filter: set filter syntax on messages.
- Exactly once delivery: enable exactly once delivery.
- Message ordering: enable ordering of messages with an order key.
- Dead lettering: enable dead lettering.
- Retry policy: set a retry policy.
For more information about Pub/Sub subscription properties, see subscription properties.
To save your changes, click Create. The linked Pub/Sub subscription is created in your project.
API
Use the projects.locations.dataExchanges.listings.subscribe
method.
POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings/LISTING_ID:subscribe
Replace the following:
PROJECT_ID
: the project ID of the listing that you want to subscribe to.LOCATION
: the location for your listing that you want to subscribe to.DATAEXCHANGE_ID
: the data exchange ID of the listing that you want to subscribe to.LISTING_ID
: the listing ID that you want to subscribe to.
In the body of the request, specify the Pub/Sub subscription where you want to create the linked Pub/Sub subscription. If successful, the response body is empty.
For more information about Pub/Sub subscriptions, see subscription overview.
Pricing
There is no additional charge for Pub/Sub topic publishers to list and share topics through Analytics Hub. Publishers are charged for the total number of bytes written (publish throughput) to the shared topic and network egress (if applicable). Subscribers are charged for the total number of bytes read (subscribe throughput) from the linked subscription and network egress (if applicable). See Pub/Sub pricing for additional details.
VPC Service Controls
For shared topics from projects that have VPC Service Controls perimeters, you can set the appropriate ingress and egress rules required to grant access to publishers and subscribers.