An Azure Event Hubs import topic lets you continuously ingest data from Azure Event Hubs as an external source and into Pub/Sub. Then you can stream the data into any of the destinations that Pub/Sub supports.
This document shows you how to create and manage Azure Event Hubs import topics. To create a standard topic, see Create a standard topic.
For more information about import topics, see About import topics.
Before you begin
Know more about the Pub/Sub publish process.
Configure the required roles and permissions to manage Azure Event Hubs import topics including the following:
Set up workload identity federation so that Google Cloud can access the external streaming service.
Required roles and permissions
To get the permissions that
you need to create and manage Azure Event Hubs import topics,
ask your administrator to grant you the
Pub/Sub Editor (roles/pubsub.editor
) IAM role on your topic or project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to create and manage Azure Event Hubs import topics. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to create and manage Azure Event Hubs import topics:
-
Create an import topic:
pubsub.topics.create
-
Delete an import topic:
pubsub.topics.delete
-
Get an import topic:
pubsub.topics.get
-
List an import topic:
pubsub.topics.list
-
Publish to an import topic:
pubsub.topics.publish
-
Update an import topic:
pubsub.topics.update
-
Get the IAM policy for an import topic:
pubsub.topics.getIamPolicy
-
Configure the IAM policy for an import topic:
pubsub.topics.setIamPolicy
You might also be able to get these permissions with custom roles or other predefined roles.
You can configure access control at the project level and the individual resource level.
Set up federated identity to access Azure Event Hubs
Workload Identity Federation lets Google Cloud services access workloads running outside of Google Cloud. With identity federation, you don't need to maintain or pass credentials to Google Cloud to access your resources in other clouds. Instead, you can use the identities of the workloads themselves to authenticate to Google Cloud and access resources.
Create a service account in Google Cloud
This is an optional step. If you already have a service account, you can use it in this procedure instead of creating a new service account. If you are using an existing service account, go to Record the service account unique ID for the next step.
For Azure Event Hubs import topics, Pub/Sub uses the service account as the identity to access resources from Azure.
For more information about creating a service account, including prerequisites, required roles and permissions, and naming guidelines, see Create service accounts. After you create a service account, you might need to wait for 60 seconds or more before you use the service account. This behavior occurs because read operations are eventually consistent; it can take time for the new service account to become visible.
Record the service account unique ID
You need a service account unique ID to set up App registration in Azure.
In the Google Cloud console, go to the Service account details page.
Click the service account that you just created or the one that you are planning to use.
From the Service account details page, record the Unique ID number.
You need the ID as part of the workflow to set up App registration in Azure.
Add the service account token creator role to the Pub/Sub service account
The Service account token creator role (roles/iam.serviceAccountTokenCreator
)
lets principals create short-lived credentials
for a service account. These tokens or credentials are used to impersonate
the service account.
For more information about service account impersonation, see Service account impersonation.
You can also add the Pub/Sub publisher role (roles/pubsub.publisher
)
during this procedure. For more information about the role and why you are adding it,
see Add the Pub/Sub publisher role to the Pub/Sub service account.
In the Google Cloud console, go to the IAM page.
Click the Include Google-provided role grants checkbox.
Look for the service account that has the format
service-{PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com
.For this service account, click the Edit Principal button.
If required, click Add another role.
Search and click the Service account token creator role (
roles/iam.serviceAccountTokenCreator
).Click Save.
Create an App registration with identity federation
To ingest data from Azure Event Hubs, register an application with the Microsoft Identity platform and enable identity federation for your service account. This allows your service account to authenticate to Azure.
For more information on registering an application with the Microsoft Identity platform, see the Azure app registration quickstart.
Sign in to the Azure portal and open the Microsoft Entra ID page.
In the navigation pane, click Manage > App registrations.
Click New registration.
Name the application and set Supported account types to anything other than Personal Microsoft accounts only.
To register the application, click Register.
Open the application and click Manage > Certificates & Secrets.
Click Add credential.
To configure a federated credential, click Other issuer.
For the Issuer field, enter
https://accounts.google.com
.For the Value field, enter the unique ID of your Pub/Sub service account.
You can find this ID in Record the service account unique ID.
Name the federated identity and then click Add.
Grant role assignments to the registered App
To enable Pub/Sub to read from your Azure event hub, grant the necessary roles to your registered app.
Sign in to the Azure portal and open your Event Hubs namespace.
To open a namespace, enter Event Hubs in the search, and then click one of the namespaces.
In the sidebar, click Access control (IAM).
Click Add > Add role assignment.
Select the Azure Event Hubs Data Receiver role, then click Next.
This grants the Pub/Sub service account read access to your Azure event hub.
Click +Select members, then search for your registered app.
Type the registered app name into the search field.
Click the name of your app and then click Select.
Click Review + Assign.
Repeat steps 3–6 and add the Azure Event Hubs Data Sender role.
This role grants the Google Cloud Service Account permission to fetch metadata from your Azure event hub.
For more information on assigning Azure roles, visit the Assign Azure roles page.
Add the Pub/Sub publisher role to the Pub/Sub principal
To enable publishing, you must assign a publisher role to the Pub/Sub service account so that Pub/Sub is able to publish to the Azure Event Hubs import topic.
Enable publishing from all topics
Use this method if you have not created any Azure Event Hubs import topics.
In the Google Cloud console, go to the IAM page.
Click the Include Google-provided role grants checkbox.
Look for the service account that has the format
service-{PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com
.For this service account, click the Edit Principal button.
If required, click Add another role.
Search and click the Pub/Sub publisher role (
roles/pubsub.publisher
).Click Save.
Enable publishing from a single topic
Use this method only if the Azure Event Hubs import topic already exists.
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Run the
gcloud pubsub topics add-iam-policy-binding
command:gcloud pubsub topics add-iam-policy-binding TOPIC_ID \ --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com" \ --role="roles/pubsub.publisher"
Replace the following:
TOPIC_ID
: the topic ID of the Azure Event Hubs import topic.PROJECT_NUMBER
: the project number. To view the project number, see Identifying projects.
Add the service account user role to the service account
The Service Account User role (roles/iam.serviceAccountUser
) includes the
permission iam.serviceAccounts.actAs
that lets a principal attach a service
account to the Azure Event Hubs import topic's ingestion settings and
use that service account for federated identity.
In the Google Cloud console, go to the IAM page.
For the principal that's issuing the create or update topic calls, click the Edit Principal button.
If required, click Add another role.
Search and click the Service account user role (
roles/iam.serviceAccountUser
).Click Save.
Use Azure Event Hubs import topics
You can create a new import topic or edit an existing topic.
Considerations
Creating the topic and subscription separately, even if done in rapid succession, can lead to data loss. There's a short window where the topic exists without a subscription. If any data is sent to the topic during this time, it is lost. By creating the topic first, creating the subscription, and then converting the topic to an import topic, you guarantee that no messages are missed during the import process.
If you need to re-create the Azure event hub of an existing import topic with the same name, you can't just delete the Azure event hub and re-create it. This action can invalidate Pub/Sub's offset management, which can lead to data loss. To mitigate this, follow these steps:
- Delete the Pub/Sub import topic
- Delete the Azure event hub
- Create the Azure event hub
- Create the Pub/Sub import topic
Data from an Azure event hub is always read from the earliest offset.
Create Azure Event Hubs import topics
To know more about properties associated with a topic, see Properties of a topic.
Ensure that you have completed the following procedures:
To create an Azure Event Hubs import topic, follow these steps:
Console
In the Google Cloud console, go to the Topics page.
Click Create topic.
In the Topic ID field, enter an ID for your Azure Event Hubs import topic. For more information about naming topics, see the naming guidelines.
Select Add a default subscription.
Select Enable ingestion.
For ingestion source, select Azure Event Hubs.
Enter the following details:
Resource group: the name of the Azure Resource Group that contains the Azure Event Hubs namespace that you are planning to ingest into Pub/Sub.
Event Hub namespace: the name of the Azure Event Hubs namespace that contains the Azure event hub that you are planning to ingest into Pub/Sub.
Event Hub: the name of the Azure event hub that you are planning to ingest into Pub/Sub.
Client ID: the application (client) ID of the registered App in Azure that contains the federated identity of the Google Cloud Service Account. This value should be a UUID. For more information, visit the Azure App Registration page.
Tenant ID: the directory (tenant) ID of the tenant that owns the Azure event hub that you are planning to ingest into Pub/Sub. This value should be a UUID. To find your tenant ID, visit the Azure Get subscription and tenant IDs page.
Subscription ID: the subscription ID of the subscription that owns the Azure Event Hubs resource group. This value should be a UUID. To find your subscription ID, visit the Azure Get subscription and tenant IDs page.
Service account: the service account that you created in Create a service account in Google Cloud.
Leave the remaining fields with the default values.
Click Create topic.
gcloud
-
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Run the
gcloud pubsub topics create
command:gcloud pubsub topics create TOPIC_ID \ --azure-event-hubs-ingestion-resource-group RESOURCE_GROUP \ --azure-event-hubs-ingestion-namespace NAMESPACE \ --azure-event-hubs-ingestion-event-hub EVENT_HUB \ --azure-event-hubs-ingestion-client-id CLIENT_ID \ --azure-event-hubs-ingestion-tenant-id TENANT_ID \ --azure-event-hubs-ingestion-subscription-id SUBSCRIPTION_ID \ --azure-event-hubs-ingestion-service-account SERVICE_ACCOUNT
Replace the following:
TOPIC_ID
: the name or ID of your Pub/Sub topic.RESOURCE_GROUP
: the name of the Azure Event Hubs resource group that contains the Azure Event Hubs namespace that you are planning to ingest into Pub/Sub.NAMESPACE
: the name of the Azure Event Hubs namespace that contains the Azure event hub that you are planning to ingest into Pub/Sub.EVENT_HUB
: the name of the Azure event hub that you are planning to ingest into Pub/Sub.CLIENT_ID
: the application (client) ID of the registered App in Azure that contains the federated identity of the Google Cloud Service Account. This value should be a UUID. For more information, visit the Azure App Registration page.TENANT_ID
: the directory (tenant) ID of the tenant that owns the Azure event hub that you are planning to ingest into Pub/Sub. This value should be a UUID. To find your tenant ID, visit theAzure event h Azure Get subscription and tenant IDs page.SUBSCRIPTION_ID
: the subscription ID of the subscription that owns the Azure event hub that you are planning to ingest into Pub/Sub. This value should be a UUID. To find your subscription ID, visit the Azure Get subscription and tenant IDs page.SERVICE_ACCOUNT
: the service account that you created in Create a service account in Google Cloud
Go
Before trying this sample, follow the Go setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Go API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
Before trying this sample, follow the Java setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Java API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
Before trying this sample, follow the Node.js setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Node.js API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
Before trying this sample, follow the Python setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Python API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
C++
Before trying this sample, follow the C++ setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub C++ API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js (TypeScript)
Before trying this sample, follow the Node.js setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Node.js API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
If you run into issues, see Troubleshooting an Azure Event Hubs import topic.
Edit Azure Event Hubs import topics
To edit the ingestion data source settings of an Azure Event Hubs import topic, follow these steps:
Console
In the Google Cloud console, go to the Topics page.
Click the Azure Event Hubs import topic.
In the topic details page, click Edit.
Update the fields that you want to change.
Click Update.
gcloud
-
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Run the
gcloud pubsub topics update
command with all the flags mentioned in the following sample:gcloud pubsub topics update TOPIC_ID \ --azure-event-hubs-ingestion-resource-group RESOURCE_GROUP \ --azure-event-hubs-ingestion-namespace NAMESPACE \ --azure-event-hubs-ingestion-event-hub EVENT_HUB \ --azure-event-hubs-ingestion-client-id CLIENT_ID \ --azure-event-hubs-ingestion-tenant-id TENANT_ID \ --azure-event-hubs-ingestion-subscription-id SUBSCRIPTION_ID \ --azure-event-hubs-ingestion-service-account SERVICE_ACCOUNT
Replace the following:
TOPIC_ID
: the name or ID of your Pub/Sub topic.RESOURCE_GROUP
: the name of the Azure Event Hubs resource group that contains the Azure Event Hubs namespace that you are ingesting into Pub/Sub.NAMESPACE
: the name of the Azure Event Hubs namespace that contains the Azure event hub that you are ingesting into Pub/Sub.EVENT_HUB
: the name of the Azure event hub that you are ingesting into Pub/Sub.CLIENT_ID
: the application (client) ID of the registered App in Azure that contains the federated identity of the Google Cloud Service Account. This value should be a UUID. For more information, see the Azure App Registration page.TENANT_ID
: the directory (tenant) ID of the tenant that owns the Azure Event Hubs that you are ingesting into Pub/Sub. This value should be a UUID. To find your tenant ID, see visit the Azure Get subscription and tenant IDs page.SUBSCRIPTION_ID
: the subscription ID of the subscription that owns the Azure event hub that you are ingesting into Pub/Sub. This value should be a UUID. To find your subscription ID, see Azure Get subscription and tenant IDs.SERVICE_ACCOUNT
: the service account that you created in Create a service account in Google Cloud
Quotas and limits
The publisher throughput for import topics is bound by the publish quota of the topic. For more information, see Pub/Sub quotas and limits.
What's next
Choose the type of subscription for your topic.
Learn how to publish a message to a topic.
Create or modify a topic with gcloud CLI, REST APIs, or Client libraries.