Create an Azure Event Hub feed

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

Supported in:

Google SecOps SIEM

This document guides you through the process of establishing an Azure Event Hub feed to ingest security data into Google Security Operations. You can create a maximum of 10 Azure Event Hub feeds, including both active and inactive feeds. To set up an Azure feed, complete the following processes:

1. Create an event hub in Azure: set up the required infrastructure in your Azure environment to receive and store the security data stream.

2. Configure the feed in Google SecOps: configure the feed in Google SecOps to connect to your Azure event hub and to begin ingesting data.

Create an event hub in Azure

To create an event hub in Azure, do the following:

1. Create an event hub namespace and event hub.

   • Set the partition count to 32 for optimal scaling (this cannot be changed later for standard and basic tiers).

     Note: Partition count does not affect cost.

   • To avoid data loss due to Google SecOps quota limits, use a long retention time for your event hub. This ensures that logs are not deleted before ingestion resumes after a quota throttle. For more information about event retention and retention time limitations, see Event retention.

   • For standard tier event hubs, enable auto inflate to automatically scale throughput as required. See Automatically scale up Azure Event Hubs throughput units for more information.

2. Obtain the event hub connection string required for Google SecOps to ingest data from the Azure event hub. This connection string authorizes Google SecOps to access and collect security data from your event hub. You have two options for providing a connection string:

   • Event hub namespace level: this connection string works for all event hubs within the namespace. It's a simpler option if you're using multiple event hubs and want to use the same connection string for all of them in your feed setup.

   • Event hub level: this connection string is specific to a single event hub. This is a secure option if you need to grant access to only one event hub. Ensure that you remove EntityPath from the end of the connection string.

   For example, change Endpoint=<ENDPOINT>;SharedAccessKeyName=<KEY_NAME>;SharedAccessKey=<KEY>;EntityPath=<EVENT_HUB_NAME> to Endpoint=<ENDPOINT>;SharedAccessKeyName=<KEY_NAME>;SharedAccessKey=<KEY>.

3. Create an Azure blob storage to store your security data and obtain the connection string. This connection string authorizes Google SecOps to access metadata stored in the Azure Blob storage container, which ensures that it accurately fetches data from your event hub.

   Note: Don't use the default $logs container that's automatically created with your storage account. Using the default container for this purpose won't work.

4. Generate an SAS token. Google SecOps needs to track your event hub data flow to scale resources. This is done using an Azure API that requires a SAS token for access.

   Set a long expiry time for your SAS token (for example, 6 months). Ensure that you update it before the expiration to prevent any service disruption.

5. Configure your applications, such as Web Application Firewall or Microsoft Defender, to send their logs to the event hub.

   Microsoft Defender users: When configuring Microsoft Defender streaming, ensure that you enter your existing event hub name. Leaving this field blank can lead to creation of unnecessary event hubs, consuming your limited feed quota. Using event hub names that match the log type is recommended for better organization.

Configure the Azure feed in Google SecOps

To configure the Azure feed in Google SecOps, do the following:

1. From the Google SecOps menu, select SIEM Settings, and then click Feeds.

2. Click Add new.

3. In the Feed name field, enter a name for the feed.

4. In the Source type list, select Microsoft Azure Event Hub.

5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.

6. Click Next. The Add feed window appears.

7. Retrieve the information from the event hub that you created earlier in the Azure portal to fill in the following fields:

   • Event hub name: the event hub name

   • Event hub consumer group: the consumer group associated with your event hub

     Caution: Don't create subscribers or retrieve data programmatically through the Data Explorer tab in the Azure portal for the consumer group. Doing so may result in data loss.

   • Event hub connection string: the event hub connection string

   • Azure storage connection string: the blob storage connection string

   • Azure storage container name: the blob storage container name

   • Azure SAS token: the SAS token

   • Asset namespace: the asset namespace

   • Ingestion labels: the label to be applied to the events from this feed

8. Click Next. The Finalize screen appears.

9. Review your feed configuration, and then click Submit.

Verify the data flow

To verify that your data is flowing into Google SecOps and your event hub is functioning correctly, you can perform these checks:

• In Google SecOps, examine the dashboards and use the Raw Log Scan or Unified Data Model (UDM) search to verify that the ingested data is present in the correct format.

• In the Azure portal, navigate to your event hub's page and inspect the graphs that display incoming and outgoing bytes. Ensure that the incoming and outgoing rates are roughly equivalent, indicating that messages are being processed and there is no backlog.