Collect Azure DevOps audit logs
Supported in:
Overview
This parser handles Azure DevOps audit logs in JSON format. It extracts fields from nested and top-level JSON structures, mapping them to the UDM. Conditional logic based on specific field values categorizes events and enriches the output with relevant security information. The parser also handles non-JSON formatted messages by attempting to extract a JSON payload using grok patterns.
Before you begin
- Ensure that you have a Google SecOps instance.
- Ensure that you have an active Azure DevOps Organization.
- Ensure that you have privileged access to Azure Devops Organization and Azure.
Configure a feed in Google SecOps to ingest the Azure Devops logs
- Go to SIEM Settings > Feeds.
- Click Add new.
- In the Feed name field, enter a name for the feed (for example, Azure Devops Logs).
- Select Webhook as the Source type.
- Select Azure Devops as the Log type.
- Click Next.
- Optional: specify values for the following input parameters:
- Split delimiter: the delimiter that is used to separate log lines, such as
\n. - Asset namespace: the asset namespace.
- Ingestion labels: the label applied to the events from this feed.
- Split delimiter: the delimiter that is used to separate log lines, such as
- Click Next.
- Review the feed configuration in the Finalize screen, and then click Submit.
- Click Generate Secret Key to generate a secret key to authenticate this feed.
- Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
- On the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
- Click Done.
Create an API key for the webhook feed
Go to Google Cloud console > Credentials.
Click Create credentials, and then select API key.
Restrict the API key access to the Google Security Operations API.
Specify the endpoint URL
- In your client application, specify the HTTPS endpoint URL provided in the webhook feed.
Enable authentication by specifying the API key and secret key as part of the custom header in the following format:
X-goog-api-key = API_KEY X-Webhook-Access-Key = SECRETRecommendation: Specify the API key as a header instead of specifying it in the URL. If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:
ENDPOINT_URL?key=API_KEY&secret=SECRET
Replace the following:
ENDPOINT_URL: the feed endpoint URL.API_KEY: the API key to authenticate to Google Security Operations.SECRET: the secret key that you generated to authenticate the feed.
Configure Auditing feature in Azure Devops
- Sign in to your organization (
https://dev.azure.com/{yourorganization}). - Select the gear icon for Organization settings.
- Select Policies under Security.
- Toggle the Log Audit Events button to ON.
Configure an Event Grid Topic in Azure
- Sign in to the Azure Portal.
- Search for and access Event Grid.
- Locate Topics under Custom events.
- Click + Create.
- Select your Subscription and Resource Group. Provide a name (For example, DevopsAuditLog) and select the region. Click Review and create.
- Access the new Topic and copy the Topic Endpoint URL.
- Go to Settings > Access Keys and copy Key 1.
Configure Azure Devops Log Stream to Event Grid
- Sign in to your organization (
https://dev.azure.com/{yourorganization}). - Select the gear icon for Organization settings.
- Select Auditing.
- Go to Streams tab and select New stream > Event Grid.
- Enter the topic endpoint and access key created in Configure an Event Grid Topic in Azure.
Configure a Webhook in Azure DevOps for Google SecOps
- In the Azure Portal, search for and access Event Grid.
- Select previously created Topic.
- Go to Entities > Event Subscription.
- Click + Event Subscription.
- Provide a descriptive name (e.g Google SecOps Integration).
- Select Web Hook and click Configure an endpoint.
- Configure the endpoint:
- Subscriber endpoint: Enter the Google SecOps API endpoint URL.
- Append
?key=<API_KEY>&secret=<SECRET_KEY>to the Payload URL. - Set the Content-Type header to application/json.
- Click Create.
UDM Mapping Table
| Log Field | UDM Mapping | Logic |
|---|---|---|
ActivityId |
metadata.product_log_id |
Directly mapped from the Id field in the raw log when the records field is not present, or from the ActivityId field within the data object when records is present. |
ActionId |
metadata.product_event_type |
Directly mapped from the ActionId field within the data object. |
ActorCUID |
additional.fields |
Included as an additional field with key "Actor CUID". |
ActorDisplayName |
principal.user.user_display_name |
Directly mapped from the ActorDisplayName field if it's not "Azure DevOps Service". If it is "Azure DevOps Service", it's added as a label to principal.resource.attribute.labels. |
ActorUPN |
principal.user.email_addresses |
Directly mapped from the ActorUPN field if it matches an email address pattern. |
ActorUserId |
principal.user.userid |
Directly mapped from the ActorUserId field. |
Area |
target.application |
Used to construct the target.application field by prepending "DevOps " to the Area value. |
AuthenticationMechanism |
extensions.auth.auth_details, security_result.rule_id |
Parsed to extract authentication details and rule ID. The authentication details are mapped to extensions.auth.auth_details. The extracted rule ID is mapped to security_result.rule_id. |
CategoryDisplayName |
security_result.action_details |
Directly mapped to security_result.action_details. |
City |
principal.location.city |
Directly mapped from the City field. |
Conditions |
additional.fields |
Added as an additional field with key "Conditions". |
Country |
principal.location.country_or_region |
Directly mapped from the Country field. |
Data.* |
Various | Fields within the Data object are mapped to different UDM fields based on their names and context. See below for specific examples. |
Data.AccessLevel |
target.resource.attribute.labels |
Added as a label with key "AccessLevel". |
Data.AgentId |
target.resource.product_object_id |
Mapped to target.resource.product_object_id if PipelineId and AuthorizationId are not present. |
Data.AgentName |
target.resource.name |
Mapped to target.resource.name if PipelineName, NamespaceName, and DisplayName are not present. |
Data.AuthorizationId |
target.resource.product_object_id |
Mapped to target.resource.product_object_id if PipelineId is not present. |
Data.CallerProcedure |
additional.fields |
Added as an additional field with key "CallerProcedure". |
Data.CheckSuiteId |
additional.fields |
Added as an additional field with key "CheckSuiteId". |
Data.CheckSuiteStatus |
additional.fields |
Added as an additional field with key "CheckSuiteStatus". |
Data.ConnectionId |
additional.fields |
Added as an additional field with key "ConnectionId". |
Data.ConnectionName |
additional.fields |
Added as an additional field with key "ConnectionName". |
Data.ConnectionType |
additional.fields |
Added as an additional field with key "ConnectionType". |
Data.DefinitionId |
additional.fields |
Added as an additional field with key "DefinitionId". |
Data.DeploymentResult |
additional.fields |
Added as an additional field with key "DeploymentResult". |
Data.DisplayName |
target.resource.name |
Mapped to target.resource.name if PipelineName and NamespaceName are not present. |
Data.EndpointIdList |
additional.fields |
Added as an additional field with key "EndpointIdList". |
Data.EnvironmentName |
additional.fields |
Added as an additional field with key "EnvironmentName". |
Data.Filter.continuationToken |
target.resource.attribute.labels |
Added as a label with key "continuation_token". |
Data.Filter.endTime |
target.resource.attribute.labels |
Added as a label with key "filter_end_time". |
Data.Filter.startTime |
target.resource.attribute.labels |
Added as a label with key "filter_start_time". |
Data.FinishTime |
additional.fields |
Added as an additional field with key "FinishTime". |
Data.GroupId |
target.group.product_object_id |
Directly mapped to target.group.product_object_id when Data.Updates.0.GroupId is not present. |
Data.GroupName |
target.group.group_display_name |
Directly mapped to target.group.group_display_name. |
Data.JobName |
additional.fields |
Added as an additional field with key "JobName". |
Data.MemberId |
target.user.userid |
Directly mapped to target.user.userid when Data.Updates.0.MemberId is not present. |
Data.MemberDisplayName |
target.user.user_display_name |
Directly mapped to target.user.user_display_name. |
Data.NamespaceId |
target.resource.product_object_id |
Mapped to target.resource.product_object_id if PipelineId, AuthorizationId, and AgentId are not present. |
Data.NamespaceName |
target.resource.name |
Mapped to target.resource.name if PipelineName is not present. |
Data.ownerDetails |
additional.fields |
Added as an additional field with key "OwnerDetails". |
Data.OwnerId |
additional.fields |
Added as an additional field with key "OwnerId". |
Data.PipelineId |
target.resource.product_object_id |
Directly mapped to target.resource.product_object_id. |
Data.PipelineName |
target.resource.name |
Directly mapped to target.resource.name. |
Data.PipelineRevision |
target.resource.attribute.labels |
Added as a label with key "PipelineRevision". |
Data.PipelineScope |
target.resource.attribute.labels |
Added as a label with key "PipelineScope". |
Data.PlanType |
additional.fields |
Added as an additional field with key "PlanType". |
Data.PreviousAccessLevel |
target.resource.attribute.labels |
Added as a label with key "PreviousAccessLevel". |
Data.PublisherName |
target.resource.attribute.labels |
Added as a label with key "PublisherName". |
Data.Reason |
additional.fields |
Added as an additional field with key "Reason". |
Data.ReleaseId |
additional.fields |
Added as an additional field with key "ReleaseId". |
Data.ReleaseName |
additional.fields |
Added as an additional field with key "ReleaseName". |
Data.RequesterId |
additional.fields |
Added as an additional field with key "RequesterId". |
Data.RetentionLeaseId |
additional.fields |
Added as an additional field with key "RetentionLeaseId". |
Data.RetentionOwnerId |
additional.fields |
Added as an additional field with key "RetentionOwnerId". |
Data.RunName |
additional.fields |
Added as an additional field with key "RunName". |
Data.Scopes |
target.resource.attribute.labels |
Added as labels with key "Scope". |
Data.StageName |
additional.fields |
Added as an additional field with key "StageName". |
Data.StartTime |
additional.fields |
Added as an additional field with key "StartTime". |
Data.TargetUser |
target.user.userid |
Directly mapped to target.user.userid. |
Data.Timestamp |
metadata.event_timestamp |
Parsed and mapped to metadata.event_timestamp. |
Data.TokenType |
target.resource.attribute.labels |
Added as a label with key "TokenType". |
Data.Updates.0.GroupId |
target.group.product_object_id |
Directly mapped to target.group.product_object_id. |
Data.Updates.0.MemberId |
target.user.userid |
Directly mapped to target.user.userid. |
Data.ValidFrom |
target.resource.attribute.labels |
Added as a label with key "ValidFrom". |
Data.ValidTo |
target.resource.attribute.labels |
Added as a label with key "ValidTo". |
DewPoint |
additional.fields |
Added as an additional field with key "DewPoint". |
Details |
metadata.description |
Directly mapped to metadata.description. |
Humidity |
additional.fields |
Added as an additional field with key "Humidity". |
Icon |
additional.fields |
Added as an additional field with key "Icon". |
Id |
metadata.product_log_id |
Directly mapped to metadata.product_log_id. |
IpAddress |
principal.ip |
Directly mapped to principal.ip. |
MoonPhase |
additional.fields |
Added as an additional field with key "MoonPhase". |
Moonrise |
additional.fields |
Added as an additional field with key "Moonrise". |
Moonset |
additional.fields |
Added as an additional field with key "Moonset". |
OperationName |
metadata.product_event_type |
Directly mapped to metadata.product_event_type. |
Precipitation |
additional.fields |
Added as an additional field with key "Precipitation". |
Pressure |
additional.fields |
Added as an additional field with key "Pressure". |
ProjectId |
target.resource_ancestors.product_object_id |
Used to populate the product_object_id field within target.resource_ancestors when the ancestor is of type CLOUD_PROJECT. |
ProjectName |
target.resource_ancestors.name, target.resource.attribute.labels |
Used to populate the name field within target.resource_ancestors when the ancestor is of type CLOUD_PROJECT. Also added as a label to target.resource.attribute.labels with key "ProjectName". |
RoleLocation |
target.location.name |
Directly mapped to target.location.name. |
ScopeDisplayName |
target.resource_ancestors.name |
Used to populate the name field within target.resource_ancestors when the ancestor is of type CLOUD_ORGANIZATION. |
ScopeId |
target.resource_ancestors.product_object_id |
Used to populate the product_object_id field within target.resource_ancestors when the ancestor is of type CLOUD_ORGANIZATION. |
ScopeType |
additional.fields |
Added as an additional field with key "ScopeType". |
Sunrise |
additional.fields |
Added as an additional field with key "Sunrise". |
Sunset |
additional.fields |
Added as an additional field with key "Sunset". |
Temperature |
additional.fields |
Added as an additional field with key "Temperature". |
TenantId |
metadata.product_deployment_id, additional.fields |
Directly mapped to metadata.product_deployment_id. Also added as an additional field with key "TenantId". |
TimeGenerated |
metadata.event_timestamp |
Parsed and mapped to metadata.event_timestamp. |
UserAgent |
network.http.user_agent, network.http.parsed_user_agent |
Directly mapped to network.http.user_agent. Also parsed and mapped to network.http.parsed_user_agent. |
UVIndex |
additional.fields |
Added as an additional field with key "UVIndex". |
Visibility |
additional.fields |
Added as an additional field with key "Visibility". |
WindDirection |
additional.fields |
Added as an additional field with key "WindDirection". |
WindSpeed |
additional.fields |
Added as an additional field with key "WindSpeed". |
_Internal_WorkspaceResourceId |
additional.fields |
Added as an additional field with key "workspace_resource_id". |
| N/A | metadata.event_type |
Determined by logic based on the OperationName and other fields. Defaults to "GENERIC_EVENT" if no specific event type is matched. Possible values include "STATUS_SHUTDOWN", "RESOURCE_CREATION", "STATUS_UPDATE", "USER_RESOURCE_DELETION", "RESOURCE_READ", "RESOURCE_WRITTEN", "RESOURCE_DELETION", and "GROUP_MODIFICATION". |
| N/A | metadata.vendor_name |
Set to "Microsoft". |
| N/A | metadata.product_name |
Set to "Azure DevOps". |
| N/A | metadata.log_type |
Set to "AZURE_DEVOPS". |
| N/A | principal.user.account_type |
Set to "SERVICE_ACCOUNT_TYPE" if AuthenticationMechanism contains "ServicePrincipal", otherwise set to "CLOUD_ACCOUNT_TYPE". |
| N/A | target.asset.attribute.cloud.environment |
Set to MICROSOFT_AZURE. |
| N/A | security_result.action |
Set to "ALLOW" for successful operations (Succeeded, Created, Modified, executed, updated, removed) and "BLOCK" for failed operations (Failed, TimedOut). |
| N/A | extensions.auth.mechanism |
Set to "USERNAME_PASSWORD" if summary is "UserAuthToken". |
| N/A | target.resource.resource_type |
Set to "SETTING" if pipeline_id is present, "CREDENTIAL" if authorization_id is present, "DEVICE" if agent_id is present, or "DATABASE" if namespace_id is present. Otherwise, it is set to "STORAGE_BUCKET" in some cases based on operationName. |
| N/A | target.resource.resource_subtype |
Set to "Pipeline" if pipeline_id is present, "Token" if authorization_id is present, "Agent" if agent_id is present, or "Namespace" if namespace_id is present. |
Changes
2024-01-19
- Changed "metadata.eventtype" value from "SERVICE*" to "USER_RESOURCE_UPDATE_CONTENT" if principal user data and target resource data are present.
- Changed mapping for "IpAddress" from "target.ip" to "principal.ip".
- Changed mapping for "ActorCUID" from "principal.user.product_object_id" to "additional.fields".
- Changed mapping for "ScopeId" from "principal.asset_id" to "resource_ancestors.product_object_id".
- Changed mapping for "_Internal_WorkspaceResourceId" from "target.resource.product_object_id" to "additional.fields".
- Changed mapping for "ProjectId" from "target.resource.attribute.labels" to "target.resource_ancestors.product_object_id".
- Changed mapping for "AuthenticationMechanism" from "security_result.summary" to "extensions.auth.auth_details".
- Changed mapping for "CorrelationId" from "network.session_id" to "additional.fields".
- Changed mapping for "ScopeDisplayName" from "additional.fields" to "target.resource_ancestors.name".
- Changed mapping for "PipelineId" from "additional.fields" to "target.resource.product_object_id".
- Changed mapping for "PipelineName" from "additional.fields" to "target.resource.name".
- Changed mapping for "PipelineScope" from "additional.fields" to "target.resource.attribute.labels".
- Changed mapping for "PipelineRevision" from "additional.fields" to "target.resource.attribute.labels".
- Changed mapping for "ProjectId" from "target.resource.resource.attribute.labels" to "target.resource_ancestors.product_object_id".
- Changed mapping for "Area" from "additional.fields" to "target.application".
- Mapped "MICROSOFT_AZURE" value to "target.asset.attribute.cloud.environment".
- When "AuthenticationMechanism" is having "ServicePrincipal" value, then set "SERVICE_ACCOUNT_TYPE" to "principal.user.account_type", else set "CLOUD_ACCOUNT_TYPE" to "principal.user.account_type".
- Mapped "Category" to "security_result.action_details".
- Mapped "ALLOW" or "BLOCK" to "security_result.action" based on "Details" field.
- Mapped "ActivityId" to "additional.fields".
2024-01-09
- Added Grok and gsub to parse the unparsed JSON logs.
- Mapped "rec.correlationId", "properties.currentHealthStatus", "properties.previousHealthStatus", "properties.type", "properties.cause", "properties.title", "properties.details", "properties.recommendationType", "properties.recommendationCategory", "properties.recommendationImpact", "properties.recommendationName", "properties.recommendationResourceLink", "properties.recommendationSchemaVersion", "properties.eventCategory", "properties.hierarchy", "properties.message", "properties.entity", "identity.claims.xms.tcdt", "identity.claims.aio", "identity.claims.appid", "identity.claims.appidacr", "identity.claims.aud", "identity.claims.exp", "identity.claims.iat", "identity.claims.idtyp", "identity.claims.iss", "identity.claims.uti", "identity.claims.rh", "identity.claims.ver", "identity.claims.nbf", "identity.authorization.evidence.roleAssignmentId", "identity.authorization.evidence.principalType", "identity.authorization.evidence.principalId", "identity.authorization.evidence.roleAssignmentScope", "identity.authorization.evidence.roleDefinitionId" to "security_result.detection_fields".
- Mapped "resultSignature.label", "rec.resultType", "Visibility", "Humidity", "Precipitation","MoonPhase", "Moonrise", "Moonset", "Pressure", "WindSpeed", "UVIndex", "DewPoint", WindDirection", "Sunrise", "Sunset", "Temperature", "Icon", "Conditions" to "additional.fields".
- Mapped "level" to "security_result.severity".
- Mapped "appname" to "target.application".
- Mapped "category.details" to "security.result.category.details".
- Mapped "rec.resourceId" to "target.resource.id".
- Mapped "res.extensionResourceName" to "principal.hostname".
2023-11-23
- Added support for a new pattern of JSON logs.
- Mapped "data.TimeGenerated" to "metadata.event_timestamp".
- When "_Internal_WorkspaceResourceId" is missing, then mapped "topic" to "target.resource.product_object_id".
- Mapped "data.Data.ConnectionId" to "additional.fields".
- Mapped "data.Data.ownerDetails" to "additional.fields".
- Mapped "data.Data.DeploymentResult" to "additional.fields".
- Mapped "data.Data.EnvironmentName" to "additional.fields".
- Mapped "data.Data.JobName" to "additional.fields".
- Mapped "data.Data.StageName" to "additional.fields".
- Mapped "data.Data.RunName" to "additional.fields".
- Mapped "data.Data.RetentionLeaseId" to "additional.fields".
- Mapped "data.Data.CheckSuiteId" to "additional.fields".
- Mapped "data.Data.CheckSuiteStatus" to "additional.fields".
- Mapped "data.Data.ApprovalRequest" to "additional.fields".
- Mapped "data.Data.ApprovalType" to "additional.fields".
- Mapped "subject" to "additional.fields".
- Mapped "data.ActorUserId" to "principal.user.userid".
- Mapped "data.ActorDisplayName" to "principal.user.user_display_name".
- Mapped "data.ActorCUID" to "principal.user.product_object_id".
- Mapped "data.ActorUPN" to "principal.user.email_addresses".
- Mapped "data.ScopeId" to "principal.asset_id".
- Mapped "data.CorrelationId" to "network.session_id".
- Mapped "data.UserAgent" to "network.http.user_agent".
- Mapped "data.ProjectId" to "target.resource.attribute.labels".
- Mapped "data.ScopeType" to "additional.fields".
- Mapped "data.ProjectName" to "target.resource.attribute.labels".
- Mapped "data.Details" to "metadata.description".
- Mapped "data.CategoryDisplayName" to "security_result.rule_name".
- Mapped "data.Area" to "additional.fields".
- Mapped "data.Id" to "metadata.product_log_id".
- Mapped "data.ActionId" to "metadata.product_event_type".
- Mapped "data.Timestamp" to "metadata.event_timestamp".
2022-06-28
- Newly created parser