Collect SailPoint IAM logs

Supported in:

This document explains how you can ingest SailPoint Identity and Access Management (IAM) logs to Google Security Operations using Amazon S3. The parser handles the logs in JSON and XML formats, transforming them into the Unified Data Model (UDM). It differentiates between single UDM events (ProvisioningPlan, AccountRequest, SOAP-ENV), multi UDM events (ProvisioningProject), and UDM entities (Identity), applying specific parsing logic and field mappings for each, including generic event handling for non-XML data.

Before you begin

Make sure you have the following prerequisites:

  • A Google SecOps instance.
  • Privileged access to SailPoint Identity Security Cloud.
  • Privileged access to AWS (S3, IAM, Lambda, EventBridge).

Collect SailPoint IAM prerequisites (IDs, API keys, org IDs, tokens)

  1. Sign in to the SailPoint Identity Security Cloud Admin Console as an administrator.
  2. Go to Global > Security Settings > API Management.
  3. Click Create API Client.
  4. Select Client Credentials as the grant type.
  5. Provide the following configuration details:
    • Name: Enter a descriptive name (for example, Google SecOps Export API).
    • Description: Enter description for the API client.
    • Scopes: Select sp:scopes:all.
  6. Click Create and save the generated API credentials in a secure location.
  7. Record your SailPoint tenant base URL (for example, https://tenant.api.identitynow.com).
  8. Copy and save in a secure location the following details:
    • IDN_CLIENT_ID.
    • IDN_CLIENT_SECRET.
    • IDN_BASE.

Configure AWS S3 bucket and IAM for Google SecOps

  1. Create Amazon S3 bucket following this user guide: Creating a bucket
  2. Save bucket Name and Region for future reference (for example, sailpoint-iam-logs).
  3. Create a User following this user guide: Creating an IAM user.
  4. Select the created User.
  5. Select Security credentials tab.
  6. Click Create Access Key in section Access Keys.
  7. Select Third-party service as Use case.
  8. Click Next.
  9. Optional: Add description tag.
  10. Click Create access key.
  11. Click Download CSV file to save the Access Key and Secret Access Key for future reference.
  12. Click Done.
  13. Select Permissions tab.
  14. Click Add permissions in section Permissions policies.
  15. Select Add permissions.
  16. Select Attach policies directly.
  17. Search for AmazonS3FullAccess policy.
  18. Select the policy.
  19. Click Next.
  20. Click Add permissions.

Configure the IAM policy and role for S3 uploads

  1. In the AWS console, go to IAM > Policies.
  2. Click Create policy > JSON tab.
  3. Copy and paste the following policy.

  4. Policy JSON (replace sailpoint-iam-logs if you entered a different bucket name):

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPutObjects",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::sailpoint-iam-logs/*"
    },
    {
      "Sid": "AllowGetStateObject",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::sailpoint-iam-logs/sailpoint/iam/state.json"
    }
  ]
}

  5. Click Next > Create policy.

  6. Go to IAM > Roles > Create role > AWS service > Lambda.

  7. Attach the newly created policy.

  8. Name the role SailPointIamToS3Role and click Create role.

Create the Lambda function

  1. In the AWS Console, go to Lambda > Functions > Create function.
  2. Click Author from scratch.

  3. Provide the following configuration details:

    Setting Value
    Name sailpoint_iam_to_s3
    Runtime Python 3.13
    Architecture x86_64
    Execution role SailPointIamToS3Role

  4. After the function is created, open the Code tab, delete the stub and paste the following code (sailpoint_iam_to_s3.py).

    #!/usr/bin/env python3
# Lambda: Pull SailPoint Identity Security Cloud audit events and store raw JSON payloads to S3
# - Uses /v3/search API with pagination for audit events.
# - Preserves vendor-native JSON format for identity events.
# - Retries with exponential backoff; unique S3 keys to avoid overwrites.

import os, json, time, uuid, urllib.parse
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

import boto3

S3_BUCKET   = os.environ["S3_BUCKET"]
S3_PREFIX   = os.environ.get("S3_PREFIX", "sailpoint/iam/")
STATE_KEY   = os.environ.get("STATE_KEY", "sailpoint/iam/state.json")
WINDOW_SEC  = int(os.environ.get("WINDOW_SECONDS", "3600"))  # default 1h
HTTP_TIMEOUT= int(os.environ.get("HTTP_TIMEOUT", "60"))
IDN_BASE    = os.environ["IDN_BASE"]  # e.g. https://tenant.api.identitynow.com
CLIENT_ID   = os.environ["IDN_CLIENT_ID"]
CLIENT_SECRET = os.environ["IDN_CLIENT_SECRET"]
SCOPE       = os.environ.get("IDN_SCOPE", "sp:scopes:all")
PAGE_SIZE   = int(os.environ.get("PAGE_SIZE", "250"))
MAX_PAGES   = int(os.environ.get("MAX_PAGES", "20"))
MAX_RETRIES = int(os.environ.get("MAX_RETRIES", "3"))
USER_AGENT  = os.environ.get("USER_AGENT", "sailpoint-iam-to-s3/1.0")

s3 = boto3.client("s3")

def _load_state():
    try:
        obj = s3.get_object(Bucket=S3_BUCKET, Key=STATE_KEY)
        return json.loads(obj["Body"].read())
    except Exception:
        return {}

def _save_state(st):
    s3.put_object(
        Bucket=S3_BUCKET,
        Key=STATE_KEY,
        Body=json.dumps(st, separators=(",", ":")).encode("utf-8"),
        ContentType="application/json",
    )

def _iso(ts: float) -> str:
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(ts))

def _get_oauth_token() -> str:
    """Get OAuth2 access token using Client Credentials flow"""
    token_url = f"{IDN_BASE.rstrip('/')}/oauth/token"

    data = urllib.parse.urlencode({
        'grant_type': 'client_credentials',
        'client_id': CLIENT_ID,
        'client_secret': CLIENT_SECRET,
        'scope': SCOPE
    }).encode('utf-8')

    req = Request(token_url, data=data, method="POST")
    req.add_header("Content-Type", "application/x-www-form-urlencoded")
    req.add_header("User-Agent", USER_AGENT)

    with urlopen(req, timeout=HTTP_TIMEOUT) as r:
        response = json.loads(r.read())
        return response["access_token"]

def _search_events(access_token: str, created_from: str, search_after: list = None) -> list:
    """Search for audit events using SailPoint's /v3/search API"""
    search_url = f"{IDN_BASE.rstrip('/')}/v3/search"

    # Build search query for events created after specified time
    query_str = f'created:">={created_from}"'

    payload = {
        "indices": ["events"],
        "query": {"query": query_str},
        "sort": ["created", "+id"],
        "limit": PAGE_SIZE
    }

    if search_after:
        payload["searchAfter"] = search_after

    attempt = 0
    while True:
        req = Request(search_url, data=json.dumps(payload).encode('utf-8'), method="POST")
        req.add_header("Content-Type", "application/json")
        req.add_header("Accept", "application/json")
        req.add_header("Authorization", f"Bearer {access_token}")
        req.add_header("User-Agent", USER_AGENT)

        try:
            with urlopen(req, timeout=HTTP_TIMEOUT) as r:
                response = json.loads(r.read())
                # Handle different response formats
                if isinstance(response, list):
                    return response
                return response.get("results", response.get("data", []))
        except (HTTPError, URLError) as e:
            attempt += 1
            print(f"HTTP error on attempt {attempt}: {e}")
            if attempt > MAX_RETRIES:
                raise
            # exponential backoff with jitter
            time.sleep(min(60, 2 ** attempt) + (time.time() % 1))

def _put_events_data(events: list, from_ts: float, to_ts: float, page_num: int) -> str:
    # Create unique S3 key for events data
    ts_path = time.strftime("%Y/%m/%d", time.gmtime(to_ts))
    uniq = f"{int(time.time()*1e6)}_{uuid.uuid4().hex[:8]}"
    key = f"{S3_PREFIX}{ts_path}/sailpoint_iam_{int(from_ts)}_{int(to_ts)}_p{page_num:03d}_{uniq}.json"

    s3.put_object(
        Bucket=S3_BUCKET, 
        Key=key, 
        Body=json.dumps(events, separators=(",", ":")).encode("utf-8"), 
        ContentType="application/json",
        Metadata={
            'source': 'sailpoint-iam',
            'from_timestamp': str(int(from_ts)),
            'to_timestamp': str(int(to_ts)),
            'page_number': str(page_num),
            'events_count': str(len(events))
        }
    )
    return key

def _get_item_id(item: dict) -> str:
    """Extract ID from event item, trying multiple possible fields"""
    for field in ("id", "uuid", "eventId", "_id"):
        if field in item and item[field]:
            return str(item[field])
    return ""

def lambda_handler(event=None, context=None):
    st = _load_state()
    now = time.time()
    from_ts = float(st.get("last_to_ts") or (now - WINDOW_SEC))
    to_ts = now

    # Get OAuth token
    access_token = _get_oauth_token()

    created_from = _iso(from_ts)
    print(f"Fetching SailPoint IAM events from: {created_from}")

    # Handle pagination state
    last_created = st.get("last_created")
    last_id = st.get("last_id")
    search_after = [last_created, last_id] if (last_created and last_id) else None

    pages = 0
    total_events = 0
    written_keys = []
    newest_created = last_created or created_from
    newest_id = last_id or ""

    while pages < MAX_PAGES:
        events = _search_events(access_token, created_from, search_after)

        if not events:
            break

        # Write page to S3
        key = _put_events_data(events, from_ts, to_ts, pages + 1)
        written_keys.append(key)
        total_events += len(events)

        # Update pagination state from last item
        last_event = events[-1]
        last_event_created = last_event.get("created") or last_event.get("metadata", {}).get("created")
        last_event_id = _get_item_id(last_event)

        if last_event_created:
            newest_created = last_event_created
        if last_event_id:
            newest_id = last_event_id

        search_after = [newest_created, newest_id]
        pages += 1

        # If we got less than page size, we're done
        if len(events) < PAGE_SIZE:
            break

    print(f"Successfully retrieved {total_events} events across {pages} pages")

    # Save state for next run
    st["last_to_ts"] = to_ts
    st["last_created"] = newest_created
    st["last_id"] = newest_id
    st["last_successful_run"] = now
    _save_state(st)

    return {
        "statusCode": 200,
        "body": {
            "success": True,
            "pages": pages,
            "total_events": total_events,
            "s3_keys": written_keys,
            "from_timestamp": from_ts,
            "to_timestamp": to_ts,
            "last_created": newest_created,
            "last_id": newest_id
        }
    }

if __name__ == "__main__":
    print(lambda_handler())

  5. Go to Configuration > Environment variables.

  6. Click Edit > Add new environment variable.

  7. Enter the environment variables provided in the following table, replacing the examples values with your values.

    Environment variables

    Key Example value
    S3_BUCKET sailpoint-iam-logs
    S3_PREFIX sailpoint/iam/
    STATE_KEY sailpoint/iam/state.json
    WINDOW_SECONDS 3600
    HTTP_TIMEOUT 60
    MAX_RETRIES 3
    USER_AGENT sailpoint-iam-to-s3/1.0
    IDN_BASE https://tenant.api.identitynow.com
    IDN_CLIENT_ID your-client-id (from step 2)
    IDN_CLIENT_SECRET your-client-secret (from step 2)
    IDN_SCOPE sp:scopes:all
    PAGE_SIZE 250
    MAX_PAGES 20

  8. After the function is created, stay on its page (or open Lambda > Functions > your-function).

  9. Select the Configuration tab.

  10. In the General configuration panel, click Edit.

  11. Change Timeout to 5 minutes (300 seconds) and click Save.

Create an EventBridge schedule

  1. Go to Amazon EventBridge > Scheduler > Create schedule.
  2. Provide the following configuration details:
    • Recurring schedule: Rate (1 hour).
    • Target: Your Lambda function sailpoint_iam_to_s3.
    • Name: sailpoint-iam-1h.
  3. Click Create schedule.

(Optional) Create read-only IAM user & keys for Google SecOps

  1. Go to AWS Console > IAM > Users.
  2. Click Add users.
  3. Provide the following configuration details:
    • User: Enter secops-reader.
    • Access type: Select Access key – Programmatic access.
  4. Click Create user.
  5. Attach minimal read policy (custom): Users > secops-reader > Permissions > Add permissions > Attach policies directly > Create policy.

  6. JSON:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::sailpoint-iam-logs/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::sailpoint-iam-logs"
    }
  ]
}

  7. Name = secops-reader-policy.

  8. Click Create policy > search/select > Next > Add permissions.

  9. Create access key for secops-reader: Security credentials > Access keys.

  10. Click Create access key.

  11. Download the .CSV. (You'll paste these values into the feed).

Configure a feed in Google SecOps to ingest SailPoint IAM logs

  1. Go to SIEM Settings > Feeds.
  2. Click + Add New Feed.
  3. In the Feed name field, enter a name for the feed (for example, SailPoint IAM logs).
  4. Select Amazon S3 V2 as the Source type.
  5. Select SailPoint IAM as the Log type.
  6. Click Next.
  7. Specify values for the following input parameters:
    • S3 URI: s3://sailpoint-iam-logs/sailpoint/iam/
    • Source deletion options: Select deletion option according to your preference.
    • Maximum File Age: Include files modified in the last number of days. Default is 180 days.
    • Access Key ID: User access key with access to the S3 bucket.
    • Secret Access Key: User secret key with access to the S3 bucket.
    • Asset namespace: The asset namespace.
    • Ingestion labels: The label applied to the events from this feed.
  8. Click Next.
  9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM mapping table

Log field UDM mapping Logic
action metadata.description The value of the action field from the raw log.
actor.name principal.user.user_display_name The value of the actor.name field from the raw log.
attributes.accountName principal.user.group_identifiers The value of the attributes.accountName field from the raw log.
attributes.appId target.asset_id "App ID: " concatenated with the value of the attributes.appId field from the raw log.
attributes.attributeName additional.fields[0].value.string_value The value of the attributes.attributeName field from the raw log, placed within an additional.fields object. The key is set to "Attribute Name".
attributes.attributeValue additional.fields[1].value.string_value The value of the attributes.attributeValue field from the raw log, placed within an additional.fields object. The key is set to "Attribute Value".
attributes.cloudAppName target.application The value of the attributes.cloudAppName field from the raw log.
attributes.hostName target.hostname, target.asset.hostname The value of the attributes.hostName field from the raw log.
attributes.interface additional.fields[2].value.string_value The value of the attributes.interface field from the raw log, placed within an additional.fields object. The key is set to "Interface".
attributes.operation security_result.action_details The value of the attributes.operation field from the raw log.
attributes.previousValue additional.fields[3].value.string_value The value of the attributes.previousValue field from the raw log, placed within an additional.fields object. The key is set to "Previous Value".
attributes.provisioningResult security_result.detection_fields.value The value of the attributes.provisioningResult field from the raw log, placed within a security_result.detection_fields object. The key is set to "Provisioning Result".
attributes.sourceId principal.labels[0].value The value of the attributes.sourceId field from the raw log, placed within a principal.labels object. The key is set to "Source Id".
attributes.sourceName principal.labels[1].value The value of the attributes.sourceName field from the raw log, placed within a principal.labels object. The key is set to "Source Name".
auditClassName metadata.product_event_type The value of the auditClassName field from the raw log.
created metadata.event_timestamp.seconds, metadata.event_timestamp.nanos The value of the created field from the raw log, converted to timestamp if instant.epochSecond is not present.
id metadata.product_log_id The value of the id field from the raw log.
instant.epochSecond metadata.event_timestamp.seconds The value of the instant.epochSecond field from the raw log, used for timestamp.
ipAddress principal.asset.ip, principal.ip The value of the ipAddress field from the raw log.
interface additional.fields[0].value.string_value The value of the interface field from the raw log, placed within an additional.fields object. The key is set to "interface".
loggerName intermediary.application The value of the loggerName field from the raw log.
message metadata.description, security_result.description Used for various purposes, including setting the description in metadata and security_result, and extracting XML content.
name security_result.description The value of the name field from the raw log.
operation target.resource.attribute.labels[0].value, metadata.product_event_type The value of the operation field from the raw log, placed within a target.resource.attribute.labels object. The key is set to "operation". Also used for metadata.product_event_type.
org principal.administrative_domain The value of the org field from the raw log.
pod principal.location.name The value of the pod field from the raw log.
referenceClass additional.fields[1].value.string_value The value of the referenceClass field from the raw log, placed within an additional.fields object. The key is set to "referenceClass".
referenceId additional.fields[2].value.string_value The value of the referenceId field from the raw log, placed within an additional.fields object. The key is set to "referenceId".
sailPointObjectName additional.fields[3].value.string_value The value of the sailPointObjectName field from the raw log, placed within an additional.fields object. The key is set to "sailPointObjectName".
serverHost principal.hostname, principal.asset.hostname The value of the serverHost field from the raw log.
stack additional.fields[4].value.string_value The value of the stack field from the raw log, placed within an additional.fields object. The key is set to "Stack".
status security_result.severity_details The value of the status field from the raw log.
target additional.fields[4].value.string_value The value of the target field from the raw log, placed within an additional.fields object. The key is set to "target".
target.name principal.user.userid The value of the target.name field from the raw log.
technicalName security_result.summary The value of the technicalName field from the raw log.
thrown.cause.message xml_body, detailed_message The value of the thrown.cause.message field from the raw log, used to extract XML content.
thrown.message xml_body, detailed_message The value of the thrown.message field from the raw log, used to extract XML content.
trackingNumber additional.fields[5].value.string_value The value of the trackingNumber field from the raw log, placed within an additional.fields object. The key is set to "Tracking Number".
type metadata.product_event_type The value of the type field from the raw log.
_version metadata.product_version The value of the _version field from the raw log.
N/A metadata.event_timestamp Derived from instant.epochSecond or created fields.
N/A metadata.event_type Determined by parser logic based on various fields, including has_principal_user, has_target_application, technicalName, and action. Default value is "GENERIC_EVENT".
N/A metadata.log_type Set to "SAILPOINT_IAM".
N/A metadata.product_name Set to IAM.
N/A metadata.vendor_name Set to "SAILPOINT".
N/A extensions.auth.type Set to "AUTHTYPE_UNSPECIFIED" in certain conditions.
N/A target.resource.attribute.labels[0].key Set to "operation".

Need more help? Get answers from Community members and Google SecOps professionals.