Create backup and restore audit logs

This page contains the steps to install and configure the necessary components to create a backup. It also describes how to restore and recover access to historic audit logs from the backup.

Set up a backup in a remote bucket

This section contains the steps to create a backup for audit logs in an S3-compliant bucket.

Before you begin

Before you can create a backup for your audit logs, you must have access to the following resources:

  • A remote S3 bucket with an endpoint, a secret access key, and an access key ID.
  • A certificate authority (CA) certificate for the storage system.
  • A working cluster.

Obtain the credentials to access the source bucket

Work through the following steps to find the credentials of the bucket that contains the audit logs:

  1. On the root admin cluster, list the buckets in your project namespace:

    kubectl get bucket -n PROJECT_NAMESPACE
    

    The output must look like the following example, where audit logging buckets show a name and an endpoint:

    NAME                          BUCKET NAME                   DESCRIPTION                                                         STORAGE CLASS   FULLY-QUALIFIED-BUCKET-NAME         ENDPOINT                                       REGION   BUCKETREADY   REASON                    MESSAGE
    audit-logs-loki-all           audit-logs-loki-all           Bucket for storing audit-logs-loki-all logs                         Standard        wwq2y-audit-logs-loki-all           https://appliance-objectstorage.zone1.google.gdch.test   zone1    True          BucketCreationSucceeded   Bucket successfully created.
    cortex-metrics-alertmanager   cortex-metrics-alertmanager   storage bucket for cortex metrics alertmanager configuration data   Standard        wwq2y-cortex-metrics-alertmanager   https://appliance-objectstorage.zone1.google.gdch.test   zone1    True          BucketCreationSucceeded   Bucket successfully created.
    cortex-metrics-blocks         cortex-metrics-blocks         storage bucket for cortex metrics data                              Standard        wwq2y-cortex-metrics-blocks         https://appliance-objectstorage.zone1.google.gdch.test   zone1    True          BucketCreationSucceeded   Bucket successfully created.
    cortex-metrics-ruler          cortex-metrics-ruler          storage bucket for cortex metrics rules data                        Standard        wwq2y-cortex-metrics-ruler          https://appliance-objectstorage.zone1.google.gdch.test   zone1    True          BucketCreationSucceeded   Bucket successfully created.
    ops-logs-loki-all             ops-logs-loki-all             Bucket for storing ops-logs-loki-all logs                           Standard        wwq2y-ops-logs-loki-all             https://appliance-objectstorage.zone1.google.gdch.test```
    
  2. Using the information from the output you obtained, set the following environment variables for the transfer:

    SRC_BUCKET= BUCKET_NAME
    SRC_ENDPOINT = ENDPOINT
    SRC_PATH= FULLY_QUALIFIED_BUCKET_NAME
    

    Replace the following:

    • BUCKET_NAME: the name of the bucket that contains the audit logs for which you want to create the backup. This value is on the BUCKET NAME field of the output.
    • ENDPOINT: the endpoint of the bucket that contains the audit logs for which you want to create the backup. This value is on the ENDPOINT field of the output.
    • FULLY_QUALIFIED_BUCKET_NAME: the fully-qualified name of the bucket that contains the audit logs for which you want to create the backup. This value is on the FULLY-QUALIFIED-BUCKET-NAME field of the output.
  3. Get the secret for the bucket you selected in the previous step:

    kubectl get secret -n PROJECT_NAMESPACE -o json| jq --arg jq_src $SRC_BUCKET '.items[].metadata|select(.annotations."object.gdc.goog/subject"==$jq_src)|.name'
    

    The output must look like the following example, where the secret name of the bucket is displayed:

    "object-storage-key-sysstd-sa-olxv4dnwrwul4bshu37ikebgovrnvl773owaw3arx225rfi56swa"
    
  4. Using the secret name from the output you obtained, set the following environment variable:

    SRC_CREDENTIALS="PROJECT_NAMESPACE/SECRET_NAME"
    

    Replace SECRET_NAME with the secret name you obtained in the previous output.

  5. Create the secret for the CA certificate of the storage system:

    kubectl create secret generic -n PROJECT_NAMESPACE audit-log-loki-ca \
    --from-literal=ca.crt=CERTIFICATE
    

    Replace CERTIFICATE with the CA certificate of the storage system.

  6. Set the following environment variable:

    SRC_CA_CERTIFICATE=PROJECT_NAMESPACE/audit-log-loki-ca
    

Obtain the credentials to access the remote bucket

Work through the following steps to find the credentials of the bucket where you want to create the backup:

  1. Set the following environment variables:

    DST_ACCESS_KEY_ID= ACCESS_KEY
    DST_SECRET_ACCESS_KEY= ACCESS_SECRET
    DST_ENDPOINT= REMOTE_ENDPOINT
    DST_PATH= REMOTE_BUCKET_NAME
    

    Replace the following:

    • ACCESS_KEY: the access key of the destination remote bucket.
    • ACCESS_SECRET: the access secret of the destination remote bucket.
    • REMOTE_ENDPOINT: the endpoint of the destination remote bucket.
    • REMOTE_BUCKET_NAME: the name of the destination remote bucket.
  2. Create a secret for the remote bucket:

    kubectl create secret generic -n PROJECT_NAMESPACE s3-bucket-credentials \
      --from-literal=access-key-id=$DST_ACCESS_KEY_ID \
      --from-literal=secret-access-key=$DST_SECRET_ACCESS_KEY
    
  3. Set the following environment variable with the location of the secret:

    DST_CREDENTIALS=PROJECT_NAMESPACE/s3-bucket-credentials
    
  4. If the destination bucket requires setting up a CA certificate, create a secret with the CA certificate of the bucket:

    kubectl create secret generic -n PROJECT_NAMESPACE s3-bucket-ca \
    --from-literal=ca.crt=REMOTE_CERTIFICATE
    

    Replace REMOTE_CERTIFICATE with the CA certificate of the destination remote bucket.

  5. Set the following environment variable with the location of the certificate:

    DST_CA_CERTIFICATE=PROJECT_NAMESPACE/s3-bucket-ca
    

Set up the transfer of audit logs

Work through the following steps to configure the transfer of audit logs from the source bucket to the destination bucket for the backup:

  1. Create a service account for the audit log transfer job. You must provide access to the service account to read the source bucket and secrets in the project namespace.

    kubectl apply -f - <<EOF
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: audit-log-transfer-sa
      namespace: PROJECT_NAMESPACE
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: read-secrets-role
      namespace: PROJECT_NAMESPACE
    rules:
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get", "watch", "list"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: read-secrets-rolebinding
      namespace: PROJECT_NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: audit-log-transfer-sa
      namespace: PROJECT_NAMESPACE
    roleRef:
      kind: Role
      name: read-secrets-role
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: audit-log-read-bucket-role
      namespace: PROJECT_NAMESPACE
    rules:
    - apiGroups:
      - object.gdc.goog
      resourceNames:
      - $SRC_BUCKET # Source bucket name
      resources:
      - buckets
      verbs:
      - read-object
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: audit-log-transfer-role-binding
      namespace: PROJECT_NAMESPACE
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: audit-log-read-bucket-role
    subjects:
    - kind: ServiceAccount
      name: audit-log-transfer-sa
      namespace: PROJECT_NAMESPACE
    ---
    EOF
    
  2. Create a transfer job to export logs to the remote bucket:

    kubectl apply -f - <<EOF
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: audit-log-transfer-job
      namespace: PROJECT_NAMESPACE
    spec:
      template:
        spec:
          serviceAccountName: audit-log-transfer-sa
          containers:
            - name: storage-transfer-pod
              image: gcr.io/private-cloud-staging/storage-transfer:latest
              imagePullPolicy: Always
              command:
                - /storage-transfer
              args:
                - '--src_endpoint=$SRC_ENDPOINT
                - '--dst_endpoint=$DST_ENDPOINT
                - '--src_path=\$SRC_PATH
                - '--dst_path=\$DST_PATH
                - '--src_credentials=$SRC_CREDENTIALS
                - '--dst_credentials=$DST_CREDENTIALS
                - '--dst_ca_certificate_reference=$DST_CA_CERTIFICATE # Optional. Based on destination type.
                - '--src_ca_certificate_reference=$SRC_CA_CERTIFICATE
                - '--src_type=s3'
                - '--dst_type=s3'
                - '--bandwidth_limit=100M' # Optional of the form '10K', '100M', '1G' bytes per second
          restartPolicy: OnFailure # Will restart on failure.
    ---
    EOF
    

After scheduling the job, you can monitor your data transfer by providing the job name as audit-log-transfer-job and your project namespace.

The job ends when all the data has been transferred to the destination bucket.

Restore audit logs from the backup

This section contains the steps to restore audit logs from the backup.

Before you begin

Before you can restore your audit logs, you must have access to the following resources:

  • The audit log backup bucket with an endpoint, a secret access key, and an access key ID.
  • A certificate authority (CA) certificate for the storage system.
  • A working cluster.

Create a bucket to restore audit logs

Work through the following steps to create a bucket to store the restored audit logs:

  1. Create the bucket resource and the service account:

    kubectl apply -f - <<EOF
    ---
    apiVersion: object.gdc.goog/v1
    kind: Bucket
    metadata:
      annotations:
        object.gdc.goog/audit-logs: IO
      labels:
        logging.private.gdch.goog/loggingpipeline-name: default
      name: audit-logs-loki-restore
      namespace: PROJECT_NAMESPACE
    spec:
      bucketPolicy:
        lockingPolicy:
          defaultObjectRetentionDays: 1
      description: Bucket for storing audit-logs-loki logs restore
      storageClass: Standard
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: audit-logs-loki-restore-buckets-role
      namespace: PROJECT_NAMESPACE
    rules:
    - apiGroups:
      - object.gdc.goog
      resourceNames:
      - audit-logs-loki-restore
      resources:
      - buckets
      verbs:
      - read-object
      - write-object
    ---
    apiVersion: v1
    automountServiceAccountToken: false
    kind: ServiceAccount
    metadata:
      labels:
        logging.private.gdch.goog/loggingpipeline-name: default
      name: audit-logs-loki-restore-sa
      namespace: PROJECT_NAMESPACE
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: audit-logs-loki-restore
      namespace: PROJECT_NAMESPACE
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: audit-logs-loki-restore-buckets-role
    subjects:
    - kind: ServiceAccount
      name: audit-logs-loki-restore-sa
      namespace: PROJECT_NAMESPACE
    EOF
    

    The bucket and secret are created.

  2. View the created bucket:

    kubectl get bucket audit-logs-loki-restore -n PROJECT_NAMESPACE
    

    The output must look like the following example. The creation of the bucket might take a few minutes.

    NAME                      BUCKET NAME               DESCRIPTION                                       STORAGE CLASS   FULLY-QUALIFIED-BUCKET-NAME     ENDPOINT                                       REGION   BUCKETREADY   REASON                    MESSAGE
    audit-logs-loki-restore   audit-logs-loki-restore   Bucket for storing audit-logs-loki logs restore   Standard        dzbl6-audit-logs-loki-restore   https://objectstorage.zone1.google.gdch.test   zone1    True          BucketCreationSucceeded   Bucket successfully created.
    

    The output must show the bucket you created. The creation of the bucket might take a few minutes.

  3. Using the information from the output you obtained, set the following environment variables:

    DST_BUCKET= RESTORE_BUCKET_NAME
    DST_ENDPOINT = RESTORE_ENDPOINT
    DST_PATH= RESTORE_FULLY_QUALIFIED_BUCKET_NAME
    

    Replace the following:

    • RESTORE_BUCKET_NAME: the name of the bucket for the restoration of audit logs. This value is on the BUCKET NAME field of the output.
    • RESTORE_ENDPOINT: the endpoint of the bucket for the restoration of audit logs. This value is on the ENDPOINT field of the output.
    • RESTORE_FULLY_QUALIFIED_BUCKET_NAME: the fully-qualified name of the bucket for the restoration of audit logs. This value is on the FULLY-QUALIFIED-BUCKET-NAME field of the output.
  4. Get the secret of the created bucket:

    kubectl get secret -n PROJECT_NAMESPACE -o json| jq --arg jq_src $DST_BUCKET '.items[].metadata|select(.annotations."object.gdc.goog/subject"==$jq_src)|.name'
    

    The output must look like the following example, where the secret name of the bucket is displayed:

    "object-storage-key-sysstd-sa-olxv4dnwrwul4bshu37ikebgovrnvl773owaw3arx225rfi56swa"
    
  5. Using the secret name from the output you obtained, set the following environment variables:

    DST_SECRET_NAME=RESTORE_SECRET_NAME
    DST_CREDENTIALS="PROJECT_NAMESPACE/RESTORE_SECRET_NAME"
    

    Replace RESTORE_SECRET_NAME with the secret name you obtained in the previous output.

  6. Create the secret for the CA certificate of the storage system:

    kubectl create secret generic -n PROJECT_NAMESPACE audit-log-loki-restore-ca \
    --from-literal=ca.crt=CERTIFICATE
    

    Replace CERTIFICATE with the CA certificate of the storage system.

  7. Set the following environment variable for the location of the certificate:

    DST_CA_CERTIFICATE=PROJECT_NAMESPACE/audit-log-loki-restore-ca
    

Obtain the credentials to access the backup bucket

Work through the following steps to find the credentials of the bucket that contains the backup of audit logs:

  1. Set the following environment variables:

    SRC_ACCESS_KEY_ID= ACCESS_KEY
    SRC_SECRET_ACCESS_KEY= ACCESS_SECRET
    SRC_ENDPOINT= REMOTE_ENDPOINT
    SRC_PATH= REMOTE_BUCKET_NAME
    

    Replace the following:

    • ACCESS_KEY: the access key of the backup bucket.
    • ACCESS_SECRET: the access secret of the backup bucket.
    • REMOTE_ENDPOINT: the endpoint of the backup bucket.
    • REMOTE_BUCKET_NAME: the name of the backup bucket.
  2. Create a secret for the backup bucket:

    kubectl create secret generic -n PROJECT_NAMESPACE s3-backup-bucket-credentials \
      --from-literal=access-key-id=$SRC_ACCESS_KEY_ID \
      --from-literal=secret-access-key=$SRC_SECRET_ACCESS_KEY
    
  3. Set the following environment variable with the location of the secret:

    SRC_CREDENTIALS=PROJECT_NAMESPACE/s3-backup-bucket-credentials
    
  4. Create a secret with the CA certificate of the bucket:

    kubectl create secret generic -n PROJECT_NAMESPACE s3-backup-bucket-ca \
    --from-literal=ca.crt=BACKUP_CERTIFICATE
    

    Replace BACKUP_CERTIFICATE with the CA certificate of the backup bucket.

  5. Set the following environment variable with the location of the certificate:

    SRC_CA_CERTIFICATE=PROJECT_NAMESPACE/s3-backup-bucket-ca
    

Set up the restoration of audit logs

Work through the following steps to configure the transfer of audit logs from the backup bucket to the restoration bucket:

  1. Create a service account for the audit log transfer job. You must provide access to the service account to read and write from the bucket and secrets in the project namespace.

    kubectl apply -f - <<EOF
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: audit-log-restore-sa
      namespace: PROJECT_NAMESPACE
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: read-secrets-role
      namespace: PROJECT_NAMESPACE
    rules:
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get", "watch", "list"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: read-secrets-rolebinding-restore
      namespace: PROJECT_NAMESPACE
    subjects:
    - kind: ServiceAccount
      name: audit-log-restore-sa
      namespace: PROJECT_NAMESPACE
    roleRef:
      kind: Role
      name: read-secrets-role
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: audit-log-restore-bucket-role
      namespace: PROJECT_NAMESPACE
    rules:
    - apiGroups:
      - object.gdc.goog
      resourceNames:
      - $DST_BUCKET # Source bucket name
      resources:
      - buckets
      verbs:
      - read-object
      - write-object
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: audit-log-restore-role-binding
      namespace: PROJECT_NAMESPACE
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: audit-log-restore-bucket-role
    subjects:
    - kind: ServiceAccount
      name: audit-log-restore-sa
      namespace: PROJECT_NAMESPACE
    ---
    EOF
    
  2. Create a transfer job to restore logs from the remote backup bucket:

    kubectl apply -f - <<EOF
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: audit-log-restore-job
      namespace: PROJECT_NAMESPACE
    spec:
      template:
        spec:
          serviceAccountName: audit-log-restore-sa
          containers:
            - name: storage-transfer-pod
              image: gcr.io/private-cloud-staging/storage-transfer:latest
              imagePullPolicy: Always
              command:
                - /storage-transfer
              args:
                - '--src_endpoint=$SRC_ENDPOINT
                - '--dst_endpoint=$DST_ENDPOINT
                - '--src_path=\$SRC_PATH
                - '--dst_path=\$DST_PATH
                - '--src_credentials=$SRC_CREDENTIALS
                - '--dst_credentials=$DST_CREDENTIALS
                - '--dst_ca_certificate_reference=$DST_CA_CERTIFICATE
                - '--src_ca_certificate_reference=$SRC_CA_CERTIFICATE # Optional. Based on destination type
                - '--src_type=s3'
                - '--dst_type=s3'
                - '--bandwidth_limit=100M' # Optional of the form '10K', '100M', '1G' bytes per second
          restartPolicy: OnFailure # Will restart on failure.
    ---
    EOF
    

After scheduling the job, you can monitor your data transfer by providing the job name as audit-log-restore-job and your project namespace.

The job ends when all the data has been transferred to the destination bucket.

Deploy an audit log instance to access the logs

You must deploy a Loki instance, also called the audit log instance, to access the restored logs.

To set up the audit log instance, use the audit-log-restore-sa service account that you created for the restoration job. Work through the following steps to deploy the instance:

  1. Create a ConfigMap object for the configuration of the instance:

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: audit-logs-loki-restore
      namespace: PROJECT_NAMESPACE
    data:
      loki.yaml: |-
        auth_enabled: true
        common:
          ring:
            kvstore:
              store: inmemory
        chunk_store_config:
          max_look_back_period: 0s
        compactor:
          shared_store: s3
          working_directory: /data/loki/boltdb-shipper-compactor
          compaction_interval: 10m
          retention_enabled: true
          retention_delete_delay: 2h
          retention_delete_worker_count: 150
        ingester:
          chunk_target_size: 1572864
          chunk_encoding: snappy
          max_chunk_age: 2h
          chunk_idle_period: 90m
          chunk_retain_period: 30s
          autoforget_unhealthy: true
          lifecycler:
            ring:
              kvstore:
                store: inmemory
              replication_factor: 1
              heartbeat_timeout: 10m
          max_transfer_retries: 0
          wal:
            enabled: true
            flush_on_shutdown: true
            dir: /wal
            checkpoint_duration: 1m
            replay_memory_ceiling: 20GB
        limits_config:
          retention_period: 48h
          enforce_metric_name: false
          reject_old_samples: false
          ingestion_rate_mb: 256
          ingestion_burst_size_mb: 256
          max_streams_per_user: 20000
          max_global_streams_per_user: 20000
          per_stream_rate_limit: 256MB
          per_stream_rate_limit_burst: 256MB
          shard_streams:
            enabled: false
            desired_rate: 3MB
        schema_config:
          configs:
          - from: "2020-10-24"
            index:
              period: 24h
              prefix: index_
            object_store: s3
            schema: v11
            store: boltdb-shipper
        server:
          http_listen_port: 3100
          grpc_server_max_recv_msg_size: 104857600
          grpc_server_max_send_msg_size: 104857600
        analytics:
          reporting_enabled: false
        storage_config:
          boltdb_shipper:
            active_index_directory: /data/loki/boltdb-shipper-active
            cache_location: /data/loki/boltdb-shipper-cache
            cache_ttl: 24h
            shared_store: s3
          aws:
            endpoint: $DST_ENDPOINT
            bucketnames: $DST_PATH
            access_key_id: ${S3_ACCESS_KEY_ID}
            secret_access_key: ${S3_SECRET_ACCESS_KEY}
            s3forcepathstyle: true
    EOF
    
  2. Deploy the instance with a service for accessing logs from the restoration bucket:

    kubectl apply -f - <<EOF
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      labels:
        app: audit-logs-loki-restore
        logging.private.gdch.goog/loggingpipeline-name: default
      name: audit-logs-loki-restore
      namespace: PROJECT_NAMESPACE
    spec:
      persistentVolumeClaimRetentionPolicy:
        whenDeleted: Retain
        whenScaled: Retain
      podManagementPolicy: OrderedReady
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app: audit-logs-loki-restore
      serviceName: audit-logs-loki-restore
      template:
        metadata:
          labels:
            app: audit-logs-loki-restore
            app.kubernetes.io/part-of: audit-logs-loki-restore
            egress.networking.gke.io/enabled: "true"
            istio.io/rev: default
            logging.private.gdch.goog/log-type: audit
        spec:
          affinity:
            nodeAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - preference:
                  matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: DoesNotExist
                  - key: node-role.kubernetes.io/master
                    operator: DoesNotExist
                weight: 1
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - audit-logs-loki-restore
                  topologyKey: kubernetes.io/hostname
                weight: 100
          containers:
          - args:
            - -config.file=/etc/loki/loki.yaml
            - -config.expand-env=true
            - -target=all
            env:
            - name: S3_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: access-key-id
                  name: $DST_SECRET_NAME
                  optional: false
            - name: S3_SECRET_ACCESS_KEY
              valueFrom:
                  secretKeyRef:
                    key: secret-access-key
                    name: $DST_SECRET_NAME
                    optional: false
            image: gcr.io/private-cloud-staging/loki:v2.8.4-gke.2
            imagePullPolicy: Always
            livenessProbe:
              failureThreshold: 3
              httpGet:
                path: /ready
                port: http-metrics
                scheme: HTTP
              initialDelaySeconds: 330
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            name: audit-logs-loki-restore
            ports:
            - containerPort: 3100
              name: http-metrics
              protocol: TCP
            - containerPort: 7946
              name: gossip-ring
              protocol: TCP
            readinessProbe:
              failureThreshold: 3
              httpGet:
                path: /ready
                port: http-metrics
                scheme: HTTP
              initialDelaySeconds: 45
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            resources:
              limits:
                ephemeral-storage: 2000Mi
                memory: 8000Mi
              requests:
                cpu: 300m
                ephemeral-storage: 2000Mi
                memory: 1000Mi
            securityContext:
              readOnlyRootFilesystem: true
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/loki
              name: config
            - mountPath: /data
              name: loki-storage
            - mountPath: /tmp
              name: temp
            - mountPath: /tmp/loki/rules-temp
              name: tmprulepath
            - mountPath: /etc/ssl/certs/storage-cert.crt
              name: storage-cert
              subPath: ca.crt
            - mountPath: /wal
              name: loki-storage
          dnsPolicy: ClusterFirst
          priorityClassName: audit-logs-loki-priority
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext:
            fsGroup: 10001
            runAsGroup: 10001
            runAsUser: 10001
          serviceAccount: audit-logs-loki-restore-sa
          serviceAccountName: audit-logs-loki-restore-sa
          terminationGracePeriodSeconds: 4800
          volumes:
          - emptyDir: {}
            name: temp
          - configMap:
              defaultMode: 420
              name: audit-logs-loki-restore
            name: config
          - emptyDir: {}
            name: tmprulepath
          - name: storage-cert
            secret:
              defaultMode: 420
              secretName: web-tls
      updateStrategy:
        type: RollingUpdate
      volumeClaimTemplates:
      - apiVersion: v1
        kind: PersistentVolumeClaim
        metadata:
          creationTimestamp: null
          name: loki-storage
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
          storageClassName: standard-rwo
          volumeMode: Filesystem
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: audit-logs-loki-restore
      namespace: PROJECT_NAMESPACE
    spec:
      internalTrafficPolicy: Cluster
      ipFamilies:
      - IPv4
      ipFamilyPolicy: SingleStack
      ports:
      - name: http-metrics
        port: 3100
        protocol: TCP
        targetPort: http-metrics
      selector:
        app: audit-logs-loki-restore
      sessionAffinity: None
      type: ClusterIP
    ---
    EOF
    

Configure the monitoring instance to view logs from the data source

Work through the following steps to configure Grafana, also called the monitoring instance, to view the restored audit logs from the audit log instance:

  1. Go to the endpoint of the monitoring instance of your project.
  2. From the navigation menu of the user interface (UI), click Administration > Data sources.
  3. Click Add new data source.
  4. On the Add data source page, select Loki.
  5. On the Settings page, enter Audit Logs - Restore as a value for the Name field.
  6. On the HTTP section, enter the following value for the URL field:

    http://audit-logs-loki-restore.PROJECT_NAMESPACE.svc:3100
    
  7. On the Custom HTTP Headers section, enter the following values in the corresponding fields:

    • Header: X-Scope-OrgID
    • Value: infra-obs

In figure 1, Loki is displayed as an option on the Add data source page of the monitoring instance UI. In figure 2, the Settings page shows the fields that you must fill out to set up the data source.

The Loki option is displayed as a data source on the Add data source page of the UI.

Figure 1. The Add data source page on the UI of the monitoring instance.

The relevant fields to configure Loki as the data source are displayed on the Settings page

Figure 2. The Settings page on the UI of the monitoring instance.

The Audit Logs - Restore option is now available as a data source in the log explorer.