Gitsync

GitSync is a robust integration built by Google Security Operations SOAR Professional Services team and designed to synchronize Google Security Operations SOAR components with a git repository. It uses git's internal operations to write directly to the repository itself, essentially making it act as a file storage service. It provides methods to perform the following:

  • Migrate Assets between Google Security Operations SOAR instances

  • Backup Google Security Operations SOAR assets

  • Automatic Documentation

  • Creates a "store" to share assets/knowledge

  • Version Control

The integration consists of multiple Google Security Operations SOAR jobs - push and pull jobs for every supported asset, and push/pull jobs for the entire Google Security Operations SOAR instance. These jobs don't need to run periodically as they were built to run manually from the IDE, but they can be used as regular jobs (For example, uploading a daily commit). 

GitSync will use Google Security Operations SOAR API to fetch the relevant asset, such as an integration or a visual family, and parse all the available information from that asset (This information will later be rendered to a README.md file that is usually displayed when browsing the repository). It then writes the asset JSON definition and the rendered README to the local repository and pushes it to the remote repository.

Another usage of GitSync is knowledge sharing. Using this integration, a git repository can act as a "store" for assets such as playbooks or ontology settings that were designed before and leverages Google Security Operations SOAR best practices to push the platform to its best.

Prerequisites

Pushing/Pulling an existing repository:

  • Authentication method to Git. Supported is a Username/Password combination (not recommended), an access token (recommended), and a base64 encoded SSH private key (recommended). When using the latter two, the username parameter is not required.

  • A local Google Security Operations SOAR user. Used to import assets. This user must have permission to write the target module (For example, a user without access to the IDE would not be able to pull integrations).

Creating a new repository:

  • All of the points mentioned in the Pushing/Pulling an existing repository above.

  • A remote repository. It is recommended to have at least 1 file in the repository. Most Git services give an option to create a README file when creating the repository.

Integration Configuration

The integration doesn't need to be connected to an existing environment in Google Security Operations SOAR, so it's best to configure it as a Shared Instance.

Integration Properties

Param Name

Description

Repo URL

Repository URL. When using user/password authentication, this value must start with https://. If using SSH authentication, this value must start with git@ or ssh://. (See Configuring Repo URL and Branch below).

Branch

The branch in the repository to sync with.

Google Security Operations SOAR Username

Authentication method to Google Security Operations SOAR.

Google Security Operations SOAR password

Git Password/Token/SSH Key

Authentication method to git. This value can be Git Password/Token/SSH Private Key. Private keys should be encoded to base64. RSA and Ed25519 are supported.

Git Username

Git username. This value is not mandatory when using SSH authentication.

Commit Author

Not mandatory. Gives the ability to specify the author of the commit. This value must be in this format: Username

Google Security Operations SOAR Verify SSL

Verify SSL to Google Security Operations SOAR API

Git Verify SSL

Verify SSL with the target git service

Configuring Repo URL and Branch

In this guide, we will demonstrate getting the right values in Bitbucket (noe that the process is the same in GitHub.)


  1. Locate the repository in Bitbucket.

    gitsync1

  2. Click the clone button at the top right corner (Code in GitHub)

    • SSH Authentication - The repo url is git@bitbucket.org:siemplifyproserv/connectors.git

      gitsync2

  3. User/Pass or Token Authentication - The repo url is https://bitbucket.org/siemplifyproserv/connectors.git . (Username can be ignored)

    gitsync3

  4. Check the current branch (master in the image below)

    gitsync4

Example Usage

Every Job in GitSync contains the following parameters:


Name

Description

Job-specific - Connector name, Integration Identifier, playbook allowlist, etc.

These parameters are for specifying what gets pushed or pulled to the repository. In GitSync, assets are referred to by their identifiers. These values are case-sensitive.

Repo URL and Branch

Add support for multiple repositories with the same credentials. Once these parameters are set, the repository configured in the integration instance will be ignored. 

Commit Message

When pushing assets to the repository, a message is required for the commit. Here you can specify the reason for pushing, indicating what was fixed, changed or added to the asset.

Readme Addon

Adds the ability to extend the documentation of the assets when pushed. In this value you can use:

  • Markdown Syntax - Supported in README files by Git providers like GitHub and Bitbucket

  • Jinja - To display information about the asset. See examples in the integration manager constants

The template is added to the end of the documentation and is saved in the metadata file GitSync.json at the root of the repository.


Pulling Assets

In this example, we'll pull a connector with the correct mappings and visual families.

  1. First, ensure the asset is located in the configured repository. Simply browse the repository directories and copy the asset identifier (Usually it's the directory name or the title of the README file).
    gitsync5

    Example from a repo in Bitbucket, In the Connectors directory. Note that directories are the integration names, and inside them are the real identifiers for the connectors.

  2. Find the suitable job in Google Security Operations SOAR IDE. In this example, we'll use the job Pull Connector.

    • Note: When pulling a connector, verify that the integration of the connector is installed as well.

  3. Click on the testing tab and configure the parameters. Since we're using one repository and it's already configured in the integration instance, we'll leave the Repo URL and Branch parameters empty, and set the other parameters to the values we need.

  4. Run the job.

  5. See Debug Output for the log of the operation. If everything goes well, the log will indicate it.

  6. Go to Google Security Operations SOAR -> Connectors and configure the connector.


Pushing Assets

In this example, we'll push a playbook and a block to the repository.

  1. Identify the playbooks you want to push. Here we'll push a new block called Failed Login and an updated playbook called Malicious Activity.

    gitsync7

  2. Find the suitable job in Google Security Operations SOAR IDE. In this example, we'll use the job Push Playbook.

  3. Click the testing tab and configure the parameters.

  4. Since both are in the same folder (Default), You can also use Folder Allowlist instead.

  5. Run the job.

  6. See Debug Output for the log of the operation. If everything goes well, the log will indicate it.

  7. Validate that the repository contains the latest versions of the playbooks.


Creating a new repository

To create a new repository, only one thing is important, and that is including a single file in the repo before configuring it with GitSync. It can be done quickly by including a README file in the root of the repository when creating the repository.
Bitbucket

gitsync8

GitHub

gitsync9

Known issues and limitations

  • After the repository is set for the first time, It uses a predefined directory structure to ensure that it knows where each asset is located. Failing to follow the directory structure with a custom commit or changes to the repository, the integration will malfunction. You can find the schema of the repository directory structure at the end of this document.

  • Be careful when using this integration with public repositories. Google Security Operations SOAR assets use parameters that hold Application IDs, Client ID, Usernames, and other sensitive information. GitSync can't tell if the parameter is sensitive or not, so every parameter that is not of type "Password" will be uploaded to the repository. Also, when pushing a Google Security Operations SOAR instance (Push Environment job) there's an option to Commit Passwords. This option tells GitSync to try and export all the password parameters from the integration configuration. Don't set this value to true if the repository is public, or all the credentials will be leaked online.

  • When pulling a Google Security Operations SOAR instance (Pull Environment job), installing all the integrations might take more than 5 minutes, and the job will fail with a timeout. It is recommended to manually install all the commercial integration from the Google Security Operations Marketplace beforehand to avoid any issue, But it is also possible to re-run the job if it fails with a timeout.

  • Commercial integrations and custom integrations are handled differently. Custom integration will be pushed as the entire zip export of the integration, for a simple import/export operation. Commercial integrations will be pushed with only the custom code. Once pulled, GitSync will install the latest version of the integration from the Google Security Operations Marketplace, and save the custom code in the official integration.

  • When pulling mappings, they will not appear in the Settings -> Ontology -> Ontology Status table until the events have actually been ingested into Google Security Operations SOAR, because they are not indexed yet.

  • The local repository is saved in /opt/siemplify/siemplify_server/GitSyncFiles/{RepoName}. Since the integration writes Git objects and not files, this folder doesn't represent the repository and will be overwritten with any changes made anytime a job is executed. It is advised to use another clone of the repository, and not the one created by GitSync.

  • This power up is not supported on a remote agent.


Creating SSH Keys to use with GitSync

  1. First, generate a key pair. When asked for a passphrase, hit enter: ssh-keygen -b 2048 -t rsa -f ./id_rsa

    Two files will be created, id_rsa (private key) and id_rsa.pub (public key). Keep the private key in a safe location.

  2. Set the public key in the repository. For example, in Bitbucket, enter the settings of the repository and click on Access Keys. Click on Add Key and paste the contents of id_rsa.pub in the Key parameter.

  3. Before the private key can be added to the integration configuration, it should be encoded to Base64.

    Use these commands to encode the file:

    • Linux:
      cat id_rsa | base64 -w 0
    • Windows:

      Open PowerShell where id_rsa is located and paste (This is one line):

      Write-Output [System.Text.Encoding]::ASCII.GetString([convert]::ToBase64String(([IO.File]::ReadAllBytes((Join-Path (pwd) 'id_rsa')))))
  4. Copy the printed value to the integration property Password/Token/SSH Key and test the integration connectivity.

GitSync repository Directory Structure

The following is the expected directory structure in the remote repository.

* This is a tree command output for an example repo. Comments are in red.


gitsync10

gitsync11