Working with the Playbook Simulator

Supported in:

The Playbook Simulator provides you with a revolutionary way to develop Playbooks in less time and with less effort. Here are some of the standout features of the Simulator:

  • Allows you to work in a pre-production environment where you can test your actions and play with your results without affecting production data (if the Playbook is turned off).
  • Test each step of a Playbook or Playbook block to verify the required flow.
  • Test all branches of the Playbook conditions.

Using the Simulator on your playbook

  1. When using an existing playbook or creating a new one, your first step is to turn the Playbook Simulator on. The entire time the Simulator is on – you will have a green sign at the top of the screen. Note that if you work with both the Playbook simulator on an enabled Playbook then all incoming alerts that trigger the playbook will be affected. This is because if you save a playbook with simulation data that you have inserted, this will be used on a live Case in production and may skew the results.
  2. Navigate to the Cases screen, place your cursor on one of the alerts in the case, click on the more_vert  icon, and select Ingest alert as test case. This creates a test case in the system which you can then use to run your simulated Playbook on. Entities which are manipulated in test cases do not have any influence on entities in regular cases.
  3. Move your cursor to the bottom of the Playbook screen and select a test Case.
  4. Note that the test case you select needs to match the Playbook you run. A good way of checking that the Case and Playbook match is to click on the Entities button on the right and see that the entities in the test case can be processed by the Playbook. The screenshot below shows the entities in the test Case we selected in the step above.

    playbooksimulator

  5. Click Run. The Simulator starts processing the steps and running the Actions and providing the results.


How to understand the results of the Simulator for each step
After you click Run, the first row in the console will be the playbook trigger – and the simulator will show you success/failure of whether the specific trigger would be attached if it was a live Playbook in runtime.
For each step that is run in the Simulator, various options will be displayed, most commonly: Case Data, View Results, Pin Results (or Insert Results). For manual steps – a manual execution button will appear, allowing you enter the required parameters/responses and execute the step.


Case Data
On the far bottom right, you have the Case Data icon.

playbooksimulator2

The case data dialog shows the case information at a specific stage (after this specific action finished running). Case data dialog will be updated at playbook runtime with the current step results. If this step added enrichment to the case you will see it here. Because it shows information for the current state of the case (what the case looks like when the step finished running) – this data will be different for each step of the simulation console. Inspecting the case data of different steps will explain what was changed in the case between the different steps execution. There are different tabs you can click through which provide more information.


View Results
Next option displayed is "View Results" – this will show you the action results of the current step. The information displayed here is similar to case overview / case wall – view results and also includes information on enrichments. You can see output message, tables, links, attachments in the main tab, and you can see action result, and JSON result (where present) in the technical details tab.
As with the previous option, there is also an option to tab through data. Another action you can take from the View Results screen is to click Set JSON result.
"Set JSON result" will replace the JSON sample of this action. The JSON sample can be changed from the IDE as well, and can be used within the Expression Builder to extract data from the JSON result.

Pin Results
This will be displayed if a step ran successfully.
Pin Results is a very useful option which allows you to take the result of the action and consider it as "fixed". This can potentially save you time by not needing to wait for third party results and save credentials by reducing the number of queries to the 3rd party
In other words, when you rerun the Playbook again, this step is "passed over", the code is not run, and the pinned results are taken as is. You also have the option here to manipulate the outcome and put in your own mock data. The screenshot below shows the Pin Results screen for this step. As you can see the Action has the simulate mode enabled and the step has changed from a blue background to a gray background to indicate that the Simulate mode is on. With regards to adding simulation data in to this screen, we will go into this in more detail in the next section. (Chronicle SOAR actions will not have the option to pin results as there is no option to run them in simulate mode) Note that enrichments are not automatically added and need to be added manually in the Enrichment tabs.

Insert Results
This will be displayed if a step failed.
Insert Results allows you to manually insert simulation data so that the next time you run this step, it will return the simulation data as the result. Once you click on this option, the action has the simulate mode enabled and the step changes from a blue background to a gray background to indicate that the Simulate mode is on. Note that "Script Result" is a mandatory field for all steps in simulation mode.
Why would I want to insert simulation (mock) data?
There are several reasons why you might want to insert mock data and we will discuss them briefly here.
  • I want to build and test my playbook on the go. In this case I can run a step, view the results and understand how I can us it further along in the playbook.
  • After a successful run, I want to pin the step results and save time by not running it again and again against third parties APIs
  • I want to change my step results to test my playbook in different scenarios. In this case I'll set different simulation data to influence conditions and actions that are using previous results. For example, if you have a Playbook with a condition that branches off to two or more branches. You can play around with the simulation data in order to "force" the playbook to take a different branch each time and hence evaluate the entire Playbook.

How do I insert simulation data?

There are several ways of inserting simulation (mock) data: via Step configuration dialog in a playbook or via Pin Results (or Insert Results) after a Simulator run.

Playbook Step configuration dialog
  1. Click on the step configuration dialog in the Playbook and toggle the Simulate mode to enable. The Action will be displayed as grey.
  2. In the Action results, insert your simulation (mock) data – such as script result, JSON result by taking specific data from the JSON code. You can use enrichments from previous simulation runs or create custom enrichment keys of your own. Additionally, the JSON result can be loaded from the sample output by clicking the Load Sample button. The Load Sample button will load the action's Sample Output (a JSON result sample which represents the result that you would expect the action to return) as the simulated step JSON result. This option is particularly useful if:
    • The simulation was never run, and the output is empty.
    • The simulation failed and there are no results to be displayed.
    • There are results displayed (from either previously pinning the results or inserting the results), but we want to override it with the sample.
    playbooksimulator3
  3. This data will now be returned every time the step runs.
Pin Results
  1. Click on Pin Results next to the Action. The Step will open up already in Simulate mode.
  2. The results from the latest simulation run will be pinned to the step. These pinned results can be used as is or they can be edited. The JSON result can be edited using the JSON editor or it can be overridden by clicking on the Load Sample button. You can use enrichments from previous simulation runs or create custom enrichment keys of your own.
  3. This data will now be returned every time the step runs.
Insert Results
  1. Click on Insert Results next to the Action.
  2. In the Action results, insert your simulation (mock) data – such as script result, JSON result by taking specific data from the JSON code. Additionally, the JSON result can be loaded from the sample by clicking the Load Sample button. You can use enrichments from previous simulation runs or create custom enrichment keys of your own.
  3. This data will now be returned every time the step runs.
Turning off the Playbook Simulator

Once you turn off the Playbook Simulator, the bottom Console is hidden and any step that might be in simulate mode will be reverted back to regular "live" mode. One exception to this rule is the Playbook Blocks – you need to turn off the block Simulator itself in order to close the simulation mode for it.
Note that any simulation mock data that you have inserted will be saved for you to use the next time you turn on the Simulator.

Working with Playbook blocks

You can also use the Playbook Simulator when building a new Playbook block. Note that when a block is in simulation mode – all the playbooks using this block will also be using the block's mock data.