Storm Reference - Automation

Background

Synapse supports large-scale analysis over a broad range of data with speed and efficiency. Many features that support this analysis are built into Synapse’s architecture, from performance-optimized indexing and storage to an extensible data model that allows you to reason over data in a structured manner.

Synapse also supports large-scale analysis through the use of automation. Synapse’s automation features include:

Triggers and Cron
Macros
Dmons

Automation frees analysts from performing tedious work and allows them to focus on more detailed analysis and complex tasks. You can also scale analytical operations by limiting the amount of work that must be performed manually.

Automation in Synapse uses the Storm query language: anything that can be written in Storm can be automated, from the simple to the more advanced. Actions performed through automation are limited only by imagination and Storm proficiency. Automation can be basic: “if X occurs, do Y” or “once a week, update Z”. However, automation can also take advantage of all available Storm features (including subqueries, variables, libraries, and control flow logic), to support highly customized tasks and workflows.

Considerations

The following items should be taken into account when planning the use of automation in your environment.

Permissions

Permissions impact the use of automation in Synapse in various ways. In some cases, you must explicitly grant permission for users to create and manage automation. In other cases, the permissions that an automated task runs under may vary based on the type of automation used. See the relevant sections below for additional detail.

Tip

For a detailed discussion of permissions in Synapse, refer to the Synapse Admin Guide.

Scope

Automation components vary with respect to where they reside and execute within Synapse; some elements are global (within a Cortex) while some reside and execute within a specific View. Organizations that leverage multiple views for their Synapse architecture or that make use of Synapse’s fork and merge capabilities should refer to the sections below for information on how views and layers may impact automation.

Tip

For a more general discussion of views and layers, including a discussion of forking and merging views, see the Views and Layers section of the Synapse User Guide.

Testing

Always test your automation before putting it into production. Storm used in automation can be syntactically correct (that is, the query uses proper Storm), but contain logical errors (the query does not actually do what you want it to do). In addition, new automation may interact with existing automation in unexpected ways. Test your automation in a development environment (either a separate development instance, or a separate fork of your production view) before implementing it in production.

Use Cases

Organizations can implement automation as they see fit. Some automation may be enterprise-wide, used to support an organization’s overall mission or analysis efforts. Other automation may be put in place by individual analysts to support their own research efforts, either on an ongoing or temporary basis.

Design

There are varying approaches for “how” to write and implement automation. For example:

Location of automation code. The Storm code run by individual triggers and cron jobs can be written and stored as part of the automation itself. This approach helps keep automation “self-contained”. However, it may provide less flexibility in executing the associated Storm compared with the use of macros.

Alternatively, tasks such as triggers and cron jobs can be written to execute minimal Storm queries whose purpose is to call more extensive Storm stored centrally in macros. This approach consolidates much of the associated Storm, which may make it easier to manage and maintain. Storm placed in macros also provides flexibility as the macro can be called by a trigger, a cron job, or a user as part of a Storm query.
Size of automation. Automation can be written as many small, individual elements. Each element can perform a relatively simple task, but the elements can work together like building blocks to orchestrate larger-scale operations. This approach keeps tasks “bite sized” and the Storm executed by a given piece of automation generally simpler. However it may result in a larger number of automation elements to maintain, and may make it more challenging to understand the potential interactions of so many different elements.

Alternatively, automation can be implemented using fewer elements that perform larger, more unified tasks (or that consolidate numerous smaller tasks into a larger set of Storm code). This approach results in fewer automation elements overall, but typically requires you to write and maintain more advanced Storm (e.g., to create a small number of macros with switch or if/else statements to each manage a variety of tasks). However, the Storm is consolidated in a few locations, which may make managing and troubleshooting easier.

Each approach has its pros and cons; there is no single “right” way. In addition, you do not have to take an “either / or” approach; what works best in your environment or for a particular task will depend on your needs (and possibly some trial and error).

Governance / Management

Consider any review or approval processes that you may need in order to manage automation effectively in your environment. Organization-wide automation requires coordination and oversight:

If multiple users have the ability to create automated tasks, it is possible for them to create duplicative or even conflicting automation. Consider who should be responsible for deconflicting and approving automation that will be deployed in your production environment.
Automation is often used to enrich indicators (i.e., query various third-party APIs to pull in more data related to a node). Some third-party APIs may impose query limits, may be subject to a license or subscription fee, or both. Consider how to balance effective use of automation without overusing or exceeding any applicable quotas.
Some automation may be used to apply tags to nodes or “push” tags from one node to related nodes - effectively automating the process of making an analytical assertion. Consider carefully under what circumstances this should be automated, and who should review or approve the analysis logic used to make the assertion.

Automation and Error Handling

If the Storm executed by a piece of automation encounters an error condition, the automation will cease execution and exit. For automation that operates over large numbers of nodes or performs a long-running task, an unexpected error can cause the automation to halt in the middle and fail to complete.

In addition to general testing of your Storm code (and Storm logic!), using the “Try” Operator when adding or modifying data will ensure your Storm code will “warn and continue” if it encounters bad data vs. generating an error and halting.

We also encourage you to build additional error-checking and error-handling into your automation as appropriate.

Users should keep the Storm Operating Concepts in mind when writing automation. Automation frequently operates on nodes or other Synapse data; knowing what nodes are in the Storm pipeline for your automation will help significantly with troubleshooting any issues.

Triggers and Cron

Triggers and cron are similar in terms of how they are implemented and managed.

Permissions. Synapse uses permissions to determine who can create, modify, and delete triggers and cron jobs. These permissions must be explicitly granted to users and/or roles. See the Cortex permissions section of the Synapse Admin Guide for a list of cron.* and trigger.* permissions.
Execution. Both triggers and cron jobs execute with the permissions of a designated user associated with the individual trigger or cron job. By default, this is the user who creates the trigger or cron job. The user can be changed (e.g., for organizations that wish to use a dedicated account for automation tasks) using the set() method of the trigger or cronjob primitives, respectively.
Scope. Both triggers and cron jobs run within a specific view; any changes made to Synapse’s data by the trigger or cron job are made (written) to the topmost (writeable) layer of the view. This view-specific behavior is transparent when using a simple Synapse implementation consisting of a single Cortex with a single layer and a single view (Synapse’s default configuration). Organizations using multiple views or that frequently fork views should consider the impact of your view architecture on automation deployment and behavior.

Note

The use of sensitive information (e.g., credentials, API keys, etc.) in the Storm that is executed by a trigger or cron job is strongly discouraged. If your automation needs to make use of any sensitive information, we recommend creating a Power-Up with associated Storm commands. The Power-Up architecture allows users to run commands that make use of sensitive information while ensuring that information is not exposed. See the Rapid Power-Up Development section of the Synapse Developer Guide for details.

Cron

Cron in Synapse is similar to the well-known cron utility. Cron jobs execute their associated Storm on a specified schedule. Jobs can be written to execute once (using the cron.at command) or on a recurring basis (using cron.add).

Tip

When scheduling cron jobs, Synapse interprets all times as UTC.

Configuration and Management

Storage. Cron jobs are stored globally (within the Cortex). When viewing cron jobs, (e.g., with the cron.list command), Synapse returns all cron jobs in the Cortex, regardless of the view the cron.list command is executed in.
Execution. Cron jobs must be assigned to a specific view where they execute. By default, this is the view where the cron job is created. If the view that a cron job runs in is deleted, the cron job remains (within the Cortex) but is effectively orphaned until it is assigned to a new view (i.e., using the $lib.cron.move(prefix, view) library) or deleted if no longer needed.
Permissions. Cron jobs execute with the privileges of a designated user (by default, the user who creates the cron job). We strongly encourage the use of least privilege; the cron job’s account should have the permissions required to execute the associated Storm, but no more. One option is for organizations to create a dedicated account for use with automation in Synapse.

The owner (creator) of a cron job can be modified using the Storm $lib.cron.get(prefix) library and the set() method of the cronjob primitive. For example:
```
$mycron=$lib.cron.get(<cron_iden>) $mycron.set(creator, <new_creator_iden>)
```
Users (and roles) cannot create or manage cron jobs by default. The various cron.* permissions must be granted to users or roles that should be allowed to work with cron jobs in your environment. See the Cortex permissions section of the Synapse Admin Guide for details on the cron.* permissions and associated gates.

Note

Where a user has admin privileges, all permissions checks are bypassed. This means that users can create and manage cron jobs in views that they fork.
Managing Cron Jobs. Cron jobs can be created, viewed, and managed using the various Storm cron commands, the cronjob primitive, or the $lib.cron libraries. In Optic, cron jobs can also be created and managed using the Admin Tool.

Use Cases

Because cron jobs are scheduled, they are most appropriate for automating routine tasks, non-urgent tasks, or resource-intensive tasks that should be scheduled to minimize impact on operations.

“What” a cron job does is limited only by your imagination (and your Storm skills). Examples of common cron use cases include:

Data ingest. Periodically ingest / synchronize data that you want to load into Synapse on a regular basis.
Housekeeping. Perform one-time or periodic “maintenance” tasks. These may include a one-time sweep to “backfill” missing data (like IP geolocation data) or a periodic sweep to check and set missing properties (such as tag definitions).
Process intensive jobs. Data enrichment may be resource intensive if it generates a significant number of write operations. If you regularly perform routine (non-urgent) enrichment, it can be scheduled to run when it will have less impact on users.
Periodic hunting. New data and annotations (tags) are continually added to Synapse. Encoding “hunt” logic in Storm and periodically running hunts can help ensure you are continually reviewing new data for updated indicators or activity.

Tip

A cron job can use Storm to call a macro, which may provide greater flexibility in storing and managing the Storm code executed by the cron job.

Syntax

In Storm, cron jobs are primarily created and managed using the Storm cron commands.

Once a cron job is created, you can modify many of its properties (such as its name and description, or the Storm associated with the job). However, you cannot modify other aspects of the job, such as its schedule. To change those conditions, you must disable (or delete) and re-create the cron job.

Variables

Every cron job has an associated Storm variable $auto that is automatically populated when the job runs. The $auto variable is a dictionary which contains the following keys:

$auto.iden
The identifier of the cron job.

$auto.type
The type of automation. For a cron job this value will be cron.

These job-specific variables can be referenced by the Storm code executed by the cron job. See Storm Reference - Advanced - Variables for information on the use of variables in Storm.

Examples

The examples below illustrate using the Storm cron commands (cron.at, cron.add) to create new cron jobs. See the help for each command for options and additional details.

One-time (non-recurring) cron job example

You want to use a one-time cron job that runs during off hours to perform some data cleanup. Specifically, you want a job that will lift all existing media:news nodes and remove (delete) the deprecated :author property at 0200 UTC.

storm> cron.at --hour 2 { media:news:author [ -:author ] }
Created cron job: b2cbe353dedb604fe18295791b97bb6b

We can view the details of this cron job using the cron.list command:

storm> cron.list
 user                      iden        view        en?  rpt?  now?  err?  # start  last start        last end          query
#############################################################################################################################
 root                      b2cbe353..  8ab94c5c..  Y    N     N           0        Never             Never             media:news:author [ -:author ]

The output of cron.list includes the following columns:

user - the user that the job runs as (typically the user who created the job).
iden - the first eight characters of the cron job’s identifier (iden).
view - the first eight characters of the iden of the view the job executes in. For “orphaned” cron jobs, this will be the job’s last view (before it was orphaned).
en? - whether the job is currently enabled or disabled ( Y/N ).
rpt? - whether the job is scheduled to repeat ( Y/N ).
now? - whether the job is currently executing ( Y/N ).
err? - whether the last job execution encountered an error ( X or empty ).
# start - the number of times the job has started / executed (since the last restart of Synapse).
last start - the date and time the job last started (or attempted to start).
last end - the date and time the job last finished (or exited).
query - the Storm query executed by the cron job.

Tip

Detailed information about individual cron jobs can be retrieved with the cron.stat command.

Hourly cron job example

You want to create a cron job that will run every hour to download the latest MISP data using the misp.sync command.

cron.add --hour +1 { misp.sync }

Tip

The misp.sync command is added by the Synapse-MISP Power-Up.

Custom schedule example

Because Synapse is continually updated with new data, you want to periodically check for any new files of interest that have not yet been identified by an analyst. Specifically, you want to look for files (file:bytes nodes) that make DNS queries (inet:dns:request) for “known bad” FQDNs (e.g., tagged #cno.mal) where the files have not yet been tagged. You want to run the job three times a week (every Tuesday, Thursday, and Saturday at 2000 UTC) and tag the files for review (#int.review.malware).

cron.add --day Tue,Thu,Sat --hour 20 { inet:fqdn#cno.mal -> inet:dns:request -> file:bytes -#cno.mal [ +#int.review.malware ] }

Triggers

Triggers are “event-driven” automation. As their name implies, they trigger (“fire”) their associated Storm when specific events occur in Synapse’s data store. Triggers can fire on the following events:

Adding a node (node:add)
Deleting a node (node:del)
Setting (or modifying) a property (prop:set)
Adding a tag to a node (tag:add)
Deleting a tag from a node (tag:del)
Adding a light edge (edge:add)
Deleting a light edge (edge:del).

When creating a trigger, you must specify the type of event the trigger should fire on. In addition, each event requires a specific object (a form, property, tag, or edge) to act upon. For example, if you write a trigger to fire on a node:add event, you must specify the type of node (form) associated with the event; you cannot create a node:add trigger that fires on “any / all nodes”. For tag:* and edge:* triggers, you have the option to limit the trigger to tags or edges associated with specific forms, or have the trigger apply to any / all forms.

Note

The node(s) that cause a trigger to fire are considered inbound to the Storm code executed by the trigger.

Proper trigger execution may depend on the timing and order of events with respect to creating nodes, setting properties, and so on. For example, you may write a trigger based on a node:add action that fails to perform as expected because you actually need the trigger to fire on a prop:set operation for that node. As always, we recommend that you test triggers (and any other automation) before putting them into production.

Configuration and Management

Storage. Triggers are stored within a view. If the view that a trigger resides in is deleted, the trigger is also deleted. Triggers can be moved to another view using the move() method of the trigger primitive. (In Optic, when using the View Task Bar to merge a view, you are prompted to also merge any triggers in the view.)

Because triggers are stored within individual views, when viewing triggers (e.g., with the trigger.list command), Synapse returns only those triggers within the current view.
Execution. Triggers “fire” when a change occurs within Synapse - that is, when a write operation occurs. As triggers reside (and execute) within a view, they fire on changes to the view’s writable (topmost) layer. A trigger will not fire on changes made in an underlying layer (i.e., of a parent view).

Similarly, a trigger that resides in a parent view will not fire on changes made in a fork of that view. However, when data from the forked view is merged into the parent view, merging (writing) the data will cause any relevant triggers to fire.

Triggers fire immediately when their associated event occurs; but they only execute when that event occurs. If the Storm executed by the trigger depends on a resource (process, service, etc.) that is not available when the trigger fires, execution will simply fail; Synapse will not “try again” later. (Whether the trigger fails silently or by logging an error will depend on the Storm and / or any additional resources used or called by the trigger.)

Similarly, triggers do not operate “retroactively” on existing data. If you write a new trigger to fire when the tag my.tag is applied to a hash:md5 node, the trigger will have no effect on existing hash:md5 nodes that already have the tag.
Permissions. Triggers execute with the privileges of a designated user (by default, the user who creates the trigger). We strongly encourage the use of least privilege; the trigger’s account should have the permissions required to execute the associated Storm, but no more. One option is for organizations to create a dedicated account for use with automation in Synapse.

The owner (user) of a trigger can be modified using the Storm $lib.trigger.get(iden) library and the set() method of the trigger primitive. For example:
```
$mytrigger=$lib.trigger.get(<trigger_iden>) $mytrigger.set(user, <new_user_iden>)
```
Triggers run as a specified user, but they execute based on particular changes to data in Synapse. This means that a lower-privileged user could make a change (such as creating a node) that causes a higher-privileged trigger to execute and perform actions that the user would not be able to do themselves.

Users (and roles) cannot create or manage triggers by default. The various trigger.* permissions must be granted to users or roles that should be allowed to work with triggers in your environment. See the Cortex permissions section of the Synapse Admin Guide for details on the trigger.* permissions and associated gates.

Note

Where a user has admin privileges, all permissions checks are bypassed. This means that users can create and manage triggers in views that they fork.
Managing Triggers. Triggers can be created, viewed, and managed using the various Storm trigger commands, the trigger primitive, or the $lib.trigger libraries. In Optic, triggers can be created and managed using the Admin Tool or the VIEWS tab of the Workspaces Tool.

Use Cases

Triggers are “event driven” and execute their Storm immediately when their associated event (change) occurs. As such, triggers are most appropriate for automating tasks that should occur right away (e.g., based on efficiency or importance). Example use cases for triggers include:

Performing enrichment. There may be circumstances where you always want to retrieve additional information about an object (node) within Synapse. For example, whenever a unicast IPv4 (inet:ipv4 node) is added to Synapse, you want to automatically look up Autonomous System (AS), geolocation, and network whois data. You can use a trigger to enrich the inet:ipv4 using the appropriate Power-Ups as soon as the IPv4 is created (e.g., by firing on a node:add event).

Similarly, when a node is assessed to be malicious (e.g., associated with a threat cluster or malware family, and tagged appropriately), you may wish to immediately collect additional information about that node. A trigger that fires on a tag:add event could be used to call multiple Power-Ups using a more extensive “enrichment” query (or macro).
Encoding analytical logic. You can use Storm to encode your analysis logic, such as the criteria or decision process used to apply a tag. As a simplified example, assume you have identified an IPv4 address as a DNS sinkhole. You assess that any FQDN resolving to the IPv4 is highly likely to be a sinkholed domain. When a DNS A node (inet:dns:a) is created where the associated IPv4 (:ipv4 property) is the IP of the sinkhole (a prop:set event), a trigger can automatically tag the associated FQDN as sinkholed. If you want an analyst to confirm the assessment (vs. applying it in a fully automated fashion), you can apply a “review” tag instead. Alternatively, if additional criteria would better support your assessment (e.g., if the DNS A resolution occurred during a particular time window when the sinkhole IP was known to be in use, or if the resolution was based on a live/active DNS lookup vs. a passively observed response), you could modify your trigger’s Storm to reflect these additional “checks” or criteria.

You can similarly encode logic in Storm to support retrohunting, threat clustering, or other workflows.
Automating repetitive tasks. Any process that analysts identify as repetitive may benefit from automation. Analysts may identify cases where, when they perform a specific action in Synapse, they always perform several additional actions. For example, when an analyst tags a particular node (such as a file:bytes node), they always want to apply the same tag to a set of “related” nodes (such as the hash:md5, hash:sha1, etc. that represent the file’s hashes). Similarly, if a file:bytes node queries a “known bad” FQDN (via an inet:dns:request node), analysts also want to apply the tag from the FQDN to both the DNS request and the file. Using a trigger (on a tag:add event) saves manual work by the analyst and ensures the additional tags are applied consistently.

Tip

A trigger can use Storm to call a macro, which may provide greater flexibility in storing and managing the Storm code executed by the trigger.

Syntax

In Storm, triggers are created, modified, viewed, enabled, disabled, and deleted using the Storm trigger commands. In Optic, triggers can also be managed through the Admin Tool or the Workspaces Tool.

Once a trigger is created, you can modify many of its properties (such as its name and description, or the Storm associated with the trigger). However, you cannot modify the conditions that cause the trigger to fire. To change those conditions, you must disable (or delete) and re-create the trigger.

Variables

Every trigger has an associated Storm variable $auto that is automatically populated when the trigger executes. The $auto variable is a dictionary which contains the following keys:

$auto.iden
The identifier of the Trigger.

$auto.type
The type of automation. For a trigger this value will be trigger.

$auto.opts
Dictionary containing trigger-specific runtime information. This includes the following keys:

$auto.opts.form
The form of the triggering node.

$auto.opts.propfull
The full name of the property that was set on the node. Only present on prop:set triggers.

$auto.opts.propname
The relative name of the property that was set on the node. Does not include a leading :. Only present on prop:set triggers.

$auto.opts.tag
The tag which caused the trigger to fire. Only present on tag:add and tag:del triggers.

$auto.opts.valu
The value of the triggering node.

$auto.opts.verb
The name of the light edge. Only present on edge:add and edge:del triggers.

$auto.opts.n2iden
The iden of the node on the other end of the edge. Only present on edge:add and edge:del triggers.

These trigger-specific variables can be referenced by the Storm code executed by the trigger. See Storm Reference - Advanced - Variables for information on the use of variables in Storm.

Examples

The examples below illustrate using the Storm trigger.add command to create new triggers. See the command help for options and additional details.

Recall from the Storm Operating Concepts that Storm operations are “chained” together to act as a pipeline through which nodes are “sent” and acted upon. Regardless of the type of trigger (prop:set, tag:add), the node associated with the trigger event is sent inbound to the Storm executed by the trigger.

Tip

By default, the Storm code executed by a trigger runs inline. In other words, when a process (typically a Storm operation or Storm query) causes a trigger to fire, the Storm associated with the trigger will run immediately and in full. Conceptually, it is as though all of the trigger’s Storm code and any additional Storm that the trigger calls (such as a macro) or causes to run (such as another trigger) are inserted into the middle of the original Storm query that fired the trigger, and executed as part of that query.

This inline execution can impact the original query’s performance, depending on the Storm executed by the trigger and the number of nodes causing the trigger to fire. Specifically, if a trigger (and associated Storm) is fired based on an interactive Storm query from a user, Synapse’s interface (the Storm CLI or the Optic UI) will “block” until the Storm executes in full.

The --async option can be used when creating a trigger to specify that the trigger should run in the background as opposed to inline. This will cause the trigger event to be stored in a persistent queue, which will then be consumed automatically by the Cortex.

Executing the trigger asynchronously means that changes made by the trigger (e.g., such as data enrichment) will not be available to the user until the associated Storm finishes running. However, the user can continue working in the meantime because the Synapse interface will not “block”.

To change whether or not an existing trigger runs asynchronously, use the Storm $lib.trigger.get(iden) library and the set() method of the trigger primitive. For example:

$mytrigger=$lib.trigger.get(<trigger_iden>) $mytrigger.set(async, $lib.true)

Or:

$mytrigger=$lib.trigger.get(<trigger_iden>) $mytrigger.set(async, $lib.false)

prop:set example

You have identified a handful of email addresses that are consistently used by various sinkhole organizations to register their sinkholed FQDNs, and tagged the email addresses #cno.infra.dns.sink.hole to show they are associated with sinkhole infrastructure. You want to add a trigger that will fire when any of these email addresses is set as the :email property of an inet:whois:contact node and tag the associated FQDN as sinkholed (#cno.infra.dns.sink.holed). Because the trigger is a simple one, you want it to run inline (i.e., you do not need the --async option).

storm> trigger.add prop:set --name 'Tag sinkholed FQDNs based on known sinkholer email' --prop inet:whois:contact:email --query { +{ :email-> inet:email +#cno.infra.dns.sink.hole } -> inet:whois:rec -> inet:fqdn [ +#cno.infra.dns.sink.holed ] }
Added trigger: f7aacb861db278429c7e0bee5fe314c3

Tip

For prop:set and prop:del triggers, the node whose property caused the trigger to fire is sent inbound to the trigger’s Storm.

We can view the newly created trigger using trigger.list:

storm> trigger.list
user       iden                             view                             en?    async? cond      object                    storm query
root       f7aacb861db278429c7e0bee5fe314c3 8ab94c5cebc56365860f61987e3ef031 true   false  prop:set  inet:whois:contact:email            +{ :email-> inet:email +#cno.infra.dns.sink.hole } -> inet:whois:rec -> inet:fqdn [ +#cno.infra.dns.sink.holed ]

The output of trigger.list includes the following columns:

user - the user that the trigger runs as (typically the user who created the trigger).
iden - the trigger’s identifier (iden).
view - the iden of the view in which the trigger resides / runs.
en? - whether the trigger is currently enabled or disabled.
async? - whether the trigger will run asynchronously / in the background.
cond - the condition that causes the trigger to fire.
object - the object the trigger operates on (in this case, the :email property of an inet:whois:contact node). May reference multiple objects (e.g., if a tag:add trigger only fires when the tag is added to a specific form, both the tag and form will be listed).
storm query - the Storm to be executed when the trigger fires.

node:add example

Whenever an IPv4 node is added to Synapse, you want to immediately retrieve the associated Autonomous System (AS), geolocation, and DNS PTR data using various Storm commands.

storm> trigger.add node:add --name 'Basic IPv4 enrichment' --form inet:ipv4 --query { maxmind | nettools.dns }
Added trigger: ea8575640b7711b1a57c65c237d95c82

Tip

For node:add and node:del triggers, the node that caused the trigger to fire is sent inbound to the trigger’s Storm.

The maxmind and nettools.dns commands are added to Synapse by the Synapse-Maxmind and Synapse-Nettools Power-Ups, respectively.

edge:add example

You are using risk:attack nodes in Synapse to represent attacks, and linking the attack to key objects used in the attack with a -(uses)> light edge. (For example, a risk:attack representing a phishing attempt can be linked to the relevant email message (file:bytes node) or email metadata nodes (such as inet:email:message, inet:email:message:attachment, or inet:email:message:link).

If the object “used” in the attack has an associated TTP tag (such as #cno.ttp.phish.attach to represent a malicious attachment used in a phishing attack), you want to pivot to the technique (ou:technique) represented by the tag (e.g., “Spear phishing attachment”), and create another uses edge to show the attack uses the specified technique.

storm> trigger.add edge:add --name 'Link technique with attack'--verb uses --form risk:attack --query { $attack=$node { -(uses)> * +$auto.opts.n2iden -> # -> ou:technique:tag [ <(uses)+ { yield $attack } ] } }
Added trigger: a125d14bb16218d59a0aac8bee56f617

Tip

Recall that edges have a direction, regardless of whether the edge relationship is specified “left to right” or “right to left” in Storm (e.g., n1 -(uses)> n2 or n2 <(uses)- n1).

For edge:add and edge:del triggers, the n1 node (source node) for the edge that caused the trigger to fire is sent inbound to the trigger’s Storm.

The Storm for this trigger is broken down below (with comments) for clarity:

// Set the variable $attack to the risk:attack node for later use
$attack=$node

// Use a subquery to traverse all 'uses' edges to any / all forms
{
   -(uses)> *

   // Filter to the n2 node that was just linked, as referenced by the
   // trigger-specific $auto.opts.n2iden variable
   +$auto.opts.n2iden

   // Pivot to the syn:tag nodes associated with any tags on n2
   -> #

   // Pivot to any techniques associated with any tags
   -> ou:technique:tag

   // Create a 'uses' edge between the technique(s) and the risk:attack node
   [ <(uses)+ { yield $attack } ]
}

tag:add example

When you associate a file with a malware family, you record that assessment on the file:bytes node with a tag (e.g., #cno.mal.redtree to represent the ‘redtree’ malware family). When you tag the file, you want to automatically copy the tag to the file’s MD5, SHA1, SHA256, and SHA512 hashes for situational awareness.

storm> trigger.add tag:add --name 'Push malware tags from file to hashes' --form file:bytes --tag cno.mal.** --query { tee { :md5 -> hash:md5 } { :sha1 -> hash:sha1 } { :sha256 -> hash:sha256 } { :sha512 -> hash:sha512 } | [ +#$auto.opts.tag ] }
Added trigger: 465a566b3d3ee3263108fc92474e593f

Tip

For tag:add and tag:del triggers, the node that was tagged (or untagged) to cause the trigger to fire is sent inbound to the trigger’s Storm.

The trigger above uses the trigger-specific $auto.opts.tag variable to reference the tag that caused the trigger to fire.

tag:del example

Similar to the example above, when your assessment changes and you want to remove a tag from a file (file:bytes node), you want to automatically remove that same tag from the file’s associated hashes.

storm> trigger.add tag:add --name 'Untag hashes when untagging a file' --form file:bytes --tag cno.mal.** --query { tee { :md5 -> hash:md5 } { :sha1 -> hash:sha1 } { :sha256 -> hash:sha256 } { :sha512 -> hash:sha512 } | [ -#$auto.opts.tag ] }
Added trigger: c899239245844fb49ba8e26b154602aa

Tip

The trigger above uses the trigger-specific $auto.opts.tag variable to reference the tag that caused the trigger to fire.

Macros

A macro is a stored Storm query / set of Storm code that can be executed on demand. Strictly speaking, macros are not automation - they do not execute on their own. However, they do provide a means to “automate” tasks by allowing an analyst to easily and consistently run Storm that would otherwise need to be manually entered as a query. Anything that assists an analyst’s workflow - from executing a commonly used pivot to performing a detailed set of data enrichment and operations - is a candidate for a macro if it simplifies analysts’ work.

Tip

In addition to macros, Optic includes features such as Node Actions and Bookmarks that can be used to save and execute Storm on demand.

Macros are also commonly used with (called by) triggers or cron jobs. In particular, where a trigger or cron job executes longer or more detailed Storm, it may be easier to store and maintain that Storm within a macro instead of within the trigger or cron job itself.

As a stored Storm query, a macro is meant to operate on nodes. You must either specify (lift) the nodes to be operated on within the macro itself, or provide the nodes as input to the macro (i.e., by sending the results of an existing Storm query to the macro.exec command that runs the macro).

A macro will error if it receives nodes that cannot be processed by the associated Storm code. A “best practice” when writing macros that take nodes as input is to include a Storm filter operation within the macro to allow only those nodes that the macro expects and can operate on; any nodes not allowed by the filter are dropped from the Storm pipeline.

Similarly, if you execute additional Storm inline after the macro runs, that Storm must be appropriate for any nodes that exit the macro.

Note

The use of sensitive information (e.g., credentials, API keys, etc.) in the Storm that is executed by a macro is strongly discouraged. If your automation needs to make use of any sensitive information, we recommend creating a Power-Up with associated Storm commands. The Power-Up architecture allows users to run commands that make use of sensitive information while ensuring that information is not exposed. See the Rapid Power-Up Development section of the Synapse Developer Guide for details.

Configuration and Management

Storage. Macros are stored within (global to) a Cortex. Macros are differentiated by name (as opposed to triggers and cron jobs, which are differentiated by a unique identifier (iden)). You can change the name of a macro using the $lib.macro.mod(name, info) library:
```
$lib.macro.mod('my old poorly chosen macro name',({'name': 'new.name'}))
```
Attempting to rename a macro to a name that already exists will result in a DupName error.
Execution. Macros are not restricted to a particular view; they can be called from and executed within any view. Any changes made by the macro’s Storm code will be made to the topmost / writeable layer of the view in which the macro is executed.
Permissions. Macros execute with the privileges of the user who calls the macro. If a macro is called by a trigger or cron job, the macro executes with the privileges of the associated trigger or cron user.

By default:
- Any user can create a macro; you do not need to explicitly grant permissions for users to create them.
- The user who creates a macro is the owner / admin of the macro, and is the only user who can edit, delete, or modify permissions on the macro.
- All users can see (read) and execute any macro. If a user attempts to execute a macro that performs actions for which the user does not have permissions, the macro will fail with an AuthDeny error.
Tip

Macro permissions can be modified / managed by using the $lib.macro.grant(name, scope, iden, level) library. (In Optic, individual macro permissions can be managed through the Storm Editor Tool.) Macros support Synapse’s easy permissions (“easy perms”) convention for setting simplified permissions on objects. You must be admin of a macro to modify its permissions. See the Examples section below for examples of modifying macro permissions.
Managing Macros. Macros can be created, viewed, and managed using the various Storm macro commands or the $lib.macro libraries. In Optic, macros can be created and edited using the Storm Editor Tool.

Use Cases

Macros are a convenient way to save and run frequently used Storm without having to create or type that Storm each time. The Storm can be as simple or advanced as you like.

Organizational use. Macros can be developed for use across entire teams or organizations to support common tasks or workflows such as enrichment or threat hunting. Using a macro makes it easier to perform the task (by calling it with a single Storm command) and also ensures that the task is performed consistently (i.e., in the same way each time) by each user.
Personal use. Users can create macros to store frequently-used or lengthy Storm queries specific to their personal workflow that can be executed easily on demand.
Automation. For triggers or cron jobs that execute longer Storm queries, saving the Storm in a macro may make it easier to set, view, edit, and manage vs. storing the Storm directly as part of the trigger or cron job.
Flexibility. Because macros are composed in Storm and executed via a Storm command, they can be executed any way Storm can be executed (e.g., on demand or called as part of a trigger or cron job). Macros are ideal for Storm that performs a task or set of tasks that you may want to execute in a variety of ways.

Syntax

In Storm, macros are created, modified, viewed, and deleted using the Storm macro commands. Permissions for macros are managed using the $lib.macro.grant(name, scope, iden, level) library.

In Optic, macros and their permissions can also be managed through the Storm Editor Tool.

Examples

As “stored Storm”, macros can contain any Storm that simplifies your analysis workflow - from the very simple to longer, more detailed queries. The command macro.exec <macro_name> is much simpler to type than Storm that you have to recreate each time. The examples below are relatively simple for illustrative purposes; that said, any Storm that you want to easily and consistently run on a regular basis, no matter how simple or complex, is a candidate for a macro (or potentially for other automation).

The examples below show creating macros using the macro.set command.

Self-contained macro - example

You have a set of IPv4 addresses that you have identified as sinkholes (tagged #cno.infra.dns.sink.hole). By analyzing associated DNS and domain whois data, you have tagged several FQDNs resolving to the IPv4s as sinkholed domains (#cno.infra.dns.sink.holed). You want to periodically check the sinkhole IPv4s for newly sinkholed FQDNs. To do this, you create a macro called sinkhole.check that will:

query a passive DNS data source to check for new FQDNs resolving to the various sinkhole IPv4s;
perform a live DNS A lookup on the FQDNs to double-check against the passive DNS results;
retrieve the domains’ current whois records; and
tag the FQDNs for review, including a timestamp so the reviewer knows when the FQDN was added to the set of domains for review.

storm> macro.set sinkhole.check { $now=$lib.time.now() inet:ipv4#cno.infra.dns.sink.hole | alienvault.otx.pdns --yield | -> inet:fqdn :zone -> inet:fqdn -#cno.infra.dns.sink.holed | uniq | nettools.dns | nettools.whois | [ +#int.review.sinkhole=$now ] }
Set macro: sinkhole.check

The Storm for this macro is broken down below (with comments) for clarity. (You can include comments within a macro; Synapse will ignore comment lines during execution.)

// Get the current time (in UTC) to use for the tag timestamp
$now=$lib.time.now()

// Lift IPv4 nodes tagged as sinkholes
inet:ipv4#cno.infra.dns.sink.hole

// Obtain PDNS information from AlienVault and yield the resulting inet:dns:a nodes
alienvault.otx.pdns --yield |

// Pivot to the FQDNs and then to the FQDN zones
-> inet:fqdn :zone -> inet:fqdn

// Filter out FQDNs that have already been identified as sinkholed
-#cno.infra.dns.sink.holed |

// De-duplicate results
uniq |

// Obtain the current / live DNS A and whois data for the FQDNs
nettools.dns | nettools.whois |

// Tag the FQDNs for review and set the current timestamp
[ +#int.review.sinkhole=$now ]

An analyst can now lift the FQDNs that need to be reviewed using the #int.review.sinkhole tag and review the associated data. If the FQDN is confirmed to be sinkholed, the analyst can apply the #cno.infra.dns.sink.holed tag and remove the #int tag once the FQDN has been reviewed.

If additional criteria would allow you to confirm that the FQDNs were sinkholed (e.g., some additional information in the whois data, or use of a particular DNS name server / DNS NS record), you could encode this logic using Storm and simply tag the relevant FQDNs as sinkholed vs. requiring analyst review.

Tip

This macro is “self-contained” (does not expect inbound ndoes); the macro’s Storm lifts the nodes that the macro operates on. (This does not prevent a user from “sending” nodes to the macro - which may cause unexpected effects - but as written, the macro can execute on its own.)

This macro can be executed “on demand” with the macro.exec command, or could be configured to run automatically as a cron job.

The alienvault.otx.pdns command is installed by the Synapse-AlienVault Power-Up. The nettools.* commands are installed by the Synapse-Nettools Power-Up.

Macro that takes nodes as input - example

One of Synapse’s strengths is the ability to retrieve data from a wide range of external sources, typically by using a Storm command provided by a Power-Up. The Storm commands allow an analyst to ingest data into Synpase and represent disparate data in a consistent manner for review and analysis. An analyst who wants to investigate an object (such as an indicator) in Synapse will commonly run multiple Storm commands to retrieve data from any / all data sources (a process commonly known as “enrichment”).

Instead of requiring the analyst to remember and run multiple individual Storm commands for enrichment, you want to create a macro called enrich that can take a variety of different nodes as input, and automatically run the appropriate Storm commands to call the data sources that can enrich the input node(s).

Once again, we create the macro with the macro.set command. Because of the length of the associated Storm, the full <storm_query> is provided below for readability.

macro.set enrich { <storm_query> }

The content of the macro (the <storm_query>, with comments):

// Filter inbound nodes to supported forms only
+(hash:md5 or hash:sha1 or hash:sha256 or inet:fqdn or inet:ipv4)

// Switch statement to handle different forms
switch $node.form() {

  ("hash:md5","hash:sha1"): {
      { | virustotal.file.enrich | virustotal.file.behavior }
  }

  "hash:sha256": {
      { | virustotal.file.enrich | virustotal.file.behavior | alienvault.otx.files }
  }

  "inet:fqdn": {

      // For FQDN zones
      {
          +:iszone=1 +:issuffix=0
          { | nettools.dns --type A AAAA CNAME MX NS SOA TXT | nettools.whois | virustotal.pdns | virustotal.commfiles | alienvault.otx.domain | alienvault.otx.pdns }
      }

      // For FQDN subdomains
      {
          +:iszone=0 +:issuffix=0
          { | nettools.dns --type A AAAA CNAME | virustotal.pdns | virustotal.commfiles | alienvault.otx.domain | alienvault.otx.pdns }
      }
  }

  "inet:ipv4": {

      +:type=unicast
      { | maxmind | nettools.dns | nettools.whois | virustotal.pdns | virustotal.commfiles | censys.hosts.enrich | alienvault.otx.ip | alienvault.otx.pdns }
  }

  // Default case for any forms not specified above
  *: {
      // Do nothing
  }
}

Instead of running multiple Storm commands, an analyst can now run a single macro that will perform enrichment by “intelligently” querying all available data sources that can provide information about a particular kind of indicator.

Tip

This macro requires nodes as input; the macro’s Storm does not lift any nodes to operate on. The initial filter statement is provided as basic error checking to ensure only nodes that the macro knows how to process are sent to the remaining macro code. (Technically, the final “default” switch option ( *: { } ) will handle any unknown forms; using the filter simply “drops” the nodes earlier in the pipeline. If you want any unhandled forms to pass through the macro (so they are available for additional Storm operations after the macro finishes), the filter should be removed.)

This macro can be executed “on demand” with the macro.exec command. If some condition should cause enrichment to occur automatically (e.g., applying a particular tag to a node), the macro could be configured to run as a trigger.

The example commands used in the macro are all installed by various Synapse Power-Ups.

The macro makes use of a Switch Statement (part of Storm’s control flow features) to handle different kinds of nodes (forms).

Macros run inline by default; when a user calls a macro, the full macro code executes to completion. For enrichment-type macros, depending on the number of data sources queried and nodes created, enrichment can take some time to run, which can cause the Synapse interface (the Storm CLI or the Optic UI) to “block” for the user until the Storm executes in full. The macro code can be run asynchronously using the Storm background command. This “unblocks” the CLI / UI, but the full set of results from enrichment will not be available until the macro finishes executing.

Modify macro permissions - examples

You can modify permissions on a macro using the $lib.macro.grant(name, scope, iden, level) library to grant (or revoke) access. (In Optic, individual macro permissions can be managed through the Storm Editor Tool.) Use the $lib.auth.users.byname(name) and $lib.auth.roles.byname(name) libraries to obtain a user or role object and retrieve the associated identifier (iden).

Make user “ron the cat” co-admin of the “sinkhole.check” macro:

storm> $user=$lib.auth.users.byname('ron the cat') $iden=$user.iden $lib.macro.grant(sinkhole.check,users,$iden,3)

Allow the “cattribution analysts” role to edit the “enrich” macro:

storm> $role=$lib.auth.roles.byname('cattribution analysts') $iden=$role.iden $lib.macro.grant(enrich,roles,$iden,2)

Deny the “interns” role access to the “enrich” macro:

storm> $role=$lib.auth.roles.byname(interns) $iden=$role.iden $lib.macro.grant(enrich,roles,$iden,0)

Tip

Easy permissions allow you to assign common permissions based on a corresponding integer value:

0 - Deny
1 - Read (including execute)
2 - Edit (modify content; includes Read)
3 - Admin (delete the object and modify its permissions; includes Edit)

See the Synapse Admin Guide for a detailed discussion of permissions, including easy permissions (“easy perms”).

Dmons

A Dmon is a long-running or recurring query or process that runs continuously in the background, similar to a traditional Linux or Unix daemon.

Variables

Dmons will have the storm variable $auto populated when they run. The $auto variable is a dictionary which contains the following keys:

$auto.iden
The identifier of the Dmon.

$auto.type
The type of automation. For a Dmon this value will be dmon.

Note

If the variable $auto was captured during the creation of the Dmon, the variable will not be mapped in.

Syntax

Users can interact with dmons using the Storm dmon commands and the $lib.dmon Storm libraries.