Storm Reference - Advanced - Variables

Storm supports the use of variables. A Variable is a value that can change depending on conditions or on information passed to the Storm query. (Contrast this with a Constant, which is a value that is fixed and does not change.)

Variables can be used in a variety of ways, such as providing more efficient ways to reference node properties; facilitating bulk operations; or writing extensions to Synapse (such as Power-Ups) in Storm.

These documents approach variables and their use from a user standpoint and aim to provide sufficient background for users to understand and begin to use variables. They do not provide an in-depth discussion of variables and their use. See the Synapse Developer Guide for more developer-focused topics.

Note

It is important to keep the high-level Storm Operating Concepts in mind when writing Storm queries or code. This is especially true when working with variables, control flow, and other more advanced concepts.

Variable Concepts

Variable Scope

A variable’s scope is its lifetime and under what conditions it may be accessed. There are two dimensions that impact a variable’s scope: its call frame and its runtime safety (“runtsafety”).

Call Frame

A variable’s call frame is where the variable is used. The main Storm query starts with its own call frame, and each call to a “pure” Storm command, function, or subquery creates a new call frame. The new call frame gets a copy of all the variables from the calling call frame. Changes to existing variables or the creation of new variables within the new call frame do not impact the calling scope.

Runtsafe vs. Non-Runtsafe

An important distinction to keep in mind when using variables in Storm is whether the variable is runtime-safe (Runtsafe) or non-runtime safe (Non-Runtsafe).

A variable that is runtsafe has a value independent of any nodes passing through the Storm pipeline. For example, a variable whose value is explicitly set, such as $string = mystring or $ipv4 = 8.8.8.8 is considered runtsafe because the value does not change / is not affected by the specific node passing through the Storm pipeline (i.e., by the Storm runtime).

A variable that is non-runtsafe has a value derived from a node passing through the Storm pipeline. For example, a variable whose value is set to a node property value may change based on the specific node passing through the Storm pipeline at the time. In other words, if your Storm query is operating on a set of DNS A nodes (inet:dns:a) and you define the variable $fqdn = :fqdn to set the variable to the value of the :fqdn property, the value of the variable will change based on the value of that property for the current inet:dns:a node.

All non-runtsafe variables are scoped to an individual node as it passes through the Storm pipeline. This means that a variable’s value based on a given node is not available when processing a different node (at least not without using special commands, methods, or libraries). In other words, the path of a particular node as it passes through the Storm pipeline is its own scope.

Note

The “safe” in non-runtsafe should not be interpreted to mean that the use of non-runtsafe variables is somehow “risky” or involves insecure programming or processing of data. It simply means the value of the variable is not safe from changing (i.e., it may change) as the Storm pipeline progresses.

Types of Variables

Storm supports two types of variables:

  • Built-in variables. Built-in variables facilitate many common Storm operations. They may vary in their scope and in the context in which they can be used.

  • User-defined variables User-defined variables are named and defined by the user. They are most often limited in scope and facilitate operations within a specific Storm query.

Built-In Variables

Storm includes a set of built-in variables and associated variable methods and libraries that facilitate Cortex-wide, node-specific, and context-specific operations.

Built-in variables differ from user-defined variables in that built-in variable names:

  • are initialized at Cortex start,

  • are reserved,

  • can be accessed automatically (i.e., without needing to define them) from within Storm, and

  • persist across user sessions and Cortex reboots.

Tip

We cover a few of the most common built-in variables here. For additional detail on Synapse’s Storm types (objects) and libraries, see the Storm Library Documentation.

Global Variables

Global variables operate independently of any node. That is, they can be invoked in a Storm query in the absence of any nodes in the Storm execution pipeline (though they can also be used when performing operations on nodes).

$lib

The library variable ( $lib ) is a built-in variable that provides access to the global Storm library. In Storm, libraries are accessed using built-in variable names (e.g., $lib.print()).

Libraries provide access to a wide range of additional functionality with Storm. See the Storm Libraries technical documentation for descriptions of the libraries available within Storm.

Node-Specific Variables

Storm includes node-specific variables that are designed to operate on or in conjunction with nodes and require one or more nodes in the Storm pipeline.

Note

Node-specific variables are always non-runtsafe.

$node

The node variable ($node) is a built-in Storm variable that references the current node in the Storm pipeline. Specifically, this variable contains the inbound node’s node object, and provides access to the node’s attributes, properties, and associated attribute and property values.

Invoking this variable during a Storm query is useful when you want to:

  • access the entire raw node object,

  • store the value of the current node before pivoting to another node, or

  • use an aspect of the current node in subsequent query operations.

The $node variable supports a number of built-in methods that can be used to access specific data or properties associated with a node. See the technical documentation for the node object or the $node section of the Storm Reference - Advanced - Methods user documentation for additional detail and examples.

$path

The path variable ($path) is a built-in Storm variable that references the path of a node as it travels through the pipeline of a Storm query.

The $path variable is not used on its own, but in conjunction with its methods. See the technical documentation for the node:path object or the $path section of the Storm Reference - Advanced - Methods user documentation for additional detail and examples.

Trigger-Specific Variables

A Trigger is used to support automation within a Cortex. Triggers use events (such as creating a node, setting a node’s property value, or applying a tag to a node) to fire (“trigger”) the execution of a predefined Storm query. Storm uses a built-in variable specifically within the context of trigger-initiated Storm queries.

$tag

For triggers that fire on tag:add events, the $tag variable represents the name of the tag that caused the trigger to fire.

For example:

You write a trigger to fire when any tag matching the expression #foo.bar.* is added to a file:bytes node. The trigger executes the following Storm command:

-> hash:md5 [ +#$tag ]

Because the trigger uses a tag glob (“wildcard”) expression, it will fire on any tag that matches that expression (e.g., #foo.bar.hurr, #foo.bar.derp, etc.). The Storm snippet above will take the inbound file:bytes node, pivot to the file’s associated MD5 node (hash:md5), and apply the same tag that fired the trigger to the MD5.

See the Triggers section of the Storm Reference - Automation document and the Storm trigger command for a more detailed discussion of triggers and associated Storm commands.

Ingest Variables

Synapse’s csvtool can be used to ingest (import) data into Synapse from a comma-separated value (CSV) file. Storm includes a built-in variable to facilitate bulk data ingest using CSV.

$rows

The $rows variable refers to the set of rows in a CSV file. When ingesting data into Synapse, CSVTool (or the Optic Ingest Tool) reads a CSV file and a file containing a Storm query that tells Synapse how to process the CSV data. The Storm query is typically constructed to iterate over the set of rows ($rows) using a For Loop that uses user-defined variables to reference each field (column) in the CSV data.

For example:

for ($var1, $var2, $var3, $var4) in $rows { <do stuff> }

Tip

The commercial Synapse UI (Optic) includes an Ingest Tool that can ingest data in CSV, JSONL, or JSON format. The $rows variable is used in the Ingest Tool to refer to either the set of rows in a CSV file or the set of lines (“rows”) in a JSONL file. In addition, the $blob variable is used to refer to the entire JSON blob when ingesting JSON data. See the ingest examples section of the Ingest Tool documentation for additional detail.

User-Defined Variables

User-defined variables can be defined in one of two ways:

  • At runtime (i.e., within the scope of a specific Storm query). This is the most common use for user-defined variables.

  • Mapped via options passed to the Storm runtime (for example, when using the Cortex API). This method is less common for everyday users. When defined in this manner, user-defined variables will behave as though they are built-in variables that are runtsafe.

Variable Names

All variable names in Storm (including built-in variables) begin with a dollar sign ( $ ). A variable name can be any alphanumeric string, except for the name of a built-in variable (see Built-In Variables), as those names are reserved. Variable names are case-sensitive; the variable $MyVar is different from $myvar.

Note

Storm will not prevent you from using the name of a built-in variable to define a variable (such as $node = 7). However, doing so may result in undesired effects or unexpected errors due to the variable name collision.

Defining Variables

Within Storm, a user-defined variable is defined using the syntax:

$<varname> = <value>

The variable name must be specified first, followed by the equals sign and the value of the variable itself.

<value> can be:

  • an explicit value (literal),

  • a node property (secondary or universal),

  • a built-in variable or method (e.g., can allow you to access a node’s primary property, form name, or other elements),

  • a tag (allows you to access timestamps associated with a tag),

  • a library function,

  • an expression, or

  • an embedded Storm query.

Examples

The examples below use the $lib.print() library function to display the value of the user-defined variable being set. (This is done for illustrative purposes only; $lib.print() is not required in order to use variables or methods.)

In some instances we include a second example to illustrate how a particular kind of variable assignment might be used in a real-world scenario. While we have attempted to use relatively simple examples for clarity, some examples may leverage additional Storm features such as subqueries, subquery filters, or control flow elements such as for loops or switch statements.

Tip

Keep Storm’s operation chaining, pipeline, and node consumption aspects in mind when reviewing the following examples. When using $lib.print() to display the value of a variable, the queries below will:

  • Lift the specified node(s).

  • Assign the variable. Note that assigning a variable has no impact on the nodes themselves.

  • Print the variable’s value using $lib.print().

  • Return any nodes still in the pipeline. Because variable assignment doesn’t impact the node(s) or transform the working set, the nodes remain in the pipeline and are returned (displayed) at the CLI.

The effect of this process is that for each node in the Storm query pipeline, the output of $lib.print() is displayed, followed by the relevant node.

In some examples the Storm spin command is used to suppress display of the node itself. We do this for cases where displaying the node detracts from illustrating the value of the variable.

Explicit values / literals

You can assign an explicit, unchanging value to a variable.

  • Assign the value 5 to the variable $threshold:

storm> $threshold=5 $lib.print($threshold)
5

Example:

  • Tag file:bytes nodes that have a number of AV signature hits higher than a given threshold for review:

storm> $threshold=5 file:bytes +{ -> it:av:filehit } >= $threshold [ +#review ]
file:bytes=sha256:00007694135237ec8dc5234007043814608f239befdfc8a61b992e4d09e0cf3f
        :sha256 = 00007694135237ec8dc5234007043814608f239befdfc8a61b992e4d09e0cf3f
        .created = 2024/04/19 14:14:36.011
        #review

Tip

The example above uses a subquery filter (Subquery Filters) to pivot to the it:av:filehit nodes associated with the file:bytes node, and compares the number of AV hits to the value of the $threshold variable.

Node properties

You can assign the value of a particular node property (secondary or universal) to a variable.

  • Secondary property: Assign the :user property from an Internet-based account (inet:web:acct) to the variable $user:

storm> inet:web:acct=(twitter.com,hacks4cats) $user=:user $lib.print($user)
hacks4cats
inet:web:acct=twitter.com/hacks4cats
        :email = [email protected]
        :site = twitter.com
        :user = hacks4cats
        .created = 2024/04/19 14:14:36.150

  • Universal property: Assign the .seen universal property from a DNS A node to the variable $time:

storm> inet:dns:a=(woot.com,1.2.3.4) $time=.seen $lib.print($time)
(1543289294000, 1565893967000)
inet:dns:a=('woot.com', '1.2.3.4')
        :fqdn = woot.com
        :ipv4 = 1.2.3.4
        .created = 2024/04/19 14:14:36.218
        .seen = ('2018/11/27 03:28:14.000', '2019/08/15 18:32:47.000')

Note

In the output above, the variable value is displayed as a pair of epoch milliseconds, which is how Synapse stores date/time values.

Example:

  • Given a DNS A record observed within a specific time period, find other DNS A records that pointed to the same IP address in the same time window:

storm> inet:dns:a=(woot.com,1.2.3.4) $time=.seen -> inet:ipv4 -> inet:dns:a +.seen@=$time
inet:dns:a=('woot.com', '1.2.3.4')
        :fqdn = woot.com
        :ipv4 = 1.2.3.4
        .created = 2024/04/19 14:14:36.218
        .seen = ('2018/11/27 03:28:14.000', '2019/08/15 18:32:47.000')
inet:dns:a=('hurr.net', '1.2.3.4')
        :fqdn = hurr.net
        :ipv4 = 1.2.3.4
        .created = 2024/04/19 14:14:36.276
        .seen = ('2018/12/09 06:02:53.000', '2019/01/03 11:27:01.000')

Tip

An interval (such as a .seen property) consists of a pair of date/time values. In the example above, the value of the variable $time is the combined pair (min / max) of times.

To access the “first seen” (minimum) or “last seen” (maximum) time values separately, use a pair of variables in the assignment:

($min, $max) = .seen

Built-in variables and methods

Built-In Variables (including Node-Specific Variables) allow you to reference common Synapse objects and their associated components. For many common user-facing tasks, the $node variable and its methods are the most useful.

  • Node object: Assign an entire FQDN node to the variable $fqdn using the $node built-in variable:

storm> inet:fqdn=mail.mydomain.com $fqdn=$node $lib.print($fqdn)
Node{(('inet:fqdn', 'mail.mydomain.com'), {'iden': '6511121afd61bf42cb4d14aed4f61daf62ebfc76042dba12d95a6506dd8b6cc4', 'tags': {}, 'props': {'.created': 1713536076334, 'host': 'mail', 'domain': 'mydomain.com', 'issuffix': 0, 'iszone': 0, 'zone': 'mydomain.com'}, 'tagprops': {}, 'nodedata': {}})}
inet:fqdn=mail.mydomain.com
        :domain = mydomain.com
        :host = mail
        :issuffix = false
        :iszone = false
        :zone = mydomain.com
        .created = 2024/04/19 14:14:36.334

Note

When you use the built-in variable $node to assign a value to a variable, the value is set to the entire node object (refer to the output above). For common user-facing tasks, it is less likely that users will need “the entire node”; more often, they need to refer to a component of the node, such as its primary property value, form name, or associated tags.

For some use cases, Synapse and Storm can “understand” which component of the node you want when referring to the full $node object. However, you can always be explicit by using the appropriate method to access the component you want (such as $node.value() or $node.form()).

See the technical documentation for the node object or the $node section of the Storm Reference - Advanced - Methods user documentation for additional detail and examples when using methods associated with the $node built-in variable.

  • Node method: Assign the primary property value of a domain node to the variable $fqdn using the $node.value() method:

storm> inet:fqdn=mail.mydomain.com $fqdn=$node.value() $lib.print($fqdn)
mail.mydomain.com
inet:fqdn=mail.mydomain.com
        :domain = mydomain.com
        :host = mail
        :issuffix = false
        :iszone = false
        :zone = mydomain.com
        .created = 2024/04/19 14:14:36.334

  • Find the DNS A records associated with a given domain where the PTR record for the IP matches the FQDN:

storm> inet:fqdn=mail.mydomain.com $fqdn=$node.value() -> inet:dns:a +{ -> inet:ipv4 +:dns:rev=$fqdn }
inet:dns:a=('mail.mydomain.com', '25.25.25.25')
        :fqdn = mail.mydomain.com
        :ipv4 = 25.25.25.25
        .created = 2024/04/19 14:14:36.389

Tip

The example above uses a subquery filter (see Subquery Filters) to pivot from the DNS A records to associated IPv4 nodes (inet:ipv4) and checks whether the :dns:rev property matches the FQDN in the variable $fqdn.

Tags

Recall that tags are both nodes (syn:tag=my.tag) and labels that can be applied to other nodes (#my.tag). Tags can also have optional timestamps (a time interval) associated with them.

There are various ways to assign tags as variables, depending on what part of the tag you want to access. Many of these use cases are covered above so are briefly illustrated here.

  • Tag value: Assign an explicit tag value (literal) to the variable $mytag:

storm> $mytag=cno.infra.dns.sinkhole

  • Tag on a node: Given a hash:md5 node, assign any malware tags (tags matching the glob pattern cno.mal.*) to the variable $mytags using the $node.tags() method:

storm> hash:md5=d41d8cd98f00b204e9800998ecf8427e $mytags=$node.tags(cno.mal.*) $lib.print($mytags)
['cno.mal.foo', 'cno.mal.bar']
hash:md5=d41d8cd98f00b204e9800998ecf8427e
        .created = 2024/04/19 14:14:36.495
        #cno.mal.bar
        #cno.mal.foo
        #cno.threat.baz

Tip

In the example above, the value of the variable $mytags is the set of two tags, cno.mal.foo and cno.mal.bar, because the MD5 hash node has two tags that match the pattern cno.mal.*.

To assign the set of any / all tags on a node to a variable, simply use $mytags=$node.tags().

Note that you can also use $node.tags() directly (this method always refers to the full set of tags on the current node) without explicitly assigning a separate variable.)

Where the value of a variable is a set, a For Loop is often used to “do something” based on each value in the set.

Example

  • Given an MD5 hash, copy any cno.mal.* tags from the hash to the associated file (file:bytes node):

storm> hash:md5=d41d8cd98f00b204e9800998ecf8427e $mytags=$node.tags(cno.mal.*) for $tag in $mytags { -> file:bytes [ +#$tag ] }
file:bytes=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        :md5 = d41d8cd98f00b204e9800998ecf8427e
        :sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709
        :sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        :size = 0
        .created = 2024/04/19 14:14:36.547
        #cno.mal.foo
file:bytes=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        :md5 = d41d8cd98f00b204e9800998ecf8427e
        :sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709
        :sha256 = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        :size = 0
        .created = 2024/04/19 14:14:36.547
        #cno.mal.bar
        #cno.mal.foo

The output above includes two “copies” of the file:bytes node because the node is output twice - once for each iteration of the for loop. The first iteration copies / applies the cno.mal.foo tag; the second iteration applies the cno.mal.bar tag. For a detailed explanation of this behavior, see Advanced Storm - Example.

Tip

The above example explicitly creates and assigns the variable $mytags and then uses that variable in a For Loop. In this case you can shorten the syntax by skipping the explicit variable assignment and using the $node.tags() method directly:

hash:md5=d41d8cd98f00b204e9800998ecf8427e for $tag in $node.tags(cno.mal.*) { -> file:bytes [ +#$tag ] }

  • Tag timestamps: Assign the times associated with Threat Group 20’s control of a malicious domain to the variable $time:

storm> inet:fqdn=evildomain.com $time=#cno.threat.t20.own $lib.print($time)
(1567900800000, 1631059200000)
inet:fqdn=evildomain.com
        :domain = com
        :host = evildomain
        :issuffix = false
        :iszone = true
        :zone = evildomain.com
        .created = 2024/04/19 14:14:36.600
        #cno.threat.t20.own = (2019/09/08 00:00:00.000, 2021/09/08 00:00:00.000)

Example

  • Find DNS A records for any subdomain associated with a Threat Group 20 FQDN (zone) during the time they controlled the domain:

storm> inet:fqdn#cno.threat.t20.own $time=#cno.threat.t20.own -> inet:fqdn:zone -> inet:dns:a +.seen@=$time
inet:dns:a=('www.evildomain.com', '1.2.3.4')
        :fqdn = www.evildomain.com
        :ipv4 = 1.2.3.4
        .created = 2024/04/19 14:14:36.653
        .seen = ('2020/07/12 00:00:00.000', '2020/12/13 00:00:00.000')
inet:dns:a=('smtp.evildomain.com', '5.6.7.8')
        :fqdn = smtp.evildomain.com
        :ipv4 = 5.6.7.8
        .created = 2024/04/19 14:14:36.661
        .seen = ('2020/04/04 00:00:00.000', '2020/08/02 00:00:00.000')

Library Functions

Storm types (Storm objects) and Storm libraries allow you to inspect, edit, and otherwise work with data in Synapse in various ways. You can assign a value to a variable based on the output of a method or library.

A full discussion of this topic is outside of the scope of this user guide. See Storm Library Documentation for additional details.

  • Assign the current time to the variable $now using $lib.time.now():

storm> $now=$lib.time.now() $lib.print($now)
1713536076718

  • Convert an epoch milliseconds integer into a human-readable date/time string using $lib.str.format():

storm> $now=$lib.time.now() $time=$lib.time.format($now, '%Y/%m/%d %H:%M:%S') $lib.print($time)
2024/04/19 14:14:36

Expressions

You can assign a value to a variable based on the computed value of an expression:

  • Use an expression to increment the variable $x:

storm> $x=5 $x=($x + 1) $lib.print($x)
6

Embedded Storm query

You can assign a value to a variable based on the output of a Storm query. To denote the Storm query to be evaluated, enclose the query in curly braces ({ <storm query> }).

  • Assign an ou:org node’s guid value to the variable $org by lifting the associated org node using its :name property:

storm> $org={ ou:org:name=vertex } $lib.print($org)
31b9fa3cc6566822dde43eb15dcb884c