Storm Reference - Lifting
Lift operations retrieve a set of nodes from a Synapse Cortex based on specified criteria. While all lift operations are retrieval operations, they can be broken down into “types” of lifts based on the criteria, comparison operator, or special handler used:
See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below.
See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.
Simple Lifts
“Simple” lifts refers to the most “basic” lift operations. That is, operations to retrieve a set of nodes based on:
The presence of a specific primary or secondary property.
The presence of a specific primary property value or secondary property value.
The presence of a specific tag or tag property.
The only difference between “simple” lifts and “lifts using comparison operators” is that we have defined simple lifts as those that use the equals ( =
) comparator, which is the easiest comparator to use to explain basic lift concepts.
Syntax:
<form> [ = <valu> ]
<form> = <valu> : <prop> [ = <pval> ]
# <tag> [ : <tagprop> [ <operator> <pval> ] ]
#: <tagprop> [ <operator> <pval> ]
Examples:
Lift by primary property (<form>):
Lift all domain nodes:
storm> inet:fqdn
Lift all mutex nodes:
storm> it:dev:mutex
Lift a specific node (<form> = <valu>):
Lift the node for the domain
google.com
:
storm> inet:fqdn = google.com
Lift the node for a specific MD5 hash:
storm> hash:md5 = d41d8cd98f00b204e9800998ecf8427e
Lift a specific compound node:
Lift the DNS A record showing that domain
woot.com
resolved to IP1.2.3.4
:
storm> inet:dns:a = (woot.com, 1.2.3.4)
Lift a specific GUID node:
Lift the organization node with the specified GUID:
storm> ou:org=2f92bc913918f6598bcf310972ebf32e
Lift a specific digraph (edge) node:
Lift the
edge:has
node linking the person node representing “Bob Smith” to his email address:
storm> edge:has=((ps:person,12af06294ddf1a0ac8d6da34e1dabee4),(inet:email, [email protected]))
Lift by the presence of a secondaray property (<prop>):
Lift the DNS SOA record nodes that have an email property:
storm> inet:dns:soa:email
Lift by a specific property value (<prop> = <pval>):
Lift the organization node with the alias
vertex
:
storm> ou:org:alias = vertex
Lift all DNS A records for the domain
blackcake.net
:
storm> inet:dns:a:fqdn = blackcake.net
Lift all the files with a PE compiled time of
1992-06-19 22:22:17
:
storm> file:bytes:mime:pe:compiled = "1992/06/19 22:22:17"
Lift all the files with a PE compiled time that falls within the year 2019:
storm> file:bytes:mime:pe:compiled=2019*
Lift all nodes with a specific tag:
Lift all nodes with the tag
#cno.infra.anon.tor
:
storm> #cno.infra.anon.tor
Lift all nodes with a specific tag property:
Lift all nodes with a tag that has a
:risk
tag property:
Lift all nodes with a specific tag and tag property:
Lift all nodes with a
#rep.symantec
tag that has a:risk
tag property:
storm> #rep.symantec:risk
Lift all nodes with a specific tag, tag property, and value:
Lift all nodes with a
#rep.symantec
tag with a:risk
tag property and a value greater than 10:
storm> #rep.symantec:risk>10
Usage Notes:
Lifting nodes by form alone (e.g., lifting all
inet:fqdn
nodes or allinet:email
nodes) is possible but generally impractical / undesirable as it will potentially return an extremely large data set.Lifting by form alone when piped to the Storm limit command may be useful for returning a small number of “exemplar” nodes.
Lifting nodes by
<form> = <valu>
is the most common method of lifting a single node.When lifting a form whose
<valu>
consists of multiple components (e.g., a compound node or digraph node), the components must be passed as a comma-separated list enclosed in parentheses.Lifting nodes by the presence of a secondary property alone (
<prop>
) may be impractical / undesirable (similar to lifting by form alone), but may be feasible in limited cases (i.e., where it is known that only a relatively small number of nodes have a given secondary property).Lifting nodes by the value of a secondary property (
<prop> = <pval>
) is useful for lifting all nodes that share a secondary property with the same value; and may be used to lift individual nodes with unique or relatively unique secondary properties in cases where entering the primary property is impractical (such as for GUID nodes).When lifting nodes by secondary property value where the value is a time (date / time), you do not need to use full
YYYY/MM/DD hh:mm:ss.mmm
syntax. Synapse allows the use of both lower resolution values (e.g.,YYYY/MM/DD
) and wildcard values (e.g.,YYYY/MM*
). In particular, wildcard syntax can be used to specify any values that match the wildcard expression. See the type-specific documentation for time types for a detailed discussion of these behaviors.Lifting nodes by tag alone (
#<tag>
) lifts nodes of all forms with that tag. To lift specific forms only, use Lift by Tag (#) or an additional filter (see Storm Reference - Filtering).Tag properties are supported in Synapse, but no tag properties are included by default. See Tag Properties for additional detail.
Try Lifts
Try lifts refer to lifts that “try” to perform a Cortex lift operation, and fail silently if Type normalization is not successful. Try lifts prevent a Cortex from throwing a runtime execution error, and terminating query execution if an invalid Type is encountered.
When lifting nodes by property value using the equals (=
) comparator, if Type validation fails for a supplied property value, the Cortex will throw a BadTypeValu
error, and terminate the query as shown below.
storm> inet:ipv4 = evil.com inet:ipv4 = 8.8.8.8
ERROR: illegal IP address string passed to inet_aton
To suppress errors, and prevent premature query termination, Storm supports the use of the try operator (?=
) when performing property value lifts. This operator is useful when you are performing multiple Cortex operations in succession within a single query, lifting nodes using external data that has not been normalized, or lifting nodes during automation, and do not want a query to terminate if an invalid Type is encountered.
Syntax:
<form>[:<prop>] ?= <pval>
Examples:
Try to lift the MD5 node
174cc541c8d9e1accef73025293923a6
:
storm> hash:md5 ?= 174cc541c8d9e1accef73025293923a6
Try to lift the DNS nodes whose
inet:dns:a:ipv4
secondary property value equals'192.168.0.100'
. Notice that an error message is not displayed, despite an invalid IPv4 address'192.168.0.1000'
being entered:
storm> inet:dns:a:ipv4 ?= 192.168.0.1000
Try to lift the email address nodes
'jack@soso.net'
and'jane@goodgirl.com'
. Notice that despite the first email address being entered incorrectly, the error message is suppressed, and the query executes to completion.
storm> inet:email ?= "jack[at]soso.net" inet:email ?= "[email protected]"
inet:[email protected]
:fqdn = goodgirl.com
:user = jane
.created = 2023/06/05 20:14:10.451
Usage Notes:
The try operator should be used when you want Storm query execution to continue even if an invalid Type is encountered.
It is not recommended to use the try operator when you want to raise an error, or stop query execution if an invalid Type is encountered.
Lifts Using Standard Comparison Operators
Lift operations can be performed using most of the standard mathematical / logical comparison operators (comparators):
=
: equals (described above)<
: less than>
: greater than<=
: less than or equal to>=
: greater than or equal to
Lifting by “not equal to” (!=
) is not supported.
Syntax:
<prop> <comparator> <pval>
Examples:
Lift using less than comparator:
Lift domain WHOIS records where the domain’s registration (created) date was before June 1, 2014:
storm> inet:whois:rec:created < 2014/06/01
Lift using greater than comparator:
Lift files whose size is larger than 1MB:
storm> file:bytes:size > 1048576
Lift using less than or equal to comparator:
Lift people (person nodes) born on or before January 1, 1980:
storm> ps:person:dob <= 1980/01/01
Lift files that were compiled in 2012 or earlier:
storm> file:bytes:mime:pe:compiled<=2012*
Lift using greater than or equal to comparator:
Lift WHOIS records retrieved on or after December 1, 2018 at 12:00:
storm> inet:whois:rec:asof >= "2018/12/01 12:00"
Usage Notes:
When lifting nodes by secondary property value where the value is a time (date / time), you do not need to use full
YYYY/MM/DD hh:mm:ss.mmm
syntax. Synapse allows the use of both lower resolution values (e.g.,YYYY/MM/DD
) and wildcard values (e.g.,YYYY/MM*
). In particular, wildcard syntax can be used to specify any values that match the wildcard expression. See the type-specific documentation for time types for a detailed discussion of these behaviors.
Lifts Using Extended Comparison Operators
Storm supports a set of extended comparison operators (comparators) for specialized lift operations. In most cases, the same extended comparators are available for both lifting and filtering:
Lift by Regular Expression (~=)
The extended comparator ~=
is used to lift nodes based on standard regular expressions.
Note
Lift by Prefix (^=) is supported for string types and can be used to match the beginning of string properties.
Syntax:
<form> [ : <prop> ] ~= <regex>
Example:
Lift files with PDB paths containing the string
rouji
:
storm> file:bytes:mime:pe:pdbpath ~= "rouji"
Lift by Prefix (^=)
Synapse performs prefix indexing on string types, which optimizes lifting nodes whose <valu> or <pval> starts with a given prefix. The extended comparator ^=
is used to lift nodes by prefix.
Syntax:
<form> [ : <prop> ] ^= <prefix>
Examples:
Lift primary property by prefix:
Lift all usernames that start with “pinky”:
storm> inet:user^=pinky
Lift secondary property by prefix:
Lift all organizations whose name starts with “International”:
storm> ou:org:name^=international
Usage Notes:
Extended string types that support dotted notation (such as the loc or syn:tag types) have custom behaviors with respect to lifting and filtering by prefix. See the respective sections in Storm Reference - Type-Specific Storm Behavior for additional details.
Lift by Time or Interval (@=)
Synapse supports numerous data forms whose properties are date / time values (<ptype> = <time>) or time windows / intervals (<ptype> = <ival>). Storm supports the custom @=
comparator to allow lifting based on comparisons among various combinations of times and intervals.
See Storm Reference - Type-Specific Storm Behavior for additional detail on the use of time and ival data types.
Syntax:
<prop> @=( <ival_min> , <ival_max> )
<prop> @= <time>
Examples:
Lift by comparing an interval to an interval:
Lift all DNS A records whose
.seen
values fall between July 1, 2018 and August 1, 2018:
storm> inet:dns:[email protected]=(2018/07/01, 2018/08/01)
Lift by comparing a time to an interval:
Lift DNS requests that occurred on May 3, 2018 (between 05/03/2018 00:00:00 and 05/03/2018 23:59:59):
storm> inet:dns:request:[email protected]=("2018/05/03 00:00:00", "2018/05/04 00:00:00")
Lift by comparing a time to a time:
Lift all WHOIS records that were retrieved on July 17, 2017:
storm> inet:whois:rec:[email protected]=2017/07/17
Lift using an interval with relative times:
Lift the WHOIS email nodes that were observed between January 1, 2019 and the present:
storm> inet:whois:[email protected]=(2019/01/01, now)
Lift the DNS requests that occurred within one day after October 15, 2018:
storm> inet:dns:request:[email protected]=(2018/10/15,"+1 day")
Lift by comparing tag time intervals:
Lift all the domain nodes that were associated with Threat Group 43 between January 2013 and January 2015:
storm> inet:fqdn#[email protected]=(2013/01/01, 2015/01/01)
Usage Notes:
When specifying an interval, the minimum value is included in the interval but the maximum value is not (the equivalent of “greater than or equal to <min> and less than <max>”). This behavior is slightly different than that for
*range=
, which includes both the minimum and maximum.When comparing an interval to an interval, Storm will return nodes whose interval has any overlap with the specified interval.
For example, a lift interval of September 1, 2018 to October 1, 2018 (2018/09/01, 2018/10/01) will match nodes with any of the following intervals:
August 12, 2018 to September 6, 2018.
September 13, 2018 to September 17, 2018.
September 30, 2018 to November 5, 2018.
When comparing a time to an interval, Storm will return nodes whose time falls within the specified interval.
When comparing a time to a time, Storm will return nodes whose timestamp is an exact match. (Interval (
@=
) syntax is supported for this comparison, but the regular equals comparator (=
) can also be used.)When specifying interval date/time values, Synapse allows the use of both lower resolution values (e.g.,
YYYY/MM/DD
) and wildcard values (e.g.,YYYY/MM*
) for the minimum and/or maximum interval values. In addition, plain wildcard time syntax may provide a simpler and more intuitive means to specify some intervals. For exampleinet:whois:rec:asof=2018*
is equivalent toinet:whois:rec:asof@=('2018/01/01', '2019/01/01')
. See the type-specific documentation for time types for a detailed discussion of these behaviors.
Lift by Range (*range=)
The range extended comparator (*range=
) supports lifting nodes whose <form> = <valu> or <prop> = <pval> fall within a specified range of values. The comparator can be used with types such as integers and times (including types that are extensions of those types, such as IP addresses).
Syntax:
<form> [ : <prop> ] *range = ( <range_min> , <range_max> )
Examples:
Lift by primary property in range:
Lift all IP addresses between 192.168.0.0 and 192.168.0.10:
storm> inet:ipv4*range=(192.168.0.0, 192.168.0.10)
Lift by secondary property in range:
Lift files whose size is between 1000 and 100000 bytes:
storm> file:bytes:size*range=(1000,100000)
Lift WHOIS records that were captured between November 29, 2013 and June 14, 2016:
storm> inet:whois:rec:asof*range=(2013/11/29, 2016/06/14)
Lift DNS requests made within one day of 12/01/2018:
storm> inet:dns:request:time*range=(2018/12/01, "+-1 day")
Usage Notes:
When specifying a range, both the minimum and maximum values are included in the range (the equivalent of “greater than or equal to <min> and less than or equal to <max>”).
When specifying a range of time values, Synapse allows the use of both lower resolution values (e.g.,
YYYY/MM/DD
) and wildcard values (e.g.,YYYY/MM*
) for the minimum and/or maximum range values. In addition, plain wildcard time syntax may provide a simpler and more intuitive means to specify some time ranges. For exampleinet:whois:rec:asof=2018*
is equivalent toinet:whois:rec:asof*range=('2018/01/01', '2018/12/31 23:59:59.999')
. See the type-specific documentation for time types for a detailed discussion of these behaviors.
Lift by Set Membership (*in=)
The set membership extended comparator (*in=
) supports lifting nodes whose <form> = <valu> or <prop> = <pval> matches any of a set of specified values. The comparator can be used with any type.
Syntax:
<form> [ : <prop> ] *in = ( <set_1> , <set_2> , … )
Examples:
Lift by primary property in a set:
Lift IP addresses matching any of the specified values:
storm> inet:ipv4*in=(127.0.0.1, 192.168.0.100, 255.255.255.254)
Lift by secondary property in a set:
Lift files whose size in bytes matches any of the specified values:
storm> file:bytes:size*in=(4096, 16384, 65536)
Lift tags that end in
foo
,bar
, orbaz
:
storm> syn:tag:base*in=(foo,bar,baz)
Lift by Proximity (*near=)
The proximity extended comparator (*near=
) supports lifting nodes by “nearness” to another node based on a specified property type. Currently, *near=
supports proximity based on geospatial location (that is, nodes within a given radius of a specified latitude / longitude).
Syntax:
<form> [ : <prop> ] *near = (( <lat> , <long> ), <radius> )
Examples:
Lift locations (
geo:place
nodes) within 500 meters of the Eiffel Tower:
storm> geo:place:latlong*near=((48.8583701,2.2944813),500m)
Usage Notes:
In the example above, the latitude and longitude of the desired location (i.e., the Eiffel Tower) are explicitly specified as parameters to
*near=
.Radius can be specified in the following metric units:
Kilometers (km)
Meters (m)
Centimeters (cm)
Millimeters (mm)
Numeric values of less than 1 (e.g., 0.5km) must be specified with a leading zero.
The
*near=
comparator works for geospatial data by lifting nodes within a square bounding box centered at <lat>,<long>, then filters the nodes to be returned by ensuring that they are within the great-circle distance given by the <radius> argument.
Lift by (Arrays) (*[ ])
Storm uses a special “by” syntax to lift (or filter) by comparison with one or more elements of an array type. The syntax consists of an asterisk ( *
) preceding a set of square brackets ( [ ]
), where the square brackets contain a comparison operator and a value that can match one or more elements in the array. This allows users to match values in the array list without needing to know the exact order or values of the array itself.
Syntax:
<form> : <prop> [ <operator> <pval> ]
Examples:
Lift the organization(s) (
ou:org
nodes) whose names include “IBM”:
storm> ou:org:names*[=ibm]
Lift the x509 certificates (
crypto:x509:cert
) that reference domains ending with.biz
:
storm> crypto:x509:cert:identities:fqdns*[="*.biz"]
Lift the organizations whose names start with “tech”:
storm> ou:org:names*[^=tech]
Usage Notes:
The comparison operator used must be valid for lift operations for the type used in the array. For example, inet:fqdn suffix matching (i.e.,
crypto:x509:cert:identities:fqdns*[="*.com"]
), can be used to lift arrays consisting of domains, but the prefix operator (^=
), which is only valid when filteringinet:fqdns
, cannot.The standard equals (
=
) operator can be used to filter nodes based on array properties, but the value specified must exactly match the full property value in question:For example:
ou:org:names=("the vertex project","the vertex project llc",vertex)
See the array section of the Storm Reference - Type-Specific Storm Behavior document for additional details.
Lift by Tag (#)
The tag extended comparator (#
) supports lifting nodes based on a form combined with a given tag; tag and tag property; tag, tag property, and tag property value; or tag and associated timestamp being applied to the node.
Note
Lifting by form and tag (<form>#<tag>
), including similar lifts using tag properties / timestamps / etc., is actually a Synapse-optimized “lift and filter” operation as opposed to a standard lift. The operation is equivalent to <form> +#<tag> except that the lift by tag syntax is optimized for performance.
Using the explicit filter (<form> +#<tag>) lifts all nodes of the specified form and then downselects to only those forms with the specified tag.
Storm optimizes the lift by tag syntax to lift only those nodes of the specified form that have the specified tag.
See Filter by Tag (#) for additional detail on filtering with tags.
Syntax:
<form> # <tag>
<form> # <tag> [ : <tagprop> [ <operator> <pval> ] ]
<form> # <tag> @= <time> | ( <min_time> , <max_time> )
Examples:
Lift forms by tag:
Lift the IPv4 addresses associated with Tor infrastructure:
storm> inet:ipv4#cno.infra.anon.tor
Lift forms by tag and tag property:
Lift the domains that have a risk score reported by DomainTools:
storm> inet:fqdn#rep.domaintools:risk
Lift forms by tag, tag property, and value:
Lift the domains that have a risk score from DomainTools of 90 or higher:
storm> inet:fqdn#rep.domaintools:risk>=90
Lift forms by tag and time:
Lift domains that were associated with Threat Cluster 15 as of October 30, 2009:
storm> inet:fqdn#[email protected]=2009/10/30
Lift forms by tag and time interval:
Lift IP addresses that were part of Tor infrastructure between October 1, 2018 and December 31, 2018:
storm> inet:ipv4#[email protected]=(2018/10/01,2018/12/31)
Usage Notes:
Tag properties are supported in Synapse, but no tag properties are included by default. See Tag Properties for additional detail.
Currently it is not possible to lift forms by tag property alone. That is,
inet:fqdn#:risk
is invalid.It is possible to perform an equivalent operation using a lift and filter operation, i.e.,
#:risk +inet:fqdn
.
Tag timestamps are interval (
ival
) types. See the time and ival sections of the Storm Reference - Type-Specific Storm Behavior document for additional details on working with times and intervals.
Recursive Tag Lift (##)
The recursive tag extended comparator (##) supports lifting nodes with any tag whose syn:tag
node is itself tagged with a specific tag.
Tags can be applied to syn:tag
nodes; that is, tags can be used to tag other tags. The ability to “tag the tags” can be used to represent certain types of analytical relationships. For example:
syn:tag
nodes representing threat groups can be tagged to indicate their assessed country of origin.syn:tag
nodes representing malware or tools can be tagged with their assessed availability (e.g., public, private, private but shared, etc.)
A recursive tag lift performs the following actions:
For the specified tag (
##<sometag>
), lift the nodes that have that tag (i.e., the equivalent of#<sometag>
), including anysyn:tag
nodes.For any lifted
syn:tag
nodes, lift all nodes tagged with those tags (including any additionalsyn:tag
nodes).Repeat #2 until no more
syn:tag
nodes are lifted.Return the tagged nodes. Note that
syn:tag
nodes themselves are not returned.
Syntax:
## <tag>
Examples:
Lift all nodes tagged with any tags (such as threat group tags) that FireEye claims are associated with Russia:
storm> ##aka.feye.cc.ru
Usage Notes:
In the example above, the tag aka.feye.cc.ru
could be applied to syn:tag
nodes representing FireEye’s “Russian” threat groups (e.g., aka.feye.thr.apt28
, aka.feye.thr.apt29
, etc.) Using a recursive tag lift allows you to easily lift all nodes tagged by any of those tags.