Storm Reference - Filtering

Filter operations are performed on the output of a previous Storm operation such as a lift or pivot. A filter operation downselects from the working set of nodes by either including or excluding a subset of nodes based on specified criteria.

  • + specifies an inclusion filter. The filter downselects the working set to only those nodes that match the specified criteria.

  • - specifies an exclusion filter. The filter downselects the working set to all nodes except those that match the specified criteria.

Similar to lift operations (Storm Reference - Lifting), filter operations can be broken down into “types” of filters based on the criteria, comparison operator, or special handler used:

Tip

In general, you can filter using the same criteria and comparison operators used for lift operations. This includes using a wildcard ( * ) to partially match form names and using Interface names to filter by all forms that inherit an interface.

Because filter operations act on a pre-selected subset of nodes, some additional methods are available for filtering that would be less efficient for initial lift operations. For example, you can filter FQDNs (inet:fqdn nodes) by prefix ( ^= ), although you cannot lift FQDNs using that operator. Similarly, you can Filter by Tag Globs but you cannot lift using that syntax.

Storm also supports specialized filters and filter operations:

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below.

See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

Filter by Form

A “filter by form” operation modifies your working set to include (or exclude) all nodes of the specified form. The wildcard (asterisk) character ( * ) can be used to filter based on forms that match a partial form name / namespace.

If the form inherits an Interface, you can specify the interface name to filter based on all forms that inherit the interface.

Filter by Form Name

Syntax:

<query> + | - <form>

Examples:

Filter the current working set to only include domains (inet:fqdn nodes):

<query> +inet:fqdn

Filter the current working set to exclude URLs (inet:url nodes):

<query> -inet:url

Filter by Form Name - Wildcard

You can use the wildcard (asterisk) character ( * ) to specify all forms that match a partial form name. Use of the wildcard is not limited to form namespace boundaries.

Note

The wildcard can only be used at the end of the partial match. It cannot be used at the beginning or in the middle of the form name. For example, both of the following are invalid:

+*:header

-it:exec:*:del

In addition, use of the wildcard does not extend to partial matching of property names. For example, the following is invalid:

+file:bytes:mime:pe:*

Syntax:

<query> + | - <partial_form_name> *

Examples:

Filter the current working set to exclude PE metadata nodes (e.g., file:mime:pe:resource, file:mime:pe:section, etc.):

<query> -file:mime:pe:*

Filter the current working set to only include antivirus / scan-related nodes (e.g., it:av:scan:result, it:av:signame):

<query> +it:av:s*

Filter Form by Interface

You can use the name of an interface to filter all forms that inherit that interface.

Note

When filtering by interface, you cannot use the wildcard ( * ) character to match multiple interface names. Synapse will interpret use of the wildcard as an attempt to match multiple form names.

Syntax:

<query> + | - <interface>

Examples:

Filter the current working set to only include host activity nodes (all nodes of all forms that inherit the it:host:activity interface):

<query> +it:host:activity

Filter the current working set to exclude taxonomy nodes (all nodes of all forms that inherit the meta:taxonomy interface):

<query> -meta:taxonomy

Filter by Property

A “filter by property” operation modifies your working set to include (or exclude) all forms that have the specified property (secondary, universal, or extended), regardless of the property value.

Tip

When filtering by property, you can specify the property using either the full property name (i.e., the combined form and property, such as inet:dns:a:ipv4) or the relative property name (i.e., the property name alone, including its separator character, such as :ipv4).

Using the relative property name allows for simplified syntax and more efficient data entry (“less typing”). Full property names can be used for clarity (i.e., specifying exactly what you want to filter on).

Full property names are required when filtering on a property using an interface. They may also be required in cases where multiple nodes in the inbound working set have the same relative property name (e.g., inet:dns:a:ipv4 and inet:url:ipv4) and you only wish to filter based on the property of one of the forms.

Each example below is shown using both the full property name (<form>:<prop>) and the relative property name (:<prop>) where applicable.

Filter by Secondary Property

Syntax:

<query> + | - [ <form> ] : <prop>

Examples:

Filter the current working set to only include threats (risk:threat nodes) that have an assessed country of origin (:country:code property):

<query> +risk:threat:country:code

<query> +:country:code

Filter the current working set to exclude articles (media:news nodes) that have a publisher name (:publisher:name property):

<query> -media:news:publisher:name

<query> -:publisher:name

Filter by Interface Property

If a form inherits an Interface, you can filter all nodes of all forms that have an interface-derived property by specifying the full name of the interface and its property.

Syntax:

<query> + | - <interface> : <prop>

Example:

Filter the current working set to only include those host activity nodes (all nodes of all forms that inherit the it:host:activity interface) that have a :time property:

<query> +it:host:activity:time

Tip

When filtering using an interface property, you must use full property syntax (i.e., the combined interface and property name).

Filter by Universal Property

Syntax:

<query> + | - [ <form> ] . <prop>

Example:

Filter the current working set to only include DNS A records (inet:dns:a nodes) that have a .seen property:

<query> +inet:dns:a.seen

<query> +.seen

Filter by Extended Property

Syntax:

<query> + | - [ <form> ] :_ <prop>

Example:

Filter the current working set to exclude those organizations (ou:org nodes) that have an “isthreat” extended property:

<query> -ou:org:_vertex:threatintel:isthreat

<query> -:_vertex:threatintel:isthreat

Tip

The :_vertex:threatintel:isthreat extended property is a Boolean property added by the Vertex-Threat-Intel Power-Up. It can be used to indicate whether an organization is tracked as a threat group.

Filter by Property Value - Standard Comparison Operators

A “filter by property value” operation modifies the current working set to include (or exclude) the node(s) whose property matches the specified value. This type of filter requires:

  • the filter operator ( + or - );

  • the property name (full or relative) to use for the filter;

  • a Comparison Operator to specify how the property value should be evaluated; and

  • the property value.

A “filter by property value” can be performed using primary, secondary, universal, or extended properties.

In Synapse, we define standard comparison operators as the following set of operators:

  • equal to ( = )

  • less than ( < )

  • greater than ( > )

  • less than or equal to ( <= )

  • greater than or equal to ( >= )

For filter operations, the not equal ( != ) operator is also supported.

When filtering by secondary or extended property value, you can specify the property using either the full property name (i.e., the combined form and property, such as inet:dns:a:ipv4) or the relative property name (i.e., the property name alone, including its separator character, such as :ipv4).

When filtering by universal property value, only the relative property name is required.

Using the relative property name allows for simplified syntax and more efficient data entry (“less typing”). Full property names can be used for clarity (i.e., specifying exactly what you want to filter on).

Full property names are required:

  • when filtering based on an interface property value.

  • in cases where multiple nodes in the inbound working set have the same relative property name (e.g., inet:dns:a:ipv4 and inet:url:ipv4, or a universal property such as .seen) and you only wish to filter based on the property of one of the forms.

Each example below is shown using both the full property name (<form>:<prop>) and the relative property name (:<prop>) where applicable.

Tip

When filtering nodes by a property value where the value is a time (date / time), you do not need to use full YYYY/MM/DD hh:mm:ss.mmm syntax. Synapse allows you to use either lower resolution values (e.g., YYYY/MM/DD) or wildcard values (e.g., YYYY/MM*). In particular, wildcard syntax can be used to specify any values that match the wildcard expression. See the type-specific documentation for time types for a detailed discussion of these behaviors.

Filter by Primary Property Value

Syntax:

<query> + | - <form> <operator> <valu>

Filter the current working set to exclude the loopback IPv4 address (127.0.0.1):

<query> -inet:ipv4 = 127.0.0.1

<query> +inet:ipv4 != 127.0.0.1

Filter by Secondary Property Value

Syntax:

<query> + | - [ <form> ] : <prop> <operator> <pval>

Filter the current working set to include only those domains (inet:fqdn nodes) that are also logical zones:

<query> +inet:fqdn:iszone = 1

<query> +:iszone  = 1

Filter the current working set to exclude any files (file:bytes nodes) with a PE compiled time of 1992-06-19 22:22:17:

<query> -file:bytes:mime:pe:compiled = '1992/06/19 22:22:17'

<query> -:mime:pe:compiled = '1992/06/19 22:22:17'

Filter the current working set to include only those files (file:bytes nodes) compiled in 2019:

<query> +file:bytes:mime:pe:compiled = 2019*

<query> +:mime:pe:compiled = 2019*

Filter thet current working set to exclude those files (file:bytes nodes) whose size is greater than or equal to 1MB:

<query> -file:bytes:size >= 1000000

<query> -:size >= 1000000

Filter by Interface Property Value

If a form inherits an Interface, you can filter all nodes of all forms with a specific interface-derived property value by using the name of the interface.

Syntax:

<query> + | - <interface> : <prop> <operator> <pval>

Examples:

Filter the current working set to only include those Microsoft Office metadata nodes (all nodes of all forms that inherit the file:mime:msoffice interface) whose :author property is ‘admin’:

<query> +file:mime:msoffice:author = admin

Filter the current working set to exclude any host activity nodes (all nodes of all forms that inherit the it:host:activity interface) observed earlier than January 1, 2024:

<query> -it:host:activity:time < 2024/01/01

Filter by Universal Property Value

Synapse has two built-in universal properties:

  • .created (a time) which represents the date and time a node was created in Synapse; and

  • .seen (an interval), a pair of date / time values that can optionally be used to represent when the object represented by a node existed or was observed.

Times (date / time values) are stored as integers (epoch milliseconds) in Synapse and can be filtered using any standard comparison operator.

Because intervals are a pair of date / time values, they can only be filtered using the equal to ( = ) standard comparison operator to specify an exact match for the interval values.

The Filter by Time or Interval (@=) and Filter by Range (*range=) extended comparison operators provide additional flexibility when filtering by times and intervals.

See the time and ival sections of the Storm Reference - Type-Specific Storm Behavior guide for additional details on working with times and intervals in Synapse.

Syntax:

+ | - [ <form> ] . <prop> <operator> <pval>

Filter the current working set to include only those nodes created on January 1, 2024 or later:

<query> +.created >= 2024/01/01

Filter the current working set to include only those FQDNs (inet:fqdn nodes) created on January 1, 2024 or later:

<query> +inet:fqdn.created >= 2024/01/01

Filter the current working set to include only the DNS A records (inet:dns:a nodes) whose .seen property exactly matches the specified interval:

<query> +inet:dns:a.seen = ('2016/06/01 12:22:47.234', '2017/06/10 02:44:55.437')

<query> +.seen = ('2016/06/01 12:22:47.234', '2017/06/10 02:44:55.437')

Filter by Extended Property Value

When filtering by extended property value, you can use any standard comparison operator supported by the property’s type. For example, if the extended property is a string, only the equal to ( = ) standard operator is supported. If the extended property is an integer, any of the standard operators can be used.

Syntax:

+ | - [ <form> ] :_ <prop> <operator> <pval>

Filter the current working set to include only those organizations (ou:org nodes) which are categorized as threats:

<query> +ou:org:_vertex:threatintel:isthreat = true

<query> +:_vertex:threatintel:isthreat = true

Tip

Boolean values can be specified using either true / false or 1 / 0.

Filter the current working set to incldue only those files whose VirusTotal “reputation” score is less than -100:

<query> +file:bytes:_virustotal:reputation < -100

<query> +:_virustotal:reputation < -100

Filter by Property Value - Extended Comparison Operators

Storm supports a set of extended comparison operators (comparators) for specialized filter operations. In most cases, the same extended comparators are available for both lifting and filtering:

Each extended comparision operator can be used with any kind of property (primary, secondary, universal, or extended) whose Type is appropriate for the comparison used. When filtering by secondary property value, you can optionally specify an Interface name and property to filter based on all forms that inherit that interface.

Filter by Regular Expression (~=)

The extended comparator ~= is used to filter nodes based on PCRE-compatible regular expressions.

Tip

Filter by Prefix (^=) can be used to filter based on the beginning of string-based properties, and is more efficient for beginning-of-string filter operations. It should be used instead of a regular expression filter where possible.

Syntax:

<query> + | - <form> ~= <regex>

<query> + | - [ <form> ] : | . | :_ <prop> ~= <regex>

<query> + | - <interface> : <prop> ~= <regex>

Examples:

Filter the current working set to include only files (file:bytes nodes) with a PDB path containing the string ‘tekide’:

<query> +file:bytes:mime:pe:pdbpath ~= tekide

<query> +:mime:pe:pdbpath ~= tekide

Filter the current working set to exclude organizations (ou:org nodes) whose name contains a string that starts with “v”, followed by 0 or more characters, followed by “x”:

<query> -ou:org:name ~= '^v.*x'

<query> -:name ~= '^v.*x'

Filter the current working set to only include taxonomy nodes (all nodes of all forms that inherit the meta:taxonomy interface) whose description (:desc property) includes the string ‘credential’:

<query> +meta:taxonxomy:desc ~= credential

Filter by Prefix (^=)

Synapse performs prefix indexing on strings and string-derived types, which optimizes filtering nodes whose <valu> or <pval> starts with a given prefix (substring). The extended comparator ^= is used to filter nodes by prefix.

Note

Extended string types that support dotted notation (such as the loc or syn:tag types) have custom behaviors with respect to lifting and filtering by prefix.

inet:fqdn nodes are indexed in reverse string order so cannot be lifted using the prefix extended operator. However, they can be filtered by prefix.

See the relevant sections in the Storm Reference - Type-Specific Storm Behavior guide for details.

Syntax:

<query> + | - <form> ^= <prefix>

<query> + | - [ <form> ] : | . | :_ <prop> ^= <prefix>

<query> + | - <interface> : <prop> ^= <prefix>

Examples:

Filter the current working set to exclude email addresses (inet:email nodes) that start with “abuse”:

<query> -inet:email ^= abuse

Filter the current working set to only include organizations (ou:org nodes) whose name starts with “ministry”:

<query> +ou:org:name ^= ministry

<query> +:name ^= ministry

Filter the current working set to only include Microsoft Office metadata nodes (all nodes of all forms that inherit the file:mime:msoffice interface) whose :author property starts with ‘Admin’:

<query> +file:mime:msoffice:author ^= Admin

Filter by Time or Interval (@=)

The time extended comparator (@=) is used to filter nodes based on comparisons among various combinations of times and intervals.

Tip

See Storm Reference - Type-Specific Storm Behavior for additional detail on the use and behavior of time and ival data types.

Syntax:

<query> + | - [ <form> ] : | . | :_ <prop> @=( <ival_min> , <ival_max> )

<query> + | - [ <form> ] : | . | :_ <prop> @= <time>

<query> + | - <interface> : <prop> @=( <ival_min> , <ival_max> )

<query> + | - <interface> : <prop> @= <time>

Examples:

Filter the current working set to include only those DNS A records (inet:dns:a nodes) whose .seen values fall between July 1, 2022 and and August 1, 2022:

+inet:dns:a.seen @= ( 2022/07/01, 2022/08/01 )

+.seen @= ( 2022/07/01, 2022/08/01 )

Filter the current working set to only include DNS requests (inet:dns:request nodes) that occurred on May 3, 2023 (between 05/03/2023 00:00:00 and 05/03/2023 23:59:59):

+inet:dns:request:time @= ( '2023/05/03 00:00:00', '2023/05/04 00:00:00' )

+:time @= ( '2023/05/03 00:00:00', '2023/05/04 00:00:00' )

Tip

Because the inet:dns:request:time property is a single date / time value, the following filters would also work:

  • +inet:dns:request:time = 2023/05/03*

  • +:time = 2023/05/03*

Filter the current working set to only include DNS A records (inet:dns:a nodes) whose resolution time window includes the date December 1, 2023:

<query> +inet:dns:a.seen @= 2023/12/01

<query> +.seen @= 2023/12/01

Filter results to include only those domain WHOIS records (inet:whois:rec nodes) where the domain was registered (created) exactly on March 19, 2019 at 5:00 UTC:

<query> +inet:whois:rec:created @= '2019/03/19 05:00:00'

<query> +:created @= '2019/03/19 05:00:00'

Note

When comparing a single time value to a time property, the @= comparator behaves just like the equal to ( = ) operator.

Filter the current working set to only include WHOIS email nodes (inet:whois:email) that were observed between July 1, 2023 and the present:

<query> +inet:whois:email.seen @= ( 2023/07/01, now )

<query> +.seen @= ( 2023/07/01, now )

Filter the current working set to only include the network flows (inet:flow nodes) that occurred within the past day:

<query> +inet:flow:time @= ( now, '-1 day' )

<query> +:time @= ( now, '-1 day' )

Filter the current working set to only include the host activity nodes (all nodes of all forms that inherit the it:host:activity interface) whose :time value is within the past three hours:

<query> +it:host:activity:time @= (now, '-3 hours')

Usage Notes:

  • When specifying an interval with the @= operator, the minimum value is included in the interval for comparison purposes but the maximum value is not. This is equivalent to “greater than or equal to <min> and less than <max>”. This behavior differs from that of the *range= operator, which includes both the minimum and maximum.

  • Comparing intervals to intervals: when using an interval with the @= operator to filter nodes based on an interval property, Synapse returns nodes whose interval value has any overlap with the specified interval. For example:

    • A lift interval of September 1, 2018 to October 1, 2018 ( 2018/09/01, 2018/10/01 ) will match nodes with any of the following intervals:

      • August 12, 2018 to September 6, 2018 ( 2018/08/12, 2018/09/06 ).

      • September 13, 2018 to September 17, 2018 ( 2018/09/13, 2018/09/17 ).

      • September 30, 2018 to November 5, 2018 ( 2018/09/30, 2018/11/05 ).

  • Comparing intervals to times: When using an interval with the @= operator to lift nodes based on a time property, Synapse returns nodes whose time value falls within the specified interval.

  • Comparing times to times: When using a time with the @= operator to filter nodes based on a time property, Synapse returns nodes whose timestamp is an exact match of the specified time. In other words, in this case the interval comparator ( @= ) behaves like the equal to comparator ( = ).

  • When specifying date / time and interval values, Synapse allows the use of both lower resolution values (e.g., YYYY/MM/DD), and wildcard values (e.g., YYYY/MM*). Wildcard time syntax may provide a simpler and more intuitive means to specify some intervals. For example inet:whois:rec:asof=2018* is equivalent to inet:whois:rec:asof@=('2018/01/01', '2019/01/01').

  • Time-based keywords (such as now) and relative time syntax (expressions such as +-1 hour or -7 days) can be used for interval values.

    See the type-specific documentation for time and ival types for a detailed discussion of these behaviors.

Filter by Range (*range=)

The range extended comparator (*range=) supports filtering nodes whose <form> = <valu> or <prop> = <pval> fall within a specified range of values. The comparator can be used with types such as integers and times.

Note

The *range= operator can be used to filter both inet:ipv4 and inet:ipv6 values (which are stored as decimal integers and strings, respectively). However, ranges of inet:ipv4 and inet:ipv6 nodes can also be filtered directly by specifying the lower and upper addresses in the range using <min>-<max> format. For example:

  • +inet:ipv4 = 192.168.0.0-192.168.0.10

  • +:ipv4 = 192.168.0.0-192.168.0.10

Because IPv6 nodes are stored as strings, the range must be enclosed in quotes:

  • +inet:ipv6 = "::0-ff::ff"

  • +:ipv6 = "::0-ff::ff"

The *range= operator cannot be used to compare a time range with a property value that is an interval (ival type). The interval ( @= ) operator should be used instead.

Syntax:

<query> + | - <form> *range = ( <range_min> , <range_max> )

<query> + | - [ <form> ] : | . | :_ <prop> *range = ( <range_min> , <range_max> )

<query> + | - <interface> : <prop> *range = ( <range_min> , <range_max> )

Examples:

Filter the current working set to exclude files (file:bytes nodes) whose size is between 1000 and 100000 bytes:

<query> -file:bytes:size *range= ( 1000, 100000 )

<query> -:size *range= ( 1000, 100000 )

Filter the current working set to only include files (file:bytes nodes) whose VirusTotal “reputation” score is between -20 and 20:

<query> +file:bytes:_virustotal:reputation *range= ( -20, 20 )

<query> +:_virustotal:reputation *range= ( -20, 20 )

Filter the current working set to exclude domain WHOIS records (inet:whois:rec nodes) that were captured / retrieved between November 29, 2013 and June 14, 2016:

<query> -inet:whois:rec:asof *range= ( 2013/11/29, 2016/06/14 )

<query> -:asof *range= ( 2013/11/29, 2016/06/14 )

Filter the current working set to only include DNS requests (inet:dns:request nodes) made within one day of December 1, 2021:

<query> +inet:dns:request:time *range= ( 2021/12/01, '+-1 day' )

<query> +:time *range= ( 2021/12/01, '+-1 day' )

Filter the current working set to only include taxonomy nodes (all nodes of all forms that inherit the meta:taxonomy interface) whose :depth is between 1 and 3 (i.e., between 2 and 4 taxonomy elements):

<query> +meta:taxonomy:depth *range= (1, 3)

Usage Notes:

  • When specifying a range (*range=), both the minimum and maximum values are included in the range (the equivalent of “greater than or equal to <min> and less than or equal to <max>”). This behavior is slightly different than that for time interval (@=), which includes the minimum but not the maximum.

  • When specifying a range of time values, Synapse allows you to use either lower resolution values (e.g., YYYY/MM/DD) or wildcard values (e.g., YYYY/MM*) for the minimum and/or maximum range values. In some cases, plain wildcard time syntax may provide a simpler and more intuitive means to specify some time ranges. For example +inet:whois:rec:asof=2018* (or +:asof=2018*) is equivalent to +inet:whois:rec:asof*range=('2018/01/01', '2018/12/31 23:59:59.999') (or +:asof*range=('2018/01/01', '2018/12/31 23:59:59.999')). See the type-specific documentation for time types for a detailed discussion of these behaviors.

  • When using keywords (such as now) or relative values (such as -1 hour) to specify a range of times, the first value in the range is calculated relative to the current time and the second value is calculated relative to the first value.

  • If you specify a range value that is nonsencical or exclusionary (such as ( 47, 16 )), Synapse will not generate an error and will simply fail to return results. (The expression is syntacticaly correct, but no value is both greater than 47 and less than 16).

Filter by Set Membership (*in=)

The set membership extended comparator (*in=) supports filtering nodes whose <form> = <valu> or <prop> = <pval> matches any of a set of specified values. The comparator can be used with any type.

Syntax:

<query> + | - <form> *in = ( <set_1> , <set_2> , … )

<query> + | - [ <form> ] : | . | :_ <prop> *in = ( <set_1> , <set_2> , … )

<query> + | - <interface> : <prop> *in = ( <set_1> , <set_2> , … )

Examples:

Filter the current working set to exclude organization names (ou:name nodes) matching any of the specified values:

<query> -ou:name *in= ( fsb, gru, svr )

Filter the current working set to only include IPv4 addresses associated with any of the specified Autonomous System (AS) numbers:

<query> +inet:ipv4:asn *in= ( 9009, 20473, 44477 )

<query> +:asn *in= ( 9009, 20473, 44477 )

Filter the current working set to only include tags (syn:tag nodes) whose final tag element matches any of the specified string values:

<query> +syn:tag:base *in= ( plugx, korplug, sogu, kaba )

<query> +:base *in= ( plugx, korplug, sogu, kaba )

Filter by Proximity (*near=)

The proximity extended comparator (*near=) supports filtering nodes by “nearness” to another node. Currently, *near= supports proximity based on geospatial location (i.e., nodes within a given radius of a specified latitude / longitude).

Syntax:

<query> + | - [ <form> ] : | . | :_ <prop> *near = (( <lat> , <long> ), <radius> )

Examples:

Filter the current working set to only include locations (geo:place nodes) within 500 meters of the Russian Cryptographic Museum (where the coordinates 55.83069,37.59781 represent the Museum’s location):

<query> +geo:place:latlong *near= ( (55.83069, 37.59781), 500m )

Usage Notes:

  • In the example above, the latitude and longitude of the desired location are explicitly specified as parameters to *near=.

  • Radius can be specified in the following units. The values in parentheses are the acceptable terms for specifying a given unit:

    • Kilometers (km / kilometer / kilometers)

    • Meters (m / meter / meters)

    • Centimeters (cm / centimeter / centimeters)

    • Millimeters (mm / millimeter / millimeters)

    • Miles (mile / miles)

    • Yards (yard / yards)

    • Feet (foot / feet)

  • Radius values of less than 1 must be specified with a leading zero (e.g., 0.5 km).

  • The *near= comparator works for geospatial data by lifting nodes within a square bounding box centered at <lat>,<long>, then filters the nodes returned by ensuring that they are within the great-circle distance given by the <radius> argument.

Filter by (Arrays) (*[ ])

Storm uses a special syntax to filter (or lift) by comparison with one or more elements of an array type. The syntax consists of an asterisk ( * ) preceding a set of square brackets ( [ ] ), where the square brackets contain a comparison operator and a value that can match one or more elements in the array. This allows users to match any value in the array list without needing to know the exact order or values of the array itself.

Note

When filtering based on a value in an array property, you must use the relative name of the property. The full property name (i.e., the combined form and property) is not supported for this type of filter.

Syntax:

<query> + | - : | . | :_ <prop> *[ <operator> <pval> ]

Examples:

<query> +:identities:fqdns *[ = '*.xyz' ]

Filter the current working set to exclude the MITRE ATT&CK groups (it:mitre:attack:group nodes) whose names include the string ‘bear’:

<query> -:names *[ ~= bear ]

Usage Notes:

  • The comparison operator used must be valid for filter operations for the type used in the array.

  • The standard equals ( = ) operator can be used to filter nodes based on array properties, but the value specified must exactly match the full property value in question:

    • For example: ou:org +:names=("the vertex project","the vertex project llc",vertex) will filter to any ou:org nodes whose :names property consists of exactly those names in exactly that order.

  • See the array section of the Storm Reference - Type-Specific Storm Behavior document for additional details on working with arrays.

Tag Filters

Tags in Synapse can represent observations or assessments. They are used to provide context to nodes (in the form of “labels” applied to nodes) and to group related nodes.

Storm supports filtering nodes based on the tags applied to nodes (including the use of tag globs), as well as filtering based on tag timestamps, tag properties, or tag property values.

The “hashtag” symbol ( # ) is used to specify a tag name when filtering by tag.

Filter by Tag (#)

A “filter by tag” operation downselects the current working set to include (or exclude) all nodes with the specified tag.

Syntax:

<query> + | - # <tag>

Filter the current working set to exclude all nodes that ESET associates with Sednit:

<query> -#rep.eset.sednit

Filter the current working set to only include nodes associated with anonymized infrastructure:

<query> +#cno.infra.anon

Tip

Tags are hierarchical, and each tag element is its own tag; the tag #cno.infra.anon consists of the tags #cno, #cno.infra, and #cno.infra.anon. Filtering nodes using a tag “higher up” in the tag hierarchy will include (or exclude) nodes with the specified tag or any tag “lower down” in the hierarchy. In other words, filtering by #cno.infra.anon will filter all “anonymized” infrastructure, whether the infrastructure is a VPN (#cno.infra.anon.vpn), a TOR node (#cno.infra.anon.tor), or an anonymous proxy (#cno.infra.anon.proxy).

Filter by Tag Globs

Synapse supports filtering based on the set of tags that match a specified glob expression using single ( * ) or double ( ** ) asterisks, or a combination of the two.

The single asterisk and double asterisk behave differently:

  • The asterisk ( * ) represents an arbitrary string that matches within a single tag element (i.e., one element as bounded by the tag’s “dot” ( . ) separators).

  • The double asterisk ( ** ) represents an arbitrary string match anywhere in the tag, including across tag elements.

Another way to look at this is that the single asterisk is constrained by the tag’s “dot” boundaries, but the double asterisk is not.

Syntax:

<query> + | - # <string> | * | ** [ . <string> | * | ** … ]

Examples:

Filter the current working set to exclude any nodes tagged as “seduploader” by any third-party reporting organization:

<query> -#rep.*.seduploader

To record assessments made by third parties, The Vertex Project uses rep (“reported by”) as the root tag element, followed by a tag element for the reporting organization (e.g., rep.eset), followed by the name of the “thing” reported (in this case, the seduploader malware family).

The tag glob filter above uses the single asterisk to match any tag element in the second position, and will match tags such as:

  • rep.eset.seduploader

  • rep.paloalto.seduploader

…etc.

Filter the current working set to include any nodes tagged as “cobaltstrike” by any third-party reporting organization whose name begins with ‘m’:

<query> +#rep.m*.cobaltstrike

The tag glob filter above uses the single asterisk to match any partial tag element in the second position that starts with ‘m’, and will match tags such as:

  • rep.malwarebazaar.cobaltstrike

  • rep.mandiant.cobaltstrike

  • rep.microsoft.cobaltstrike

Tip

The filter above would not match on a tag such as rep.malwarebazaar.3p.anyrun.cobaltstrike, because the string cobaltstrike is not the third tag element. A double asterisk, which matches across a tag’s “dot” boundaries, would match this tag as well as the example tags above:

rep.m**cobaltstrike

Filter the current working set to exclude any nodes tagged as “seduploader” either internally or by any third-party reporting organization:

<query> -#*.*.seduploader

The Vertex Project uses the cno root tag to represent our own internal assessments (and distinguish them from third party-assessments), and the mal element to represent assessments related to malware. The tag glob filter above uses two single asterisks to match any element in both the first and second positions, and will match all of the following:

  • rep.eset.seduploader

  • rep.paloalto.seduploader

  • cno.mal.seduploader

…etc.

Filter the current working set to include any nodes reported by Microsoft whose tags end in “blizzard”:

<query> +#rep.microsoft.**blizzard

The tag glob filter above uses a double asterisk to match any Microsoft tag (tag that begins rep.microsoft) that ends in “blizzard”, regardless of tag depth. The filter will match all of the following:

  • rep.microsoft.aqua_blizzard

  • rep.microsoft.cadet_blizzard

  • rep.microsoft.forest_blizzard

  • rep.microsoft.very.long.tag.thatendswithblizzard

…etc.

Filter the current working set to exclude any nodes tagged with any tag that starts with “cno” and is followed by any string:

<query> -#cno**

The tag glob filter above uses a double asterisk to match any string following “cno”. The filter will match all of the following:

  • cno.mal

  • cno.threat.t42

  • cnoooo.you_get_a_cno.and_you_get_a_cno

…etc.

Note

The double asterisk must match “something” - the filter above will not match a node that simply has the tag #cno.

Filter the current working set to include any nodes tagged by any third-party reporting organization where the tag contains the string “2017”:

<query> +#rep.*.**2017**

The tag glob filter above uses both a single and double asterisk. The single asterisk matches any tag element in the second position; the double asterisk matches any string that includes “2017”, including across “dot” boundaries. The filter will match all of the following:

  • rep.alienvault.cve20178291

  • rep.malwarebazaar.3p.reversinglabs.document_ole_exploit_cve_2017_11882

  • rep.vt.cve_2017_0199

Note

The double asterisk must match “something” - the filter above matches strings where “2017” appears between other arbitrary characters. The filter would not match tags such as rep.foo.2017 or rep.bar.baz.cve2017.

Filter Using Tag Timestamp Values

A tag timestamp can be thought of as a specialized “property” of a tag that happens to be a date / time range (interval). You can filter nodes based on tag timestamp values using any comparison operator supported by interval (ival types). The time / interval extended operator ( @= ) is used most often, but equal to ( = ) can also be used to exactly match the values in the interval.

See Filter by Time or Interval (@=) for additional detail on the use of the @= operator.

Syntax:

<query> + | - # <tag> @= <time> | ( <min_time> , <max_time> )

Filter the current result set to only include nodes that were associated with anonymous VPN infrastructure between December 1, 2023 and January 1, 2024:

<query> +#cno.infra.anon.vpn @= ( 2023/12/01, 2024/01/01 )

Filter the current working set to only include nodes that were owned / controlled by Threat Cluster 15 as of October 30, 2021:

<query> +#cno.threat.t15.own @= 2021/10/30

Filter Using Tag Properties

Tag Properties can be used to provide additional context to tags. Storm supports filtering nodes whose tags have a specific tag property (regardless of the value of the property).

Note

In many cases, information previously recorded using a tag property is better suited to the use of an Extended Property.

Syntax:

<query> + | - # <tag> | * : <tagprop>

Filter the current working set to only include nodes with a “:risk” property reported by Symantec:

<query> +#rep.symantec:risk

Filter the current working set to include nodes with a “:risk” property associated with any tag:

<query> +#**:risk

Tip

When filtering based on the existence of a tag property, you can use tag glob syntax (see Filter by Tag Globs) to specify the associated tags. Filters such as +#rep.*.*:risk or (per the above example) +#**:risk are supported.

When filtering for a specific tag property that appears on any tag, either a double asterisk (tag glob, as above) or single asterisk can be used, e.g.: +#*:risk. The single asterisk in this instance is not a tag glob, but a special syntax helper for this specific use case. (That is, because entering +#*:risk instead of +#**:risk is a common user error, Synapse automatically handles this case to “do what you mean”.)

Filter Using Tag Property Values

Storm supports filtering nodes based on the value of a tag property (similar to filtering by the value of a node property).

You can filter nodes based on tag property values using any comparison operator supported by the property’s Type. For example, if the tag property is defined as an integer (int) type, you can use any comparison operator supported by integers.

Note

Tag glob syntax (see Filter by Tag Globs) is not supported when filtering based on a tag property value. For example, a filter such as +#rep.**:risk>20 will generate a syntax error.

Syntax:

<query> + | - # <tag> : <tagprop> <operator> <pval>

Filter the current working set to include nodes with a “:risk” property value of 100 as reported by ESET:

<query> +#rep.eset:risk = 100

Filter the current working set to exclude nodes with a “:risk” property value less than 90 as reported by domaintools:

<query> -#rep.domaintools:risk < 90

Filter the current working set to include nodes with a “:risk” property with a value between 45 and 70 as reported by Symantec:

<query> +#rep.symantec:risk *range= ( 45, 70 )

Compound Filters

Storm supports the use of the logical operators and, or, and not (including and not) to construct compound filters. You can use parentheses to group portions of the filter statement to indicate order of precedence and clarify logical operations when evaluating the filter.

Note

  • Logical operators must be specified in lower case.

  • Synapse evalutes compound filters in order from left to right. Depending on the filter, left-to-right order may differ from the standard Boolean order of operations (not then and then or).

  • Parentheses should be used to logically group portions of the filter statement if necessary to clarify order of operations.

Syntax:

<query> + | - ( <filter> and | or | not | and not … )

Examples:

Filter the current working set to exclude files (file:bytes nodes) that are less than or equal to 16384 bytes in size and were compiled prior to January 1, 2014:

<query> -(file:bytes:size <= 16384 and file:bytes:mime:pe:compiled < 2014/01/01)

<query> -(:size <= 16384 and :mime:pe:compiled < 2014/01/01)

Filter the current working set to only include files (file:bytes nodes) or domains (inet:fqdn nodes) that ESET associates with Sednit:

<query> +( ( file:bytes or inet:fqdn ) and #rep.eset.sednit )

Filter the current working set to include only files (file:bytes nodes) and domains (inet:fqdn nodes) that ESET associates with Sednit that are **not* sinkholed:*

<query> +( ( file:bytes or inet:fqdn ) and ( #rep.eset.sednit and not #cno.infra.dns.sinkhole ) )

Subquery Filters

You can use Storm’s subquery syntax (Storm Reference - Subqueries) to create filters. A subquery (enclosed in curly braces ( { } ) ) can be placed within a larger Storm query.

Most filter operations in Storm will modify (reduce) your current set of nodes based on some criteria of the nodes themselves (e.g., a node’s form, property, or tag).

Subquery filters allow you to filter your current set of nodes based on some criteria of nearby nodes. You use the subquery filter to effectively “look ahead” at nodes one or more pivots away from your current nodes, and filter your current nodes based on the properties of those “nearby” nodes.

When nodes are passed to a subquery filter, they are evaluated against the filter’s criteria:

  • Nodes are excluded (“consumed”, discarded) if they evaluate false.

  • Nodes are included (not “consumed”, retained) if they evaluate true.

The subquery pivot operation (used to “look ahead” at other nodes) is effectively performed in the background (without navigating away from your current working set), which provides a more powerful and efficent way to filter your data. (The alternative would be to actually navigate to the nearby nodes, filter those nodes, and then navigate back to the data you are interested in.)

You can optionally use a standard (mathematical) comparison operator with a subquery filter, in order to filter your current set of nodes based on the number of results returned by executing the subfilter’s Storm query.

Refer to the Storm Reference - Subqueries guide for additional information on subqueries and subquery filters.

Syntax:

<query> + | - { <query> } [ <standard operator> <value> ]

Examples:

Filter the current working set of FQDNs (inet:fqdn nodes) to only FQDNs that have resolved to an IPv4 address that Trend Micro associates with Pawn Storm (i.e., an IP address tagged #rep.trend.pawnstorm):

<inet:fqdn> +{ -> inet:dns:a -> inet:ipv4 +#rep.trend.pawnstorm }

The subquery filter above takes the inbound inet:fqdn nodes and (within the subquery):

  • pivots to the associated DNS A records (inet:dns:a nodes);

  • pivots to the asssociated IPv4 addresses (inet:ipv4 nodes);

  • checks the IPv4 for the presence of a #rep.trend.pawnstorm tag.

The subquery filter returns only those inet:fqdn nodes where, if you performed the operations within the subquery, would (based on the inclusive filter) result in an inet:ipv4 node with a #rep.trend.pawnstorm tag.

Filter the current working set of IPv4 addresses (inet:ipv4 nodes) to exclude any IPv4 associated with an Autonomous System (AS) whose name starts with “makonix”:

<inet:ipv4> -{ :asn -> inet:asn +:name ^= makonix }

The subquery filter above takes the inbound inet:ipv4 nodes and (within the subquery):

  • pivots to the associated inet:asn nodes; and

  • checks the inet:asn nodes for a :name value that starts with “makonix”.

The subquery filter returns only those inet:ipv4 nodes where, if you performed the operations within the subquery, would not (based on the exclusive filter) result in an inet:asn node with a :name value starting with “makonix”.

Tip

See Embedded Property Syntax for an alternative way to perform this query.

Filter the current working set of files (file:bytes nodes) to include only files that are detected as malicious in (10) or more scans (i.e., files that are associated with 10 or more it:av:scan:result nodes whose :verdict property value is “malicious”):

<file:bytes> +{ -> it:av:scan:result +:verdict=malicious }>=10

The subquery filter above takes the inbound file:bytes nodes and (within the subquery):

  • pivots to the associated it:av:scan:result nodes; and

  • filters the results to include only those nodes whose it:av:scan:result:verdict property value is malicious; and

  • counts the number of resulting it:av:scan:result nodes for each file.

The subquery filter returns only those file:bytes nodes with 10 or more associated it:av:scan:result nodes with a malicious verdict.

Tip

This is a simplified example. it:av:scan:result nodes represent a scan performed at a given point in time; the filter above does not provide any time constraints so will count any / all “malicious” results, regardless of “when” the scan was performed. Results could include files detected as malicious by ten different vendors during a single scan as well as files detected as malicious by only one vendor during ten different scans.

Filter the current working set of x509 certificates (crypto:x509:cert nodes) to only include certificates linked to more than one FQDN (inet:fqdn) identity:

<crypto:x509:cert> +{ :identities:fqdns -> inet:fqdn }>1

The subquery filter above takes the inbound crypto:x509:cert nodes and (within the subquery):

  • uses the :identities:fqdns array property to pivot to any associated FQDNs (inet:fqdn nodes); and

  • counts the number of inet:fqdn nodes associated with each certificate.

The subquery filter returns only those crypto:x509:cert nodes associated with more than one FQDN.

Tip

See Expression Filters below for an alternative way to perform this query.

Expression Filters

An expression filter is used to downselect your current working set based on the evaluation of a particular expression. Expression filters are useful when:

  • you need to compute a value that you want to use for the filter, or

  • when you want to filter based on a value that may change (e.g., when using Storm queries that assign variables; see Storm Reference - Advanced - Variables).

Syntax:

<query> + | - $( <expression> )

Examples:

Filter the current working set of x509 certificates (crypto:x509:cert nodes) to only include certificates linked to more than one FQDN (inet:fqdn) identity:

<crypto:x509:cert> $fqdns=:identities:fqdns +$( $fqdns.size() > 1 )

This example assigns the list of domains in the crypto:x509:cert:identities:fqdns property to the user-defined variable $fqdns, computes the number of domains in the list using size(), and checks to see if the result is greater than 1.

(See the Storm Library Documentation for additional detail on Storm types and Storm libraries.)

Tip

This certificate example is identical to the final example under Subquery Filters above, and shows an alternative way to return the same data.

This expression filter is more efficient than the subquery filter because the expression filter simply evaluates the expression (“what is the size of the :identities:fqdns array property?”), where the subquery filter needs to pivot to the adjacent nodes in order to evaluate the results. This difference in performance is negligible for small data sets but more pronounced when working with large numbers of nodes.

Filter the current working set of network flows (inet:flow nodes) to only include flows where the total number of bytes transferred in the flow between the source (inet:flow:src:txbytes) and destination (inet:flow:dst:txbytes) is greater than 100MB (~100,000,000 bytes):

<inet:flow> +$( :src:txbytes + :dst:txbytes >=100000000 )

Filter the current set of nodes associated with any threat group or threat cluster (e.g., tagged ``#cno.threat.<threat_name>``), to include only those nodes that are attributed to more than one threat (e.g., that have more than one #cno.threat.<threat_name> tag):

#cno.threat +$( $node.globtags(cno.threat.*).size() > 1 )

This query may identify nodes that are incorrectly attributed to more than one group, or instances where two or more threat clusters overlap (which may indicate that the clusters actually represent a single set of activity).

This example uses the $node.globtags() method to select the set of tags on each node that match the specified expression (cno.threat.*) and size() to count the number of matches.

Filter the current working set of DNS A records (inet:dns:a nodes) to only include those whose .seen interval falls WITHIN the past 30 day window (e.g., where the <min> value of the .seen interval is greater than or equal to the date 30 days in the past:

$ival = $lib.cast( ival, ( now, -30 days ) )
( $start, $stop ) = $ival
inet:dns:a.seen @= $ival
( $min, $max ) = .seen
+$( $min >= $start )

The interval comparison operator ( @= ) will lift or filter interval properties (such as .seen) if the node’s interval has any overlap with the comparison value. Using current Storm syntax, this means it is not possible to directly lift or filter for interval values that fall within the comparison interval value.

The above query uses variables (see Storm Reference - Advanced - Variables) to calculate the date/time exactly 30 days prior to the current date/time ( $start and $stop ) and uses an expression filter to ensure that the <min> value of the node’s .seen property is more recent than “30 days ago”.

The query is repeated here with inline comments to note what each line is doing:

$ival = $lib.cast( ival, ( now, -30 days ) )  // Set the variable $ival to a pair of date/time values specified
                                              // using the keyword "now" and the relative value "-30 days".
( $start, $stop ) = $ival                     // Set the variables $start and $stop to the individual date/times
                                              // from $ival.
inet:dns:a.seen @= $ival                      // Lift all inet:dns:a nodes whose .seen property has any **overlap**
                                              // with the past 30 days.
( $min, $max ) = .seen                        // Set the variables $min and $max to the individual date/times of the
                                              // .seen interval
+$( $min >= $start )                          // Use an expression filter to ensure that the $min time of the node's
                                              // .seen value is greater than or equal to the $start time of "30 days ago".

Embedded Property Syntax

Storm includes a shortened syntax consisting of two colons (::) that can be used to reference a secondary property of an adjacent node. Because the syntax can be used to “pull in” a property or property value from a nearby node, it is known as “embedded property syntax”.

Embedded property syntax expresses something that is similar (in concept, though not in practice) to a secondary-to-secondary property pivot (see Storm Reference - Pivoting). The syntax expresses navigation:

  • From a secondary property of a form (such as inet:ipv4:asn), to

  • The form for that secondary property (i.e., inet:asn), to

  • A secondary property (or property value) of that target form (such as inet:asn:name).

Tip

This process can be repeated to reference properties of forms more than one pivot away.

Despite its similarity to a pivot operation, embedded property syntax is commonly used for:

Syntax:

<query> [ + | - ] : <prop> :: <prop> [ :: <prop> … ]

<query> [ + | - ] : <prop> [ :: <prop> … ] :: <prop> <operator> <pval>

Note

When using embedded property syntax in Storm, the leading colon (before the name of the initial secondary property) is required - e.g., :asn::name.

When using this syntax in Optic (the Synapse UI) to create an embed column in Tabular display mode, the initial colon should be omitted - e.g, asn::name. Optic effectively prepends the initial colon for you.

Filter Examples:

The examples below illustrate the use of embedded property syntax in a filter expresssion.

Filter the current working set of IPv4 addresses (inet:ipv4 nodes) to exclude any IPv4 associated with an Autonomous System (AS) whose name starts with “makonix”:

<inet:ipv4> -:asn::name ^= makonix

Tip

This example is an alternative way to return the same data as the second example under Subquery Filters above:

<inet:ipv4> -{ :asn -> inet:asn +:name ^= makonix }

Filter the current working set of sandbox “file add” operations (it:exec:file:add nodes) to only those “add” operations performed by a file that has a PDB path (:mime:pe:pdbpath property):

<it:exec:file:add> +:sandbox:file::mime:pe:pdbpath

The :sandbox:file property of an it:exec:file:add node represents the file (file:bytes node) that was executed in the sandbox environment. If this filter were written as a subquery filter, the pivot syntax within the subquery would look like this:

<it:exec:file:add> +{ :sandbox:file -> file:bytes +:mime:pe:pdbpath }

Instead, embedded property syntax is used to represent the pivot from the :sandbox:file property of the it:exec:file:add node, to the associated file:bytes node, to the file’s :mime:pe:pdbpath property.

Filter the current working set of sandbox “file add” operations (it:exec:file:add nodes) to only those “add” operations performed by a self-extracting RAR file (i.e., a file with a PDB path whose base file name is sfxrar.pdb):

<it:exec:file:add> +:sandbox:file::mime:pe:pdbpath::base = sfxrar.pdb

This example expands on the previous example to use two instances of embedded property syntax. If this filter were written as a subquery filter, the pivot syntax within the subquery would look like this:

<it:exec:file:add> +{ :sandbox:file -> file:bytes :mime:pe:pdbpath -> file:path +:base = sfxrar.pdb }

Instead, embedded property syntax is used to represent the pivots from:

  • the :sandbox:file property of the it:exec:file:add node, to the associated file:bytes node, to its :mime:pe:pdbpath property; and

  • the :mime:pe:pdbpath property of the file:bytes node, to the associated file:path node, to its :base property.

Variable Assignment Example:

Embedded property syntax can also be used when assigning variables (see Storm Reference - Advanced - Variables).

Set the variable $name to the name of the Autonomous System (AS) associated with a given IPv4 address:

<inet:ipv4> $name=:asn::name

This example uses embedded property syntax to pivot from the inbound inet:ipv4 node, to the ASN (inet:asn node) associated with the IPv4’s :asn property, and assigns the value of the ASN’s :name property to the variable $name.