Storm Reference - Pivoting

Pivot operations are performed on the output of a Storm query. Pivot operators are used to navigate from one set of nodes to another based a specified relationship. The pivot operations available within Storm are:

Note

When pivoting from a secondary property (<prop> = <pval>), the secondary property must be specified using the relative property name only (:baz vs. foo:bar:baz). Specifying the full property name before the pivot would be interpreted as an additional lift (i.e., <inet:dns:a> inet:dns:a:fqdn -> inet:fqdn would be interpreted as “take a set of inet:dns:a records from an initial query, lift all inet:dns:a records with an :fqdn property (i.e., every inet:dns:a node in the Cortex), and then pivot to the associated inet:fqdn nodes”).

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below.

See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

Pivot Out Operator

The pivot out operator ( -> ) is the primary Storm pivot operator. The pivot out operator pivots from a primary or secondary property value of the current set of nodes to a primary or secondary property value of another set of nodes.

The pivot out operator is used to:

  • pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes,
  • pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes,
  • pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and
  • pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes.

Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes are covered separately below.

Syntax:

<query> -> <form> : <prop>

<query> : <prop> -> <form>

<query> -> *

<query> : <prop> -> <form> : <prop>

Examples:

Pivot from primary property (<form> = <valu>) to secondary property (<prop> = <pval>):

  • Pivot from a set of domains to all of their subdomains regardless of depth:
<inet:fqdn> -> inet:fqdn:zone
  • Pivot from a set of domains to their DNS A records:
<inet:fqdn> -> inet:dns:a:fqdn

Pivot from secondary property (<prop> = <pval>) to primary property (<form> = <valu>):

  • Pivot from a set of DNS A records to the resolution IP addresses contained in those records:
<inet:dns:a> :ipv4 -> inet:ipv4

Pivot from all secondary properties to all forms (<prop> = <pval> to <form> = <valu>):

  • Pivot from a set of WHOIS records to all nodes whose primary property equals any of the secondary properties of the WHOIS record (the asterisk * is a wildcard that indicates pivot to any applicable node):
<inet:whois:rec> -> *

Pivot from secondary property (<prop> = <pval>) to secondary property (<prop> = <pval>):

  • Pivot from the WHOIS records for a set of domains to the DNS A records for the same domains:
<inet:whois:rec> :fqdn -> inet:dns:a:fqdn

Usage Notes:

  • Pivoting out using the asterisk wildcard ( * ) is sometimes called a refs out pivot because it pivots from all secondary properties of the inbound nodes to all nodes referenced by those properties. That is, for each inbound node, the “refs out” pivot will pivot from the node’s secondary properties to all the nodes that have a primary property equal to that type and value.
  • Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot out to properties that match both <type> and <valu> / <pval>. This means that the following nodes will not be returned by a wildcard pivot out:
    • Nodes with matching <valu> / <pval> but of different <type>. For example, if a node’s secondary property is a string (type <str>) that happens to contain a valid domain (type <inet:fqdn>), a wildcard pivot out from the node with the string value will not return the inet:fqdn node.
    • Digraph (edge) nodes, whose properties are of type <ndef> (node definition, or <form>,<valu> tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms.
  • It is possible to perform an explicit pivot between properties of different types. For example: <inet:dns:query> :name -> inet:fqdn
  • See Pivot Out and Walk for a more comprehensive alternative to the wildcard pivot out (-> *).

Pivot In Operator

The pivot in ( <-) operator is similar to but separate from the pivot out ( ->) operator. The pivot in operator pivots to the set of nodes that reference the current set of nodes.

Logically, any pivot in operation can be expressed as an equivalent pivot out operation. For example, the following two pivots would be functionally equivalent:

  • Pivot from a set of domains to their associated DNS A records:

    <inet:fqdn> -> inet:dns:a:fqdn

  • Use “pivot in” to navigate from a set of domains to the DNS A records that reference a set of domains:

    <inet:fqdn> <- inet:dns:a:fqdn

Because of this equivalence, and because “left to right” logic is generally more intuitive, only pivot out has been fully implemented in Storm. (The second example, above, will actually return an error.) The pivot in operator exists, but is only used for certain special case pivot operations:

Syntax:

<query> <- *

Example:

Pivot from all primary properties to all nodes with an equivalent secondary property (<form> = <valu> to <prop> = <pval>):

  • Pivot from a set of domains to all nodes with a secondary property that references the domains:
<inet:fqdn> <- *

Usage Notes:

  • Pivoting in using the asterisk wildcard ( * ) is sometimes called a refs in pivot because it pivots from the inbound nodes to all nodes that reference those nodes. That is, for each inbound node, the “refs in” pivot will pivot from the primary property of a node to all nodes that have a secondary property equal to that type and value.

  • Pivoting in using the wildcard will return an instance of a node for each matching secondary property. For example, where a node may have the same <pval> for two different secondary properties (such as :domain and :zone on an inet:fqdn node), the pivot in will return two copies of the node. Results can be de-duplicated using the Storm uniq command.

  • Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot in from properties that match both <type> and <valu> / <pval>. This means that the following nodes will not be returned by a wildcard pivot in:

    • Nodes with matching <valu> / <pval> but of different <type>. For example, if a node’s primary property (such as a domain, type <inet:fqdn>) - happens to be referenced as as a different type (such as a string, type <str>) as a secondary property of another node, a wildcard pivot in to the inet:fqdn node will not return the node with the string value.
    • Digraph (edge) nodes, whose properties are of type <ndef> (node definition, or <form>,<valu> tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms.
  • Other than digraph (edge) node navigation / traversal, pivot in can only be used with the wildcard ( * ). That is, pivot in does not support specifying a particular target form:

    inet:fqdn=woot.com <- inet:dns:a:fqdn

    The above query will return an error. A filter operation (see Storm Reference - Filtering) can be used to downselect the results of a wildcard pivot in operation to a specific set of forms:

    inet:fqdn=woot.com <- * +inet:dns:a

  • See Pivot In and Walk for a more comprehensive alternative to the wildcard pivot in (<- *).

Pivot With Join

The pivot and join operator ( -+> ) performs the specified pivot operation but joins the results with the inbound set of nodes. That is, the inbound nodes are retained and combined with the results of the pivot.

Another way to look at the difference between a pivot and a join is that a pivot operation consumes nodes (the inbound set is discarded and only nodes resulting from the pivot operation are returned) but a pivot and join does not consume the inbound nodes.

The pivot and join operator is used to:

  • retain the inbound nodes and pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes,
  • retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes,
  • retain the inbound nodes and pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and
  • retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes.

Syntax:

<query> -+> <form> : <prop>

<query> : <prop> -+> <form>

<query> -+> *

<query> : <prop> -+> <form> : <prop>

Examples:

Pivot and join from primary property (<form> = <valu>) to secondary property (<prop> = <pval>):

  • Return a set of domains and all of their immediate subdomains:
<inet:fqdn> -+> inet:fqdn:domain

Pivot and join from secondary property (<prop> = <pval>) to primary property (<form> = <valu>):

  • Return a set of DNS A records and their associated IP addresses:
<inet:dns:a> :ipv4 -+> inet:ipv4

Pivot and join from all secondary properties to all forms (<prop> = <pval> to <form> = <valu>):

  • Return a set of WHOIS records and all nodes whose primary property equals any of the secondary properties of the WHOIS record (the asterisk ( * ) is a wildcard that indicates pivot to any applicable node):
<inet:whois:rec> -+> *

Pivot and join from secondary property (<prop> = <pval>) to secondary property (<prop> = <pval>):

  • Return the WHOIS records for a set of domains and the DNS A records for the same domains:
<inet:whois:rec> :fqdn -+> inet:dns:a:fqdn

Usage Notes:

  • A pivot and join using the wildcard ( * ) will pivot to all nodes whose primary property (<form> = <valu>) matches a secondary property (<prop> = <pval>) of the inbound nodes. This excludes digraph nodes (such as edge:refs or edge:has nodes) because their primary property is a pair of ndefs (node definitions, or <form>, <valu> tuples).

Traverse (Walk) Light Edges

The traverse (walk) light edges operator ( -(<verb>)>, <(<verb>)- ) is used to traverse from a set of inbound nodes to the set of nodes they are linked to by the specified light edge(s). Because a light edge is not a node, the navigation is technically a “traversal” of the light edge as opposed to a property-to-property pivot. (Contrast with Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes below.)

Syntax:

<query> -( <verb> )> * | <form>

<query> -( ( <verb1> , <verb2> [ , <verb3> …] ) )> * | <form>

<query> -( * )> * | <form>

<query> <( <verb> )- * | <form>

<query> <( ( <verb1> , <verb2> [ , <verb3> …] ) )- * | <form>

<query> <( * )- * | <form>

Examples:

Traverse the “refs” light edge from a media:news node to all of the FQDNs “referenced” by the node:

<media:news> -(refs)> inet:fqdn

Traverse the “refs” light edge from a media:news node to all of the nodes “referenced” by the node:

<media:news> -(refs)> *

Traverse the “hasip” light edge from an inet:ipv4 to the CIDR block(s) the IP is part of:

<inet:ipv4> <(hasip)- inet:cidr4

Traverse the “hasip” and “ipwhois” light edges from an inet:ipv4 to any nodes linked via those light edges (i.e., typically the CIDR block(s) the IP is part of and the netblock registration record(s) for the IP):

<inet:ipv4> <((hasip, ipwhois))- *

Traverse any / all light edges from a media:news node to all nodes referenced by those light edges:

<media:news> -(*)> *

Usage Notes:

  • The traversal syntax allows specification of a single verb, a list of verbs, or the “wildcard” / asterisk ( * ) to reference any / all light edge verbs that may be present.
  • The Storm model commands can be used to list and work with any light edge verbs in a Cortex.

Pivot Out and Walk

The pivot out and walk (traverse) light edges operator ( --> ) combines a wildcard pivot out (“refs out”) operation ( -> * ) with a wildcard walk light edges operation ( -(*)> ).

Syntax:

<query> —-> *

Examples:

Pivot from an IP netblock registration record to all nodes referenced by the record’s secondary properties and all nodes linked to the record by light edges:

<inet:whois:iprec> --> *

Usage Notes:

  • The pivot out and walk operator can only be used with a wildcard ( * ); it is not possible to specify a particular form as the target of the pivot. A filter operation can be used to refine the results of the pivot and walk operation if necessary.
  • The pivot and walk operators (pivot out and walk / pivot in and walk) are useful for “exploring” data in a Cortex as they will return all the nodes “next to” the working set of nodes without requiring the user to have specific knowledge of the data model.

Pivot In and Walk

The pivot in and walk (traverse) light edges operator ( <-- ) combines a wildcard pivot in (“refs in”) operation ( <- * ) with a wildcard walk light edges operation ( <(*)- ).

Syntax:

<query> <—- *

Examples:

Pivot from a set of IP addresses to all nodes that reference the IPs and all nodes linked to the IPs by light edges:

<inet:ipv4> <-- *

Usage Notes:

  • The pivot in and walk operator can only be used with a wildcard ( * ); it is not possible to specify a particular form as the target of the pivot. A filter operation can be used to refine the results of the pivot and walk operation if necessary.
  • The pivot and walk operators (pivot out and walk / pivot in and walk) are useful for “exploring” data in a Cortex as they will return all the nodes “next to” the working set of nodes without requiring the user to have specific knowledge of the data model.

Pivot to Digraph (Edge) Nodes

Digraph (edge) nodes (Digraph (Edge) Form) are of type edge or timeedge. These nodes (forms) are unique in that their primary property value is a pair of node definitions (type Ndef) - that is, <form>, <valu> tuples. (timeedge forms are comprised of two <form>, <valu> tuples and an additional <time> value). Each <form>, <valu> tuple from the primary property is broken out as secondary property :n1 or :n2. This means that pivoting to and from digraph nodes is a bit different than pivoting to and from nodes whose properties are a simple <valu> or <pval>.

Syntax:

<query> -> <edge> | <timeedge> [:n2]

<query> -+> <edge> | <timeedge> [:n2]

<query> <- <edge> | <timeedge>

Examples:

Pivot out from a set of nodes whose ndefs (<form>, <valu>) are the first element (:n1) in a set of a digraph nodes:

  • Pivot out from a person node to the set of digraph nodes representing things that person “has”:
<ps:person> -> edge:has
  • Pivot out from a person node to the set of timeedge digraph nodes representing places that person has been to (and when):
<ps:person> -> edge:wentto

Pivot in from a set of nodes whose ndefs (<form>, <valu>) are the second element (:n2) in a set of a digraph nodes:

  • Pivot in from an article to the set of digraph nodes representing things that “have” the article (e.g., people or organizations who authored the article):
<media:news> <- edge:has

Usage Notes: - The pivot out and pivot in operators have been optimized for digraph nodes. Because digraphs use ndef properties, Storm makes the following assumptions:

  • When pivoting to or from a set of nodes to a set of digraph nodes, pivot using the ndef (<form>,<valu>) of the inbound nodes and not their primary property (<valu>) alone.
  • When pivoting out to a digraph node, the inbound nodes’ <form>,<valu> ndef will be the first element (:n1) of the digraph. You must explicitly specify :n2 to pivot to the second element.
  • When pivoting in to a digraph node, the inbound nodes’ <form>,<valu> ndef will be the second element (:n2) of the digraph. It is not possible to pivot in to :n1.
  • Pivoting to / from digraph nodes is one of the specialized use cases for the pivot in ( <-) operator, however the primary use case of pivot in with digraph nodes is reverse edge traversal (see Pivot Across Digraph (Edge) Nodes). See Pivot In Operator for general limitations of the pivot in operator.

Pivot Across Digraph (Edge) Nodes

Because digraph nodes represent generic edge relationships, analytically we are often more interested in the nodes on “either side” of the edge than in the digraph node itself. For this reason, the pivot operators have been optimized to allow a syntax for easily navigating “across” these digraphs (edges).

Syntax:

<query> -> <edge> | <timeedge> -> * | <form>

<query> <- <edge> | <timeedge> <- * | <form>

Examples:

  • Traverse a set of edge:has nodes to pivot from a person to all the things the person “has”:
<ps:person> -> edge:has -> *
  • Traverse a set of edge:wentto nodes to pivot from a person to the locations the person has visited:
<ps:person> -> edge:wentto -> *

Usage Notes:

  • Storm makes the following assumptions to optimize the two pivots:
    • For pivots out, the first pivot is to the digraph nodes’ :n1 property and the second pivot is from the digraph nodes’ :n2 property.
    • For pivots in, the first pivot is to the digraph nodes’ :n2 property and the second pivot is from the digraph nodes’ :n1 property.
  • Pivoting “across” the digraph nodes still performs two pivot operations (i.e., to the digraph nodes and then from them). As such it is still possible to apply an optional filter to the digraph nodes themselves before the second pivot.

Pivot to Tags

Pivot to tags syntax allows you to pivot from a set of nodes to the set of syn:tag nodes for the tags applied to those nodes. This includes:

  • pivot to all leaf tag nodes,
  • pivot to all tag nodes,
  • pivot to all tag nodes matching a specified prefix, and
  • pivot to tag nodes matching an exact tag.

See the Synapse background documents <link> for additional discussion of tags and syn:tag nodes.

Syntax:

<query> -> # [ * | # <tag> .* | # <tag> ]

Examples:

Pivot to all leaf tag nodes:

  • Pivot from a set of domains to the syn:tag nodes for all leaf tags applied to those domains:
<inet:fqdn> -> #

Pivot to ALL tag nodes:

  • Pivot from a set of files to the syn:tag nodes for all tags applied to those files:
<file:bytes> -> #*

Pivot to all tag nodes matching the specified prefix:

  • Pivot from a set of IP addresses to the syn:tag nodes for all tags applied to those IPs that are part of the anonymized infrastructure tag tree:
<inet:ipv4> -> #cno.infra.anon.*

Pivot to tag nodes exactly matching the specified tag:

  • Pivot from a set of nodes to the syn:tag node for #foo.bar (if present on the inbound set of nodes):
<query> -> #foo.bar

Usage Notes:

  • Pivot to all tags ( #* ) and pivot by prefix matching ( #<tag>.* ) will match all tags in the relevant tag trees from the inbound nodes, not just the leaf tags. For example, for an inbound node with tag #foo.bar.baz, #* will return the syn:tag nodes for foo, foo.bar, and foo.bar.baz.

Pivot from Tags

Pivot from tags syntax allows you to pivot from a set of syn:tag nodes to the set of nodes that have those tags.

Syntax:

<syn:tag> -> * | <form>

Examples:

  • Pivot to all domains tagged with tags from any of the inbound syn:tag nodes:
<syn:tag> -> inet:fqdn
  • Pivot to all nodes tagged with tags from any of the inbound syn:tag nodes:
<syn:tag> -> *

Usage Notes:

  • In many cases, pivot from tags is functionally equivalent to Lift by Tag (#). That is, the following queries will both return all nodes tagged with #aka.feye.thr.apt1:

    syn:tag=aka.feye.thr.apt1 -> *

    #aka.feye.thr.apt1

    Pivoting from tags is most useful when used in conjunction with Pivot to Tags - that is, taking a set of inbound nodes, pivoting to the syn:tag nodes for any associated tags (pivot to tags), and then pivoting out again to other nodes tagged with some or all of those tags (pivot from tags).

Implicit Pivot Syntax

If the target or source property of a pivot is readily apparent - that is, given the inbound and target forms, only one set of properties makes sense for that pivot - the properties do not have to be explicitly specified. This implicit pivot syntax allows users to enter more concise pivot queries in some cases.

Implicit pivot syntax can be used to pivot from a primary property to a secondary property, as well as from a secondary property to a primary property.

Implicit pivot syntax is a query optimization made possible by Storm’s Type Awareness.

Examples:

Pivot from primary property (<form> = <valu>) to implicit secondary property (<prop> = <pval>):

  • Pivot from a set of domains to their associated DNS A records:

Regular (full) syntax:

<inet:fqdn> -> inet:dns:a:fqdn

Implicit syntax:

<inet:fqdn> -> inet:dns:a

With implicit syntax, the target property :fqdn can be omitted because it is the only logical target given a set of inet:fqdn nodes as the source.

Pivot from implicit secondary property (<prop> = <pval>) to primary property (<form> = <valu>):

  • Pivot from a set of DNS A records to their associated IP addresses:

Regular (full) syntax:

<inet:dns:a> :ipv4 -> inet:ipv4

Implicit syntax:

<inet:dns:a> -> inet:ipv4

With implicit syntax, the source property :ipv4 can be omitted because it is the only logical source given a set of inet:ipv4 nodes as the target.

Use of multiple implicit pivots:

  • Pivot from a set of domains to their DNS A records and then to the associated IP addresses:

Regular (full) syntax:

<inet:fqdn> -> inet:dns:a:fqdn :ipv4 -> inet:ipv4

Implicit syntax:

<inet:fqdn> -> inet:dns:a -> inet:ipv4

Raw Pivot Syntax

For certain edge cases, standard Storm pivot syntax (explicit or implicit) is insufficient. In these instances raw pivot syntax acts as a “get out of jail free” card to perform specialized pivot operations. These include:

  • primary-to-primary property pivots;
  • pivots where the value of the target property (primary or secondary) is computed from the input node(s);
  • extramodel pivots.

In raw pivot syntax, the target of the pivot is specified as a Storm query enclosed in curly braces. Raw pivots often involve specifying a variable derived from the inbound node(s) and performing the raw pivot using the variable, though this is not technically required. (See Storm Reference - Advanced - Variables for a discussion of using variables in Storm).

For some raw pivot syntax use cases, you can compose an equivalent Storm query using lift and filter operations, i.e.,:

  • lift a set of nodes;
  • define a variable based on those nodes;
  • lift a second set of nodes using the variable;
  • filter out the original nodes you lifted, thus leaving only the second set of lifted nodes.

However, executing this type of query using raw pivot syntax is slightly more efficient; the Storm query within the raw pivot’s curly braces may still be a lift operation, but performing it inside a raw pivot means you do not have to explicitly drop (filter out) your original nodes. (As with a regular pivot, the inbound nodes are consumed by the pivot operation itself, eliminating the need for the filter.)

As always, these efficiencies may be trivial for smaller queries but can be significant for larger queries.

Syntax:

<query> -> { <query> }

Examples:

  • Pivot from a string (it:dev:str) representing an FQDN to the inet:fqdn node for that FQDN (i.e., pivot between two primary properties of different types).

Standard syntax (no raw pivot, lift / filter only):

<it:dev:str> $fqdn=$node.value() inet:fqdn=$fqdn -it:dev:str

Raw pivot syntax:

<it:dev:str> $fqdn=$node.value() -> { inet:fqdn=$fqdn }