Package Documentation

Storm Package: synapse-urlscan

The following Commands are available from this package. This documentation is generated for version 1.12.0 of the package.

Storm Commands

This package implements the following Storm Commands.

urlscan.enrich

Enrich nodes by issuing URLScan queries for inbound nodes.

This command constructs a URLScan query using inbound nodes and submits it
to the search endpoint to retrieve archived scans. The results are used to create
inet:url nodes. In cases where the URL queried does not match the final URL
returned by URLScan, inet:urlredir nodes will be created to represent the redirect
chain of URLs.

If IP address information is available, the corresponding inet:dns:* nodes will
be created to represent it.

To download the full scan output of each search result, the --detail option can
be used. This will query the results API and poll until the scan is complete.
Once results are available, they will be downloaded and ingested into the Cortex.
Using this option will cause command execution to take significantly longer and
consume additional quota.

When the --detail option is used, ``requests`` light edges will be created from
the inet:url node to each inet:http:request node that is created. Additionally,
the final DOM tree will be downloaded to the Axon and have the file:bytes node
linked to the result inet:url with a ``dom`` light edge.

When --detail is specified, the --download-responses option may be specified with
a value of "primary" or "all" to download HTTP response data available in the
results. If "primary" is specified, only the primary response or file download
will be retrieved. If "all" is specified, all available HTTP responses will be
downloaded.

Note:

  By default, only the overall URLScan maliciousness score will be populated.
  The URLScan specific score (not including community verdicts), is only available
  --detail is specified.

Examples:

  // Enrich an inet:url node
  inet:url=https://support.vertex.link | urlscan.enrich

  // Enrich an inet:fqdn node and yield the inet:url nodes created
  inet:fqdn=vertex.link | urlscan.enrich --yield


Usage: urlscan.enrich [options]

Options:

  --help                      : Display the command usage.
  --time <time>               : Filter by results within a specific time interval.
  --detail                    : Download and ingest the full scan results from the results endpoint.
  --download-responses <download_responses>: Download HTTP responses bodies and file downloads in the result (must be specified with
                                --detail). (choices: primary, all)
  --dom                       : Download the DOM tree for the result.
  --screenshot                : Download the screenshot for the result.
  --yield                     : Yield the newly created nodes.
  --debug                     : Print debugging messages.
  --asof <asof>               : This argument is deprecated and no longer has any effect.
  --size <size>               : Limit the number of results ingested to the given size (per-node).

urlscan.quota

Print out usage and usage limits

Usage: urlscan.quota [options]

Options:

  --help                      : Display the command usage.

urlscan.scan

Submit URLs to be scanned to URLScan.

This command takes an inet:url node as input and submits the url URLScan to
be scanned.

After submitting the URL to the scan endpoint, this command will query the results
API and poll until the scan is complete. Once results are available, they will be
downloaded and ingested into the Cortex.

When ingesting the results, ``requests`` light edges will be created from
the inet:url node to each inet:http:request node that is created. Additionally,
the final DOM tree will be downloaded to the Axon and have the file:bytes node
linked to the resulting inet:url with a ``dom`` light edge.

The --download-responses option may be specified with a value of "primary" or
"all" to download HTTP response data available in the results. If "primary" is
specified, only the primary response or file download will be retrieved. If "all"
is specified, all available HTTP responses will be downloaded.

In cases where the URL queried does not match the final URL returned by URLScan,
inet:urlredir nodes will be created to represent the redirect chain of URLs.

NOTE - This command will also create an it:exec:query node to represent the
       query syntax and link resulting nodes to it via -(found)> edges.

Examples:

  // Scan an inet:url node
  inet:url=https://vertex.link | urlscan.scan

  // Scan an inet:url node with a custom HTTP referer string
  inet:url=https://vertex.link | urlscan.scan --referer "https://example.com/"


Usage: urlscan.scan [options]

Options:

  --help                      : Display the command usage.
  --customagent <customagent> : Override User-Agent for this scan.
  --referer <referer>         : Override HTTP referer for this scan.
  --visibility <visibility>   : One of public, unlisted, private. Defaults to your API key's configured default
                                visibility.
  --country <country>         : Specify which country the scan should be performed from (2-Letter ISO-3166-1 alpha-2
                                country). (choices: au, ca, de, dk, es, fi, fr, gb, is, it, jp, nl, no, nz, pl, se, sg,
                                us)
  --download-responses <download_responses>: Download HTTP responses bodies and file downloads in the result. (choices: primary,
                                all)
  --debug                     : Print debugging messages.

urlscan.search.certificates

Search for certificates with the URLScan search API endpoint.

This command takes inbound inet:fqdn nodes and searches the
URLScan certificates datasource for SSL certificates. The --query
argument may instead be used to provide a query string in Lucene/ES
syntax. The results are used to create crypto:x509:cert nodes.

This command will also create an it:exec:query node to represent the
query syntax and link resulting nodes to it via -(found)> edges.

Note: This command requires an API key with urlscan Pro access.

For more information on the query syntax for this command, please visit
the URLScan documentation at https://urlscan.io/docs/search/

Examples:

    // Search for certificates using an inet:fqdn node
    inet:fqdn=vertex.link | urlscan.search.certificates

    // Search using --query and yield the created crypto:x509:cert nodes
    urlscan.search.certificates --query vertex.link --yield


Usage: urlscan.search.certificates [options]

Options:

  --help                      : Display the command usage.
  --query <query>             : Lucene/ES query syntax. See URLScan docs for details.
  --yield                     : Yield the newly created nodes.
  --debug                     : Print debugging messages.
  --asof <asof>               : This argument is deprecated and no longer has any effect.
  --size <size>               : Limit the number of results ingested to the given size (per-node).

urlscan.search.hostnames

Search for hostnames with the URLScan search API endpoint.

This command takes inbound inet:fqdn nodes and searches the
URLScan hostnames datasource for matches. The --query argument
may instead be used to provide a query string in Lucene/ES syntax.
The results are used to create inet:fqdn nodes.

This command will also create an it:exec:query node to represent the
query syntax and link resulting nodes to it via -(found)> edges.

Note: This command requires an API key with urlscan Pro access.

For more information on the query syntax for this command, please visit
the URLScan documentation at https://urlscan.io/docs/search/

Examples:

    // Search for hostnames using an inet:fqdn node
    inet:fqdn=vertex.link | urlscan.search.hostnames

    // Search using --query and yield the created inet:fqdn nodes
    urlscan.search.hostnames --query vertex.link --yield


Usage: urlscan.search.hostnames [options]

Options:

  --help                      : Display the command usage.
  --query <query>             : Lucene/ES query syntax. See URLScan docs for details.
  --yield                     : Yield the newly created nodes.
  --debug                     : Print debugging messages.
  --asof <asof>               : This argument is deprecated and no longer has any effect.
  --size <size>               : Limit the number of results ingested to the given size (per-node).

urlscan.search.incidents

Search for incidents with the URLScan search API endpoint.

This command takes inbound inet:fqdn, inet:ipv4, inet:ipv6, or inet:url
nodes and searches the URLScan incidents datasource for existing
incidents. The --query argument may instead be used to provide a query
string in Lucene/ES syntax. The results are used to create meta:rule nodes.

This command will also create an it:exec:query node to represent the
query syntax and link resulting nodes to it via -(found)> edges.

Note: This command requires an API key with urlscan Pro access.

For more information on the query syntax for this command, please visit
the URLScan documentation at https://urlscan.io/docs/search/

Examples:

    // Search for incidents using an inet:fqdn node
    inet:fqdn=vertex.link | urlscan.search.incidents

    // Search for incidents flagged as malicious and yield the
    // created meta:rule nodes
    urlscan.search.incidents --query "allTags:ismalicious" --yield


Usage: urlscan.search.incidents [options]

Options:

  --help                      : Display the command usage.
  --query <query>             : Lucene/ES query syntax. See URLScan docs for details.
  --yield                     : Yield the newly created nodes.
  --debug                     : Print debugging messages.
  --size <size>               : Limit the number of results ingested to the given size (per-node).

urlscan.setup.apikey

Set the urlscan.io API key.

Usage: urlscan.setup.apikey [options] <apikey>

Options:

  --help                      : Display the command usage.
  --self                      : Set the key as a user variable. If not used, the key is set globally.

Arguments:

  <apikey>                    : The urlscan.io API key string.

urlscan.setup.tagprefix

Set the tag prefix used when recording urlscan.io data as tags.
The default tag prefix is "rep.urlscan" if not specified.
Any tags provided by the urlscan.io API will be added within the given namespace.
For example, the item "foo" would result in "#rep.urlscan.foo".  Any
characters incompatible with tag names are replaced with "_".


Usage: urlscan.setup.tagprefix [options] <tagname>

Options:

  --help                      : Display the command usage.

Arguments:

  <tagname>                   : The tag prefix to use.

Storm Modules

This package does not export any Storm APIs.