Package Documentation
Storm Package: synapse-urlscan
The following Commands are available from this package. This documentation is generated for version 1.5.0 of the package.
Storm Commands
This package implements the following Storm Commands.
urlscan.enrich
Enrich nodes by issuing URLScan queries for inbound nodes.
This command constructs a URLScan query using inbound nodes and submits it
to the search endpoint to retrieve archived scans. The results are used to create
inet:url nodes. In cases where the URL queried does not match the final URL
returned by URLScan, inet:urlredir nodes will be created to represent the redirect
chain of URLs.
If IP address information is available, the corresponding inet:dns:* nodes will
be created to represent it.
To download the full scan output of each search result, the --detail option can
be used. This will query the results API and poll until the scan is complete.
Once results are available, they will be downloaded and ingested into the Cortex.
Using this option will cause command execution to take significantly longer and
consume additional quota.
When the --detail option is used, ``requests`` light edges will be created from
the inet:url node to each inet:http:request node that is created. Additionally,
the final DOM tree will be downloaded to the Axon and have the file:bytes node
linked to the result inet:url with a ``dom`` light edge.
Examples:
// Enrich an inet:url node
inet:url=https://support.vertex.link | urlscan.enrich
// Enrich an inet:fqdn node and yield the inet:url nodes created
inet:fqdn=vertex.link | urlscan.enrich --yield
Usage: urlscan.enrich [options]
Options:
--help : Display the command usage.
--time <time> : Filter by results within a specific time interval.
--detail : Download and ingest the full scan results from the results endpoint.
--dom : Download the DOM tree for the result.
--screenshot : Download the screenshot for the result.
--yield : Yield the newly created nodes.
--debug : Print debugging messages.
--asof <asof> : Specify the maximum age for a cached result. To disable caching, use --asof now. (default: -30days)
--size <size> : Limit the number of results ingested to the given size (per-node).
urlscan.quota
Print out usage and usage limits
Usage: urlscan.quota [options]
Options:
--help : Display the command usage.
urlscan.scan
Submit URLs to be scanned to URLScan.
This command takes an inet:url node as input and submits the url URLScan to
be scanned.
After submitting the URL to the scan endpoint, this command will query the results
API and poll until the scan is complete. Once results are available, they will be
downloaded and ingested into the Cortex.
When ingesting the results, ``requests`` light edges will be created from
the inet:url node to each inet:http:request node that is created. Additionally,
the final DOM tree will be downloaded to the Axon and have the file:bytes node
linked to the resulting inet:url with a ``dom`` light edge.
In cases where the URL queried does not match the final URL returned by URLScan,
inet:urlredir nodes will be created to represent the redirect chain of URLs.
NOTE - This command will also create an it:exec:query node to represent the
query syntax and link resulting nodes to it via -(found)> edges.
Examples:
// Scan an inet:url node
inet:url=https://vertex.link | urlscan.scan
// Scan an inet:url node with a custom HTTP referer string
inet:url=https://vertex.link | urlscan.scan --referer "https://example.com/"
Usage: urlscan.scan [options]
Options:
--help : Display the command usage.
--customagent <customagent> : Override User-Agent for this scan.
--referer <referer> : Override HTTP referer for this scan.
--visibility <visibility> : One of public, unlisted, private. Defaults to your API key's configured default visibility.
--country <country> : Specify which country the scan should be performed from (2-Letter ISO-3166-1 alpha-2 country). (choices: au, ca, de, dk, es, fi, fr, gb, is, it, jp, nl, no, nz, pl, se, sg, us)
--debug : Print debugging messages.
urlscan.search
Ingest data from the URLScan search API endpoint.
This command takes a query string in Lucene/ES syntax and submits it
to the search endpoint to retrieve archived scans. The results are used to create
inet:url nodes. In cases where the URL queried does not match the final URL
returned by URLScan, inet:urlredir nodes will be created to represent the redirect
chain of URLs.
If IP address information is available, the corresponding inet:dns:* nodes will
be created to represent it.
To download the full scan output of each search result, the --detail option can
be used. This will query the results API and poll until the scan is complete.
Once results are available, they will be downloaded and ingested into the Cortex.
Using this option will cause command execution to take significantly longer and
consume additional quota.
When the --detail option is used, ``requests`` light edges will be created from
the inet:url node to each inet:http:request node that is created. Additionally,
the final DOM tree will be downloaded to the Axon and have the file:bytes node
linked to the result inet:url with a ``dom`` light edge.
NOTE - This command will also create an it:exec:query node to represent the
query syntax and link resulting nodes to it via -(found)> edges.
For more information on the query syntax for this command, please visit
the URLScan documentation at https://urlscan.io/docs/search/
Examples:
// Search by domain and yield the created inet:url nodes
urlscan.search domain:time.gov --yield
Usage: urlscan.search [options] <query>
Options:
--help : Display the command usage.
--detail : Download and ingest the full scan results from the results endpoint.
--dom : Download the DOM tree for the result.
--screenshot : Download the screenshot for the result.
--yield : Yield the newly created nodes.
--debug : Print debugging messages.
--asof <asof> : Specify the maximum age for a cached result. To disable caching, use --asof now. (default: -30days)
--size <size> : Limit the number of results ingested to the given size (per-node).
Arguments:
<query> : Lucene/ES query syntax. See URLScan docs for details.
urlscan.setup.apikey
Set the urlscan.io API key.
Usage: urlscan.setup.apikey [options] <apikey>
Options:
--help : Display the command usage.
--self : Set the key as a user variable. If not used, the key is set globally.
Arguments:
<apikey> : The urlscan.io API key string.