storm.lint

The Synapse storm.lint tool checks Storm code for syntax errors, pipeline usage which may not do what was intended, and Storm which could be written more simply. With --v3-ready it also reports Storm which must change for Synapse 3.x. It parses the Storm locally and does not connect to a Cortex.

Syntax

storm.lint is executed using python -m synapse.tools.storm.lint. The command usage is as follows:

python -m synapse.tools.storm.lint -h

usage: synapse.tools.storm.lint [-h] [--select SELECT] [--ignore IGNORE]
                                [--enable ENABLE] [--v3-ready | --no-v3-ready]
                                [--config CONFIG] [--format {text,json}]
                                [--show-rules]
                                [paths ...]

Lint Storm code.

Each path is a file, a directory which is searched recursively for *.storm files, or - to read stdin.

Without --config, settings are read from the [tool.stormlint] table of the nearest pyproject.toml
which has one, searching upward from the current directory.

With --format json, each finding or file error is printed as one JSON object per line.

With --v3-ready, the rules which report Storm that must change for Synapse 3.x are added to the
selected rules.

Exit codes: 0 when clean, 1 when there are findings, 2 on a usage, config or file error.

positional arguments:
  paths                 Files or directories to lint (use - for stdin).

options:
  -h, --help            show this help message and exit
  --select SELECT       Comma-separated rule IDs or prefixes to run in place
                        of the default rules.
  --ignore IGNORE       Comma-separated rule IDs or prefixes to disable.
  --enable ENABLE       Comma-separated rule IDs or prefixes to add to the
                        selected rules, such as rules which are off by
                        default.
  --v3-ready, --no-v3-ready
                        Add the rules which report Storm that must change for
                        Synapse 3.x.
  --config CONFIG       Path to a pyproject.toml with a [tool.stormlint]
                        table.
  --format {text,json}  Print findings as text, or as one JSON object per
                        line.
  --show-rules          Print the lint rules and exit.

Each path may be:

a Storm file, which is linted whatever its extension,
a directory, which is searched recursively for *.storm files, or
-, which reads Storm code from stdin.

For example, to lint every Storm file in a Power-Up and a query read from stdin:

python -m synapse.tools.storm.lint acme-hello/storm/
echo '$x = $lib.list()' | python -m synapse.tools.storm.lint -

Output

Each finding is printed as <path>:<line>:<column>: <rule> <message>, followed by a count of the findings:

example.storm:1:6: S005 Use a list literal such as ([]) instead of $lib.list().
example.storm:3:13: P001 Avoid spin, isolate the query in a function.
Found 2 errors.

When there are no findings, the tool prints All checks passed!.

JSON Output

--format json prints each finding as one JSON object per line, with path, line, column, rule, severity and mesg keys, and no summary line. A file which could not be read is printed as an object with path and error keys. When there are no findings, nothing is printed:

python -m synapse.tools.storm.lint --v3-ready --format json example.storm
{"path":"example.storm","line":1,"column":6,"rule":"R001","severity":"high","mesg":"$lib.true was removed, use (true)."}
{"path":"example.storm","line":2,"column":1,"rule":"M001","severity":"high","mesg":"inet:ipv4 is renamed to inet:ip in 3.x."}

The exit codes are the same for both formats. A usage or configuration error is printed as text. --show-rules always prints text.

Exit Codes

Code	Meaning
`0`	No findings.
`1`	One or more findings.
`2`	A usage error, an invalid configuration, an unknown rule ID, or a file which could not be read.

Selecting Rules

Three options choose which rules run:

--select is exclusive: it replaces the default rule set, so only the selected rules run. Without it, every rule which is on by default is run.
--enable is additive: it adds rules to the selected set, which is mostly used to turn on rules which are off by default.
--ignore removes rules from the selected set.

Each option takes a comma-separated list of rule IDs or ID prefixes. A prefix matches every rule whose ID starts with it, so S matches every style rule and S01 matches S010 through S015. A rule named by --select or --enable runs whether or not it is on by default.

--v3-ready adds every M rule and every R rule other than R004 to the selected set, after --select and --enable and before --ignore. --no-v3-ready turns it off when the configuration turns it on. See Checking Storm for Synapse 3.x.

# only the 3.x syntax and runtime rules
python -m synapse.tools.storm.lint --select R mypkg/storm/

# the default rules, plus S010 and S015 which are off by default
python -m synapse.tools.storm.lint --enable S010,S015 mypkg/storm/

# the default rules, except the style rules
python -m synapse.tools.storm.lint --ignore S mypkg/storm/

# the default rules and the 3.x rules, except the data model rules
python -m synapse.tools.storm.lint --v3-ready --ignore M mypkg/storm/

--show-rules prints every rule with its severity, whether it is on by default, its name and the first line of its description, and exits. The DEFAULT column is yes or no, or v3 for a rule which is added by --v3-ready.

Suppressing Findings

A // noqa comment suppresses findings reported on its line. A // noqa: <ID> comment suppresses only the named rules, and takes a comma-separated list of rule IDs or ID prefixes:

$x = $lib.true            // noqa
$y = $lib.dict()          // noqa: S005
$z = $lib.list($lib.null) // noqa: R001, S005

A parse failure (E001) is always reported and cannot be suppressed.

Configuration

Default options may be set in a [tool.stormlint] table in a pyproject.toml file. Without --config, the tool uses the nearest pyproject.toml with a [tool.stormlint] table, searching upward from the current directory. --config <path> reads the table from the given pyproject.toml instead.

[tool.stormlint]
select = ["E", "P", "S"]
ignore = ["S002"]
enable = ["S015"]
v3_ready = true
exclude = [
    "tests/files/*",
    "storm/vendored.storm",
]

Key	Description
`select`	Rule IDs or prefixes to run, in place of the default rules. Only the selected rules run.
`ignore`	Rule IDs or prefixes to skip.
`enable`	Rule IDs or prefixes to add to the selected rules, such as rules which are off by default.
`v3_ready`	A boolean. When `true`, the rules which report Storm that must change for Synapse 3.x are added to the selected rules, as with `--v3-ready`.
`exclude`	Glob patterns, relative to the directory of the `pyproject.toml`, for files and directories which are not linted.

Each value other than v3_ready is a list of strings. A command line option replaces the configured value for the same key rather than adding to it, and --v3-ready or --no-v3-ready replaces v3_ready. An unknown key or a value of the wrong type is a configuration error.

Rules

Rule IDs are grouped by prefix: E rules report errors, P rules report pipeline usage which may not do what was intended, S rules report Storm which could be written more simply, R rules report Storm which does not run as intended on Synapse 3.x, and M rules report data model names which are renamed or removed in Synapse 3.x. The M rules and every R rule other than R004 are off by default, and are added by --v3-ready. R004 runs by default, since a bare true, false or null is also a string on 2.x.

ID	Name	Severity	Default	Description
`E001`	`parse-failure`	high	yes	The Storm code failed to parse. This finding is always reported and cannot be suppressed.
`E002`	`invalid-escape`	high	yes	A double-quoted or backtick string contains a backslash escape which is not a valid Python string escape. Single-quoted strings do not process escapes and are not checked.
`E003`	`duplicate-function`	high	yes	A function is defined more than once in the same scope.
`P001`	`spin`	low	yes	The `spin` command discards the nodes in the pipeline. Isolate the query whose nodes are not needed in a function which uses `return()` or a `fini` block.
`P002`	`spin-return`	medium	yes	`spin \| return(...)` runs the return after consuming every node. Use `return(...)` when only one node can flow, or `fini { return(...) }` when more than one can.
`S001`	`guid-seed`	medium	yes	A guid form node is added with a value from `$lib.guid()`. Use a list seed such as `[ form=(a, $b) ]`, or `[ form=* ]` for a random guid. The resulting guid may differ, so verify it before changing existing code.
`S002`	`unnecessary-parens`	low	yes	Parens open an expression. Inside one which is already open, a literal, variable, call, or nested dict or list needs no parens of its own, and a lone variable assigned, passed as an argument or returned needs none either: `($y)` is just `$y`. Parens are still required around a literal value such as `(true)` or `(10)`, a dict or list literal used as a value, and grouping which changes precedence. Grouping a compound sub-expression is never reported.
`S003`	`edge-yield`	low	yes	An edge edit uses a subquery which only yields a variable. Use the variable directly, such as `+(refs)> $v` instead of `+(refs)> { yield $v }`.
`S004`	`return-null`	low	yes	`return((null))` and `return($lib.null)` return the same value as `return()`.
`S005`	`lib-list-dict`	low	yes	`$lib.list()` and `$lib.dict()` should be list and dict literals such as `([])` and `({})`.
`S006`	`print-kwargs`	low	yes	`$lib.print()` and `$lib.warn()` format their message with kwargs. Use a backtick format string.
`S008`	`guarded-for`	low	yes	An `if` statement which only guards a `for` loop over the variable it tests. An empty or null value iterates zero times, so the guard is not needed. The guard is only redundant when no node is in the pipeline, because a `for` loop over an empty or null value drops the inbound node. Verify it before changing existing code.
`S010`	`guarded-prop-set`	low	no	An `if` statement which only sets a property to the variable it tests. Use `?=` which skips a null value. The `if` statement also skips a value which is not null but is false, such as `0`, an empty string or an empty list, while `?=` sets it, and `?=` also skips a value which fails to norm. Verify the value can not be one of those before changing existing code.
`S011`	`subquery-lift-value`	low	yes	A property is set from a subquery which only lifts a form by its primary value. Set the property to the value directly when the property is the form type.
`S012`	`lift-filter`	low	yes	A query lifts a form and then filters on one of its properties by a value which does not read the node. Lift by the property instead.
`S014`	`try-cast`	low	yes	A `try` block calls `$lib.cast()`. Use `$lib.trycast()` and check its result instead.
`S015`	`return-import`	medium	no	A function returns a module from `$lib.import()`. In a privsep module the returned module inherits root privileges, and binding the import as `$__name` does not prevent that. Never return an imported module.
`R001`	`lib-constant`	high	v3	`$lib.true`, `$lib.false` and `$lib.null` were removed in 3.x. Use `(true)`, `(false)` and `(null)`.
`R002`	`gtor-call`	high	v3	`$node.value`, `$node.form`, `$node.ndef`, `$lib.version.synapse` and `$lib.version.commit` are gtors in 3.x and can not be called. Drop the parentheses.
`R003`	`node-iden`	high	v3	`$node.iden` was removed in 3.x.
`R004`	`bare-constant`	medium	yes	A bare `true`, `false` or `null` used as an assignment value, call argument, return value or function argument default is the string `"true"`, `"false"` or `"null"`. Use `(true)`, `(false)` or `(null)`, or quote the string.
`R005`	`removed-api`	high	v3	A `$lib` API which was removed in 3.x is used, such as `$lib.str`, `$lib.text`, `$lib.user`, `$lib.ps`, `$lib.bytes.put()` or the `$lib.gen` helpers. The message names the replacement when there is one. `$lib.vars`, `$lib.globals`, `$lib.env` and user vars (`$user.vars`) are dict-like in 3.x, so calling `get()`, `set()`, `pop()`, `list()` and similar methods on them is reported. Read and assign `$lib.globals.<name>`, assign `$lib.undef` to delete, and iterate with a `for` loop.
`R006`	`deref-space`	high	v3	Whitespace before the `.` of a variable deref or a tag name segment, such as `($foo .bar)` or `$(#foo .bar)`. 3.x does not allow whitespace before the dot, and reads a dot which follows whitespace as a virtual property. Whitespace after the dot is allowed.
`R007`	`pivot-in-prop`	high	v3	An inbound pivot or join from a named form, such as `<- edge:refs`. 3.x removed it, and the `edge:` and `graph:` forms it pivots from are light edges in 3.x, so use a walk such as `<(refs)- *`.
`R008`	`dotted-prop`	high	v3	A prop name with a dotted segment which is not a universal prop, such as `:foo.bar` or `inet:dns:a.newp`. 3.x reads the dotted segment as a virtual property.
`M001`	`renamed-form`	high	v3	A form which is renamed in 3.x is lifted, filtered, pivoted to or added, such as `inet:ipv4` (`inet:ip`), `hash:` (`crypto:hash:`) or `it:prod:soft` (`it:software`). A full prop name of the form is reported too.
`M002`	`removed-form`	high	v3	A form which is removed in 3.x is lifted, filtered, pivoted to or added, such as `inet:ssl:cert`, `inet:web:*` or `it:av:sig`. The message names the replacement when there is one.
`M003`	`renamed-prop`	high	v3	A prop which is renamed or removed in 3.x is used, such as `inet:dns:a:ipv4` (`inet:dns:a:ip`), `ou:org:orgtype` (`ou:org:type`), `risk:vuln:cve` (`risk:vuln:id`) or the `it:av:scan:result:target:*` props (`it:av:scan:result:target`). The `:ipv4`, `:ipv6`, `:ipv4s`, `:ipv6s`, `:net4` and `:net6` props of forms which 3.x keeps are all reported, such as `inet:server:ipv4` (the `.ip` virtual prop of `inet:server`). A relative prop is only checked when its form is known: in the edit block of a node add, or after a single-form lift which starts its query.
`M004`	`univ-seen`	high	v3	The universal `.seen` prop is lifted, filtered, read, set or deleted. 3.x has no universal props, use `:seen` on forms which implement `meta:observable`.

Checking Storm for Synapse 3.x

--v3-ready, or v3_ready = true in the configuration, adds the rules which report Storm that must change before it runs on Synapse 3.x. The Storm is still parsed by the Synapse 2.x parser, so these rules report code which runs on 2.x today:

The R rules report Storm which fails, or which runs with a different meaning, on 3.x. These include the $lib constants and APIs which were removed, gtors which can no longer be called, bare true, false and null values, whitespace before a . deref, inbound pivots from a named form or prop, and dotted prop names which 3.x reads as virtual properties.
The M rules report data model forms and props which are renamed or removed in 3.x, and the universal .seen prop. They check the form and prop names used in lifts, filters, pivots, node adds and property sets. A relative prop such as :ipv4 is only checked when its form is known: inside the edit block of a node add, or after a single-form lift which starts its query. The message names the 3.x replacement when there is one. Storm which names a removed or renamed form or prop fails on 3.x, such as inet:ipv4=1.2.3.4 or inet:fqdn +.seen, so the data and the Storm must both be migrated.

The R and M findings have high or medium severity and fail the check like any other finding. Use --ignore M to skip the M rules, or --select M to list only them.

The R and M rules may also be run without --v3-ready, by naming them with --select or --enable.

Note

The R and M rules cover the documented 3.x changes which can be found from Storm syntax alone. Review the Synapse 3.x change notes for changes which need more context, such as renamed Storm commands.

Linting from Python

The synapse.lib.stormlint.StormLinter class lints Storm text directly. It takes the same select, ignore and enable options as the tool, and a v3ready option which matches --v3-ready. Its lint() method returns a list of Finding tuples, each with rule, severity, line, column and mesg fields, sorted by line, column and rule:

import synapse.lib.stormlint as s_stormlint

linter = s_stormlint.StormLinter(enable=('S015',), v3ready=True)
for finding in linter.lint('$x = $lib.true'):
    print(finding.rule, finding.line, finding.column, finding.mesg)

By default the rules which check form and property names, such as S011 and S012, use the built-in data model. Pass a data model dictionary, in the shape returned by the Cortex getModelDict() API, as model to check names against that model instead, including its extended model forms and properties:

linter = s_stormlint.StormLinter(model=await core.getModelDict())

StormLinter raises BadArg for an unknown rule ID or prefix. lint() honors noqa comments; the [tool.stormlint] table is only read by the storm.lint tool.