Synapse 2.0.0 Changes¶

Features and Enhancements¶

Refactored Layer Storage Format

This release includes significant storage optimizations for both performance and size. However, the 2.0.0 migration contains no model migrations and is strictly limited to the internal LMDB layer storage format. The new format provides performance enhancements that significantly improve data ingest performance and reduce the memory footprint of the layer. See 0.1.x to 2.x.x Migration for details on migrating your 0.1.x Cortex to 2.0.0.

View/Layer Management

Views and layers may now be managed via simple storm commands or manipulated by the storm API. Additionally, permissions may be assigned on a per-view or per-layer basis allowing granular control of users and roles. With proper permission, a view may now be easily forked to create an analyst owned “sandbox” where their edits are written to a separate layer which can later be merged by someone with appropriate permissions. A user may now set a “default view” profile option to specify the view to be used when they issue a default storm query.

Mirrors Are Now A Fully Operational Battle Station

Previous support for Cortex “mirrors” has been limited to mirroring data from a single layer. The mechanism for change control and distribution within a Cortex has been updated to facilitate true mirroring of all changes within a Cortex. Each change within a Cortex is assigned a sequential identifier which allows mirrors to be taken offline and catch-up once they are returned to operation. Additionally, mirrors have been updated to facilitate “write-back” edits which propagate changes via their upstream mirror, making the mirror appear writable. With this new mechanism, a set of mirrors may be configured to handle read offloading as well as write consolidation.

Cron/Trigger Management

Cron jobs and triggers may now be managed via simple storm commands or manipulated by the storm API. Additionally, permissions are now enforced on a per-object basis.

Experimental Spawn Runtime Improvements

The (still experimental) spawn: True option to the storm runtime, which facilitates execution of a storm query in a sub-process, has made progress toward feature parity. Specifically, a number of storm library APIs have been updated to facilitate access from within a spawned sub-process transparently. While the spawn option is still considered experimental, it has reached a level of maturity which may warrant review due to the powerful performance benefit for large or long running read queries.

Backward Compatibility Breaks¶

Modified Node Permission Names¶

What changed

Node permissions were modified to allow all basic node edits to fall within a single hierarchy in order to facilitate easily granting permissions for all node edits.

The following permission table provides translation:

Old Permission	New Permission
node:add	node.add
node:del	node.del
prop:set	node.prop.set
prop:del	node.prop.del
tag:add	node.tag.add
tag:del	node.tag.del

Why make the change

By placing all node edit permissions under a single hierarchy, administrators may easily grant access to all node modifications by adding the permission node. As previously, more granular versions of the given permissions are available to individually control access.

What you need to do

The Synapse 2.0 migration tool should take care of translating all relevant permissions changes within a migrated Cortex. However, any 3rd party code that modifies permissions should be updated to the new naming convention.

Node Edits vs Splices¶

What changed

The various splice events provided by the storm runtime have been consolidated and streamlined into a single node:edits event type. The new node:edits events contain a more compressed and aggregated representation of node changes and include changes that were previous unreported. Callers may now specify the format for the node edits reported by the storm runtime by using the editformat optional parameter to the storm runtime, but the default output will now be node:edits rather than splices. The following table enumerates the options for the editformat parameter:

editformat	Storm runtime behavior
nodeedits	Provide new `node:edits` events.
splices	Provide `0.1.x` backward-compatible splice events.
count	Provide `node:edits:count` events with simple counts.
none	Do not transmit any representation for node edits.

Why make the change

The splice format was originally designed to facilitate a single atomic edit per-splice. As such, it required the potentially large primary property value to be embedded in each splice. When making multiple edits, this representation is inefficient and causes the retransmission, and potential storage, of duplicate data. Additionally, the key-value structure of the splice format provided unnecessary extensibility at the cost of transmission/storage size.

What you need to do

Update any code that consumes/indexes the various splice events to handle the new node:edits format. Additionally, callers may specify editformat: "splices" within their storm runtime options to enable backward-compatible splice generation. External APIs for retrieving splices have been marked as deprecated and will be removed in v3.0.0.

Removed Remote Layers¶

What changed: Support for remote layers has been removed.
Why make the change: The performance characteristics and stability of remote layers has never reached what we consider production deployable status. Additionally, complexities with model versions, migrations, and model synchronization have made the use of remote layers highly fragile. While we may eventually design a new remote layer capability, the current implementation is being removed due to being unsupportable.
What you need to do: If you have remote layers deployed in production, you should update the view configuration to contain an “upstream” layer. This will create a copy of the remote layer data to the local Cortex and keep it in sync.

Removed Pushing Splices¶

What changed: The configuration options to enable pushing splices to a cryotank or to another cortex have been removed.
Why make the change: The archival of splices to a cryotank and the responsibility of a Cortex to “push” changes to another Cortex have long been essentially vestigial. Additionally, these options required a Cortex reboot to take effect and were not runtime configurable. The current mechanisms for mirroring and upstream layers allow for a more scalable and dynamic configuration.
What you need to do: It is unlikely that this change will effect any known deployments.

Removed Monolithic Feed Configuration¶

What changed: The monolithic configuration option for pulling “feed” data from a Cryotank has been removed.
Why make the change: The ability to feed a Cortex directly from a Cryotank represents a very early approach to automate data ingest into a Cortex. This capability has been superseded by Storm Services which provide a dynamically configurable way to integrate services and data.
What you need to do: It is unlikely that this change will effect any known deployments.

Removed Tag Prop Lifting Without Tag¶

What changed: The ability to lift nodes by the presence of a tag property without specifying the tag name has been removed. Given a tag property of “confidence”, the #:confidence and #:confidence>90 style syntax are no longer valid. However, lifting by tag property with the tag, such as #foo.bar:confidence and #foo.bar:confidence>90 remain valid.
Why make the change: The necessary indexing to provide a performant way to lift nodes by the tag property without the tag is too expensive for the analytically dubious use case.
What you need to do: Any instances of lifting nodes by tag property without the tag will need to be updated to include the tag name.

Removed Insecure Mode¶

What changed: The “insecure” option in cell.yaml has been removed.
Why make the change: Insecure mode of operation was a vestigial option originally designed to aid in bootstrapping and setting up initial admin users. Telepath now allows for cell:// and unix:// connection schemes that can bypass authentication for local users making insecure mode unnecessary. Additionally, it is currently possible to bootstrap a root password directly using command line arguments, environment variables, or configuration files.
What you need to do: If you have services deployed in insecure mode, they will need to be transitioned to using proper authentication.

Removed Default Values From Model¶

What changed: Model properties may no longer have default values.
Why make the change: The root reason for this change is a complex cascade of requirements which hinge on the simple concept of populating a default value. In Synapse 2.x, nodes may be created and edited without lifting them. This means that ingest speeds can be significantly increased by taking an “upsert” approach. However, it also has the side effect of making it very difficult to know if a given node already has a value specified in another layer without lifting and fusing the node from all the properties in all the layers within the view. Ultimately, by removing the expectation of default values for a given property, we have been able to allow the Cortex to create nodes without needing to lift them, creating a large performance benefit.
What you need to do: If you have custom model elements that have default values, they will no longer be populated by default. As a work around, you may create a trigger which populates the property when the node is added, but use caution when merging properties from multiple layers when populating defaults.

Node Data Fields Must Be JSON Compatible¶

What changed: Node properties of type data as well as untyped node data must be JSON serializable.
Why make the change: Node properties of type data are stored using msgpack serialization, however many of our externally facing APIs use JSON serialization. By adding the requirement that these fields are JSON compatible, we prevent issues which would preclude nodes from being returned via HTTP APIs.
What you need to do: If you’re using data fields or node data to store raw bytes or other data structures that are incompatible with JSON serialization, you will need to migrate these values to JSON compatible structures prior to running your 2.0.0 migration.

Storm Command Argument Passing¶

What changed

The Storm command architecture has been updated to allow passing variables, properties, and tag values as parameters. As such, a couple commands have had parameters change to bring them in line with the new convention.

Why make the change

This change allows dynamic resolution of command line arguments on a per-node basis and standardizes a mechanism for giving inputs to commands. Several variants of similar behavior existed in one-off implementations and this change standardizes the various emerging patterns into a single supported design pattern.

What you need to do

The following storm commands have changed:

storm cmd	Description of the change	Old Use Example	New Use Example
movetags	No longer takes `#` prefixes on tags.	`movetags #foo.bar #baz.faz`	`movetags foo.bar baz.faz`
min/max	No longer take full property names	`min file:bytes:size`	`min :size`
scrape	Argument convention refactored.	`scrape --props foo`	`scrape :foo`

Centralized Cortex Configuration¶

What changed

Standing up a Cortex was previously configured with a pair of files, cell.yaml and boot.yaml, in addition to command line arguments provided to to the server, synapse.servers.cortex. The Cortex (and other Cell) startup configuration data has been centralized. The boot.yaml file has been removed, along with the auth:admin value. auth:passwd has been added to the cell.yaml file that can be used to set the password for the root user. In addition, the command line server can be used to pass various configuration elements, and may also be used to pass configuration elements in via environment variables. Configuration structures are also schema validated, to prevent erroneous values from being set by users.

Why make the change

This change reduced the overhead in managing and running a Cortex (and other Cells) by centralizing configuration to a single location, as well as supporting environment variables for all boot-time configuration options.

What you need to do

The auth:admin value in any boot.yaml, after splitting off the username, would need to be moved to a cell.yaml file.

$cat boot.yaml
---
auth:admin: root:superSekri7!
...

# Updated into the new cell.yaml file
$cat cell.yaml
---
auth:passwd: superSekri7!
...

Streamlined Expression Syntax¶

In order to streamline the arithmetic expression syntax, the $( ) “dollar” expressions no longer require a leading $. The only backward compatibility issue is for explicit “list of one item” declarations in storm. Any list containing only one element must have a trailing , similar to python “tuple” syntax. Essentially $var = (foo) as a list of one element needs to be updated to $var = (foo,). All other list declarations such as $var = () and $var = (foo,bar) are unaffected.

Additional Changes¶

map_async is now enabled by default for all slabs.
Synapse tools may not be used to connect to services of a different minor version.
Storm API methods now support user-impersonation by providing a user iden in the opts dictionary. This ability is permission enforced.
Deprecated annotations added to APIs that will be removed in 3.0.0. This includes the Cortex .eval() API.
The sudo command has been deprecated and does nothing.
Removed cortex offset storage.
SYNDEV_OMIT_FINI_WARNS was added to silence tear down warnings.
Provenance is disabled by default. Enable by setting provenance:en: True in cell.yaml.
The CellApi @adminapi decorator now must be called as a function, @adminapi().
The CellApi’s used for managing users and roles have been updated to be iden oriented, as opposed to being name oriented. User and Role management APIs have also been exposed via Storm.
The CoreApi.splices() method now takes a nodeedit offset tuple instead of an integer. It now yields a offset, splice tuple together instead of just splices.
Added lightweight edges to allow for digraph edge creation in the graph which are lighter than making Edge type of nodes.
Added Storm Macro support, to allow for storing and executiong stored Storm queries.
Complete View and Layer management interface available inside of Storm.
Complete User and Permissions management interface available inside of Storm.