Synapse Changelog

v2.7.3 - 2020-09-16

Deprecations

  • The 0.1.x to 2.x.x Migration tool and and associated Cortex sync service will be removed from Synapse in the 2.9.0 release. In order to move forward to 2.9.0, please make sure that any Cortexes which still need to be migrated will first be migrated to 2.8.x prior to attempting to use 2.9.x. (#1880)

Bugfixes

  • Remove duplicate words in a comment. This was a community contribution from enadjoe. (#1874)
  • Fix a nested Nexus log event in Storm Service deletion. The del event causing Storm code execution could lead to nested Nexus events, which is incongruent with how Nexus change handlers work. This now spins off the Storm code in a free-running coroutine. This does change the service del semantics since any support Storm packages a service had may be removed by the time the handler executes. (#1876)
  • Fix an issue where the cull parameter was not being passed to the multiqueue properly when calling .gets() on a Storm Types Queue object. (#1876)
  • Pin the nbconvert package to a known working version, as v6.0.0 of that package broke the Synapse document generation by changing how templates work. (#1876)
  • Correct min and max integer examples in tagprop documentation and tests. (#1878)

v2.7.2 - 2020-09-04

Features and Enhancements

  • Update tests for additional test code coverage. This was a community contribution from blackout. (#1867)
  • Add implicit links to documentation generated for Storm services, to allow for direct linking inside of documentation to specific Storm commands. (#1866)
  • Add future support for deprecating model elements in the Synapse data model. This support will produce client and server side warnings when deprecated model elements are used or loaded by custom model extensions or CoreModules. (#1863)

Bugfixes

  • Update FixedCache.put() to avoid a cache miss. This was a community contribution from blackout. (#1868)
  • Fix the ioloop construction to be aware of SYN_GREEDY_CORO environment variable to put the ioloop into debug mode and log long-running coroutines. (#1870)
  • Fix how service permissions are checked in $lib.service.get() and $lib.service.wait() Storm library calls. These APIs now first check service.get.<service iden> before checking service.get.<service name> permissions. A successful service.get.<service name> check will result in a warning to the client and the server. (#1871)

v2.7.1 - 2020-08-26

Features and Enhancements

  • Refactor an Axon unit test to make it easier to test alternative Axon implementations. (#1862)

Bugfixes

  • Fix an issue in synapse.tools.cmdr where it did not ensure that the users Synapse directory was created before trying to open files in the directory. (#1860) (#1861)

Improved Documentation

  • Fix an incorrect statement in our documentation about the intrinsic Axon that a Cortex creates being remotely accessible. (#1862)

v2.7.0 - 2020-08-21

Features and Enhancements

  • Add Telepath and HTTP API support to set and remove global Storm variables. (#1846)
  • Add Cell level APIs for performing the backup of a Cell. These APIs are exposed inside of a Cortex via a Storm Library. (#1844)
  • Add support for Cron name and doc fields to be editable. (#1848)
  • Add support for Runtime-only (runt) nodes in the PivotOut operation (``-> * ``). (#1851)
  • Add :nicks and :names secondary properties to ps:person and ps:persona types. (#1852)
  • Add a new ou:position form and a few associated secondary properties. (#1849)
  • Add a step to the CI build process to smoke test the sdist and wheel packages before publishing them to PyPI. (#1853)
  • Add support for representing nodedata in the command hinting for Storm command implementations and expose it on the syn:cmd runt nodes. (#1850)
  • Add package level configuration data to Storm Packages in the modconf value of a package definition. This is added to the runtime variables when a Storm package is imported, and includes the svciden for packages which come from Storm Services. (#1855)
  • Add support for passing HTTP params when using $lib.inet.http.* functions to make HTTP calls in Storm. (#1856)
  • Log Storm queries made via the callStorm() and count() APIs. (#1857)

Bugfixes

  • Fix an issue were some Storm filter operations were not yielding CPU time appropriately. (#1845)

Improved Documentation

  • Remove a reference to deprecated eval() API from quickstart documentation. (#1858)

v2.6.0 - 2020-08-13

Features and Enhancements

  • Support +hh:mm and +hh:mm timezone offset parsing when normalizing time values. (#1833)
  • Enable making mirrors of Cortex mirrors work. (#1836)
  • Remove read-only properties from inet:flow and inet:http:request forms. (#1840)
  • Add support for setting nodedata and light edges in the syn.nodes ingest format. (#1839)
  • Sync the LMDB Slab replay log if it gets too large instead of waiting for a force commit operation. (#1838)
  • Make the Agenda unit tests an actual component test to reduce test complexity. (#1837)
  • Support glob patterns when specifying files to upload to an Axon with synapse.tools.pushfile. (#1837)
  • Use the node edit metadata to store and set the .created property on nodes, so that mirrors of Cortexes have consistent .created timestamps. (#1765)
  • Support parent runtime variables being accessed during the execution of a macro.exec command. (#1841)
  • Setting tags from variable values in Storm now calls s_stormtypes.tostr() on the variable value. (#1843)

Bugfixes

  • The Storm tree command now catches the Synapse RecursionLimitHit error and raises a StormRuntimeError instead. The RecursionLimitHit being raised by that command was, in practice, confusing. (#1832)
  • Resolve memory leak issues related to callStorm and Base object teardowns with exceptions. (#1842)

v2.5.1 - 2020-08-05

Features and Enhancements

  • Add performance oriented counting APIs per layer, and expose them via Stormtypes. (#1813)
  • Add the ability to clone a layer, primarily for benchmarking and testing purposes. (#1819)
  • Update the benchmark script to run on remote Cortexes. (#1829)

Bugfixes

  • Sanitize passwords from Telepath URLs during specific cases where the URL may be logged. (#1830)

Improved Documentation

  • Fix a few typos in docstrings. (#1831)

v2.5.0 - 2020-07-30

Features and Enhancements

  • Refactor the Nexus to remove leadership awareness. (#1785)
  • Add support for client-side certificates in Telepath for SSL connections. (#1785)
  • Add multi-dir support for CertDir. (#1785)
  • Add a --no-edges option to the Storm graph command. (#1805)
  • Add :doc:url to the syn:tag form to allow recording a URL which may document a tag. (#1805)
  • Add CoreApi.reqValidStorm() and a /api/v1/reqvalidstorm Cortex HTTPAPI endpoint to validate that a given Storm query is valid Storm syntax. (#1806)
  • Support Unicode white space in Storm. All Python s (Unicode white space + ASCII separators) is now treated as white space in Storm. (#1812)
  • Refactor how StormLib and StormPrim objects access their object locals, and add them to a global registry to support runtime introspection of those classes. (#1804)
  • Add smoke tests for the Docker containers built in CircleCI, as well as adding Docker healthchecks to the Cortex, Axon and Cryotank images. (#1815)
  • Initialize the names of the default view and layer in a fresh Cortex to default. (#1814)
  • Add HTTPAPI endpoints for the Axon to upload, download and check for the existend of files. (#1817) (#1822) (#1824) (#1825)
  • Add a $lib.bytes.has() API to check if the Axon a Cortex is configured with knows about a given sha256 value. (#1822)
  • Add initial model for prices, currences, securities and exchanges. (#1820)
  • Add a :author field to the it:app:yara:rule form. (#1821)
  • Add an experimental option to set the NexusLog as a map_async slab. (#1826)
  • Add an initial transportation model. (#1816)
  • Add the ability to dereference an item, from a list of items, in Storm via index. (#1827)
  • Add a generic $lib.inet.http.request() Stormlib function make HTTP requests with arbitrary verbs. (#1828)

Bugfixes

  • Fix an issue with the Docker builds for Synapse where the package was not being installed properly. (#1815)

Improved Documentation

  • Update documentation for deploying Cortex mirrors. (#1811)
  • Add automatically generated documentation for all the Storm $lib... functions and Storm Primitive types. (#1804)
  • Add examples of creating a given Form to the automatically generated documentation for the automatically generated datamodel documentation. (#1818)
  • Add additional documentation for Cortex automation. (#1797)
  • Add Devops documentation for the list of user permissions relevant to a Cell, Cortex and Axon. (#1823)

v2.4.0 - 2020-07-15

Features and Enhancements

  • Update the Storm scrape command to make refs light edges, instead of edge:refs nodes. (#1801) (#1803)
  • Add :headers and :response:headers secondary properties to the inet:http:request form as Array types, so that requests can be directly linked to headers. (#1800)
  • Add :headers secondary property to the inet:email:messaage form as Array types, so that messages can be directly linked to headers. (#1800)
  • Add additional model elements to support recording additional data for binary reverse engineering. (#1802)

v2.3.1 - 2020-07-13

Bugfixes

  • Prohibit invalid rules from being set on a User or Role object. (#1798)

v2.3.0 - 2020-07-09

Features and Enhancements

  • Add ps.list and ps.kill commands to Storm, to allow introspecting the runtime tasks during (#1782)
  • Add an autoadd mode to Storm, which will extract basic indicators and make nodes from them when executed. This is a superset of the behavior in the lookup mode. (#1795)
  • Support skipping directories in the synapse.tools.backup tool. (#1792)
  • Add prefix based lifting to the Hex type. (#1796)

Bugfixes

  • Fix an issue for prop pivot out syntax where the source data is an array type. (#1794)

Improved Documentation

  • Add Synapse data model background on light edges and update the Storm data modification and pivot references for light edges. (#1784)
  • Add additional terms to the Synapse glossary. (#1784)
  • Add documentation for additional Storm commands. (#1784)
  • Update documentation for Array types. (#1791)

v2.2.2 - 2020-07-03

Features and Enhancements

  • Add some small enhancements to the Cortex benchmarking script. (#1790)

Bugfixes

  • Fix an error in the help for the macro.del command. (#1786)
  • Fix rule indexing for the synapse.tools.cellauth tool to correctly print the rule offsets. (#1787)
  • Remove extraneous output from the Storm Parser output. (#1789)
  • Rewrite the language (and private APIs) for the Storm model.edge related commands to remove references to extended properties. That was confusing language which was unclear for users. (#1789)
  • During 2.0.0 migrations, ensure that Cortex and Layer idens are unique; and make minimum 0.1.6 version requirement for migration. (#1788)

v2.2.1 - 2020-06-30

Bugfixes

  • The Axon test suite was missing a test for calling Axon.get() on a file it did not have. This is now included in the test suite. (#1783)

Improved Documentation

  • Improve Synapse devops documentation hierarchy. Add note about Cell directories being persistent. (#1781)

v2.2.0 - 2020-06-26

Features and Enhancements

  • Add a postAnit() callback to the synapse.lib.base.Base() object which is called after the __anit__() call chain is completed, but before Base.anit() returns the object instance to the caller. This is used by the Cell to defer certain Nexus actions until the Cell has completed initializing all of its instance attributes. (#1768)
  • Make synapse.lib.msgpack.en() raise a SynErr.NotMsgpackSafe exception instead of passing through the exception raised by msgpack. (#1768)

Bugfixes

  • Add a missing toprim() call in $lib.globals.set(). (#1778)
  • Fix an issue in the quickstart documentation related to permissions. Thank you enadjoe for your contribution. (#1779)
  • Fix an Cell/Cortex startup issue which caused errors when starting up a Cortex when the last Nexus event was replayed. This has a secondary effect that Cell implementers cannot be making Nexus changes during the __anit__ methods. (#1768)

Improved Documentation

  • Add a minimal Storm Service example to the developer documentation. (#1776)
  • Reorganize the Synapse User Guide into a more hierarchical format. (#1777)
  • Fill out additional glossary items. (#1780)

v2.1.2 - 2020-06-18

Bugfixes

  • Disallow command and bare string contensts from starting with // and /* in Storm syntax. (#1769)

v2.1.1 - 2020-06-16

Bugfixes

  • Fix an issue in the autodoc tool which failed to account for Storm Service commands without cmdargs. (#1775)

v2.1.0 - 2020-06-16

Features and Enhancements

  • Add information about light edges to graph carving output. (#1762)
  • Add a geo:json type and geo:place:geojson property to the model. (#1759)
  • Add the ability to record documentation for light edges. (#1760)
  • Add the ability to delete and set items inside of a MultiQueue. (#1766)

Improved Documentation

  • Refactor v2.0.0 changelog documentation. (#1763)
  • Add Vertex branding to the Synapse documentation. (#1767)
  • Update Backups documentation in the Devops guide. (#1764)
  • Update the autodoc tool to generate documentation for Cell confdefs and StormService information. (#1772)
  • Update to separate the devops guides into distinct sections. (#1772)
  • Add documentation for how to do boot-time configuration for a a Synapse Cell. (#1772)
  • Remove duplicate information about backups. (#1774)

v2.0.0 - 2020-06-08

Initial 2.0.0 release. See Synapse 2.0.0 Changes for notable new features and changes, as well as backwards incompatible changes.

v0.1.X Changelog

For the Synapse 0.1.x changelog, see 01x Changelog located in the v0.1.x documentation.