Synapse Changelog

v0.1.58 - 2019-04-10

Features and Enhancements

  • Add Storm syntax for easily adding and removing elements from array properties. (#1670) (#1672)
  • Add risk model with attack, campaign and vulnerability forms. (#1665) (#1673)
  • Add Storm tree command to facilitate recursive tree walking of nodes. (#1665) (#1673)
  • Add :rss:feed property to media:news form. This was cherry-picked from master. (#1674)

v0.1.57 - 2019-04-03

Features and Enhancements

  • Add an initial auth model.

  • Add filterinput and yieldfiltered flags to the graph rules.

  • Add $lib.warn() to stormtypes

  • Add synapse.lib.spool.

  • Add synapse.lib.base.Base.ctx() helper.

    These changes were all cherry-picked from master. (#1657) (#1660)

v0.1.56 - 2019-03-27

Features and Enhancements

  • Unify syntax checking into setup.cfg. This was cherry-picked from master. (#1632)

Bugfixes

  • Fix a Telepath Proxy call issue which could return an errant None. This was cherry-picked from master. (#1636)
  • Fix an issues with the cron command not displaying Monday’s properly. This was cherry-picked from master. (#1646)

v0.1.55 - 2019-03-17

Bugfixes

  • Packaging only release. The readme-renderer library had a incompatible version which prohibited CI builds from completing to publish v0.1.54.

v0.1.54 - 2019-03-17

Features and Enhancements

  • Add provisional support to Telepath to allow it to do discovery of service information via Hasicorp Consul. The semantics of how this works is subject to change in the future. (#1574) (#1610)

Improved Documentation

  • Add an Storm graph command example to the userguid. (#1538)

v0.1.53 - 2019-03-02

Features and Enhancements

  • Add support for loading YAML files when using the --optsfile argument to the Cmdr CLI to execute a Storm query. (#1554)

Bugfixes

  • Make the Agenda subsystem respect leap years properly. (#1550)
  • Make the trigger.add Storm command add the trigger as the current user. (#1549)

v0.1.52 - 2019-02-27

Features and Enhancements

  • Allow the benchmarking tool to save records off in JSON format. Add a script to post-process records for aggregation. (#1535) (#1544)
  • Add per-user default View options. (#1537)
  • Add client and associated subs to the inet:web:mesg and inet:web:post forms. (#1545)
  • Lock the command line tools in synapse.tools with interact with remote Cortex and Cell objects to only work if the remote objects are also served up from 0.1.x server code. (#1546)
  • Embed Synapse version information into the Cell and Cortex objects for future version checking. (#1547)

Bugfix

  • Corrected spelling for ou:org:dissolved. This secondary property was previously disolved. This does have an associated model migration. (#1527)
  • Corrected an issue with the hide-code library breaking our documentation tests. (#1527)
  • Fix an issue where norming a geo:latlong value could produce an different value from the input system-mode value. (#1539)
  • Fix an issue with the memory lock loop tests to be more reliable. (#1543)

Improved Documentation

  • Update User Guide documentation with additional glossary documentation. (#1540)

v0.1.51 - 2019-02-13

Features and Enhancements

  • Add inet:whois:ipquery, inet:whois:iprec, inet:whois:ipcontact and inet:whois:regid forms to track IP WHOIS information. Add a $lib.inet.whois.guid() Stormtypes function that can be used to assist in stable guid generation for the inet:whois:iprec forms. (#1495)

Bugfixes

  • Fix an issue where Provenance claims did not contain the user iden during a feed:data claim. (#1531)
  • Fix an issue where tag:add splices did not record oldv. (#1534)

v0.1.50 - 2019-02-07

Features and Enhancements

  • Add Storm commands for managing triggers and cron jobs from inside of Storm directly. The existing trigger and cron commands available in the Cmdr tool have been marked as PendingDeprecation and will be removed in 0.3.0. Cmdr will emit these warnings directly if used. Related Telepath APIs have also been marked for PendingDeprecation. Cortex administrators can enable Python warnings to see those APIs being used. (#1513)

v0.1.49 - 2019-02-03

Bugfixes

  • Fix an issue where pivot operations which pointed to syn:splice nodes raised exceptions which could teardown a Storm runtime unexpectedly. Attempting to directly lift syn:splice nodes by primary or secondary properties now generates no nodes. (#1529)

v0.1.48 - 2019-02-03

Features and Enhancements

  • Add ou:conference:event and ou:conference:attendee forms to facilitate more fine grained conference tracking. (#1523)
  • Make the syn.nodes ingest aware of tagprops. (#1525)

Bugfixes

  • Make the LMDB Memory Locking loop safer. (#1522) (#1526)
  • Address a test race encountered during the teardown of Link objects where server side CellAPI instances were not having fini() called. (#1524)

v0.1.47 - 2019-01-30

Bugfixes

  • Fix a bug related to LMDB Abbreviation helpers where it would be off by on restarts. (#1518)
  • Fix issues related to the memory locking code used by an LMDB slab. (#1516)
  • Fix a bug related to Tagprop indexing preventing the correct storage of multiple tagprops of the same name on a node. (#1520)

v0.1.46 - 2019-01-17

Features and Enhancements

  • Add StormTypes and StormCmd support for enumerating and working with Cortex View’s via the Storm runtime. (#1507)
  • In StormCmd’s implemented via pure Storm (PureCmd), push the parent runtime variables down into the subruntime when there are no nodes feed into the cmd during execution. (#1479)
  • Harden the syn.nodes ingest function so that it tries to make every input node given to it, instead of potentially exiting on an error. (#1511)
  • Make the $lib.time.sleep() and $lib.time.ticker() StormType functions flush the Snap cache of nodes. This can prevent Node caching issues for StormDmon users. (#1512)
  • Add $node.pack() function to the $node StormType object to return the packed node inside of the Storm runtime. (#1514)

Improved Documentation

  • Update User Guide documentation for Storm Variables. (#1509)

v0.1.45 - 2019-01-17

Features and Enhancements

  • Rewrote the base Synapse benchmark script for a modern Cortex. (#1503)
  • StormDmon objects now respect the view argument in their stormopts structure, and the $lib.dmon.add() StormType function will capture the current View iden to populate that structure. (#1504)
  • Add Storm commands to list and undo splices. (#1496)
  • Add the API plumbing to allow a forked View to have its splices merged back into the parent view. (#1505)

Improved Documentation

  • Update User Guide documentation for Tag Properties (tagprops). (#1502)

v0.1.44 - 2019-01-08

Features and Enhancements

  • Add synapse.lib.spawn to implement support for multiprocessing-based Cortex parallel query execution. This allows parallel queries to be executed by the Cortex in a manner that offloads the processing for that query to a separate process. The maximum number of processes which are kept around (but not the maximum number of possible parallel queries) is configured via the Cortex configuration option spawn:poolsize, which defaults to 8. Users may opt-into this behavior by setting the flag spawn to True in the opts dictionary passed to a Storm query. Attempting to create or edit nodes in the graph with a Spawn query will fail. The support for this is currently experimental. (#1400) (#1499) (#1500)
  • Add a +?# try operator for setting tags in Storm edit blocks. This will silently fail to set the tag if the tag or tag value is not valid. (#1465) (#1498)
  • Add support for iterating backwards through LMDB slabs and iterating backwards through Splice history. (#1473)
  • Add a config flag to the Cortex, splices:en, that can be used to disable storage of Splices. (#1488)
  • Push Storm opts into task information dictionaries and clamp the output of the cmdr ps command by default. Add a -v switch to the ps command to avoid clamping. (#1476) (#1499)
  • Add tel:call and tel:txtmesg Guid forms to the datamodel to track telephone calls and text messages. (#1483) (#1493)
  • Firm up the API boundary between the Snap and Cortex object. Referencing the Cortex from the Snap, instead of calling APIs on the Snap, should be avoided in any new code. (#1400)
  • Add special handling for creating read-only LMDB databases. (#1400)
  • The Daemon t2call implementation was moved to its own function and support for DmonSpawn added to the t2call routine. (#1400)

Bugfixes

  • Add missing inline terminal names in the Storm grammar to assist in parsing Syntax errors. (#1487) (#1498)
  • Convert Core.nodes() from a direct call to returning the result of a scheduled coroutine. Avoids a potential teardown issue in test code. (#1490)

Improved Documentation

  • Update User Guide documentation for Array types. (#1486) (#1489)

v0.1.43 - 2019-01-02

Features and Enhancements

  • Allow users to list their current permissions and roles with the synapse.tools.cellauth tool. (#1460) (#1464)
  • Allow bytes to be converted to strings, and strings to be converted to bytes, inside of Storm. This is done with decode() and encode() StormType functions. (#1469)

Bugfixes

  • Fix an error in the synapse.lib.scrape regular expressions which prevented sequential FQDN and email addresses from being scrapped. (#1482)

Improved Documentation

  • Start a Synapse glossary document. (#1478)
  • Update the Storm Command reference documentation. (#1478)

v0.1.42 - 2019-12-24

Features and Enhancements

  • Allow for StormType query objects to be executed similar to a local function. (#1456)
  • Add syn:cmd runt nodes to represent Storm commands available to a given Cortex as Nodes. (#1446)

Bugfixes

  • Make CI tests do eager updates of third party libraries to prevent cache key rolling when a buggy library is published and a subsequent fix is released. (#1468)
  • Fix implicit pivot support for runt nodes. (#1470)
  • Allow the Storm tee command to execute queries if there were no inbound nodes. (#1468)
  • Allow the PropPivotOut AST node to pivot from an array value to a runt node. (#1471)
  • Add some asyncio.sleep(0) calls to the AST where Python loops could hog CPU time. (#1472)
  • Fix an issue with the Storm scrape command where a non-str repr was not properly scrapped. (#1474)
  • Fix an issue with the Storm scrape command where a relative property was not accepted as an argument to the command. Relative property syntax is now supported. (#1474)

v0.1.41 - 2019-12-16

Features and Enhancements

  • Allow Storm expression syntax to handle None values gracefully for comparison purposes. (#1459)

Bugfixes

  • Fix a bug where removal of a tagprop didn’t update the Node object in memory. (#1454)
  • Speed up unit test execution by adding an additional Storm parsing cache. (#1455)
  • Bump CircleCI cache keys due to a bad multidict release poisoning build caches. (#1463)
  • Added an empty layer migration to prevent Cortex downgrading prior to v0.1.41. This is to prevent a user from running a Cortex on older code, as reverting a Cortex created/used with a Synapse version greater than or equal to v0.1.33 and v0.1.34, to a version prior than those, can result in apparent data loss. Data is not actually lost but would require non-trivial effort to recover. (#1458)

Improved Documentation

  • Fix a broken link. Thank you aaronst for the bug report. (#1448) (#1463)

v0.1.40 - 2019-12-10

Features and Enhancements

  • Add a refs flag to the graph rules which allow getting all refs including edges and make this the default behavior when graph option is set to True when calling storm()/eval() apis. (#1453)

Bugfixes

  • Make error messages for geo:dist type normalization more understandable. (#1447)
  • Fix permissions checks for Storm queue objects being made by non-object owners. (#1452)

Improved Documentation

  • Some small wordsmithing. (#1449)

v0.1.39 - 2019-12-03

Features and Enhancements

  • Group Storm commands which came from Storm packages together when the storm help command is issued. (#1440)
  • Add the secondary property :url to the ou:conference form to track inet:url nodes associated with a conference. (#1441)
  • Add :names as a secondary property to ou:org to allow tracking multiple, secondary names for an organization. (#1444)
  • The StormType $lib.ingest.feed() now places the runtime Snap into less-strict mode during execution to prevent potentially data causing the Storm runtime to be torn down. Errors during node creation or property sets will now cause warn messages to be emitted in the message stream. (#1442)

Bugfixes

  • Fix an issue where Storm command names were truncated when the storm help command was issued. (#1440)
  • Fix a Storm grammar issue that disallowed $(1 and 1 and 0) style of expressions. (#1439)
  • Fix the CryoApi and CryoCell delete() APIs used to remove a Cryotank from a CryoCell. (#1443)
  • Make NoSuchName errors in StormTypes more friendly by always including the missing name. (#1445)
  • When deferencing a value off of a StormType, if a NoSuchName was thrown it was unclear what type of object was being deferenced, which could lead to difficult to debug Storm code. The NoSuchName errors now includes the class name of the object to assist with debugging those errors. (#1445)

v0.1.38 - 2019-11-22

Features and Enhancements

  • Add the query text to the error message when a remote Storm query encounters a fatal exception. (#1432)
  • Add provenance claim information to provenance recursion errors for better debugging of provenance issues. (#1432)
  • Capture print events from inside of a Storm Daemon and log those to the synapse.lib.storm logger. (#1434)
  • Add $lib.list() to StormTypes to allow construction of a List StormType. (#1434)
  • Add :loc secondary property to mat:item, it:host, and geo:nloc forms. (#1437)
  • Split a Storm Daemon task into two components; a storm:dmon:main and storm:dmon:loop task. Killing the storm:dmon:loop task will cause the storm:dmon:main task to respawn the inner task executing Storm. (#1436)

Bugfixes

  • Fix a bug where a Storm Daemon coroutine was promoted to a Synapse Task with an incorrect user value. (#1435)
  • Fix a Storm Runtime variable scoping issue preventing Storm functions from calling other Storm functions in the same scope they are declared in. (#1435)
  • Fix an bug producing an ambiguity in the Storm switch case statement. This bugfix does require switch case statements with a space in them to be enclosed in single or double quotes. (#1438)

v0.1.37 - 2019-11-19

Features and Enhancements

  • Allow CryoCell and CryoTank Telepath APIs to be overridden by subclasses. (#1426)
  • Add .has() method to the Set Stormtype to allow for set membership checking. (#1429)
  • Set map_async to True for Cryotank slabs. (#1427)
  • Push the Cryotank conf value into the lmdbslab **kwargs. (#1427)
  • Add user defined functions to Storm. (#1419)
  • Add packages to Storm, which may define importable command blocks or pure Storm commands. Storm services no longer deliver commands via a separate data structure and must deliver then via Storm packages. (#1419) (#1430)
  • Allow StormSvc’s to deliver packages as part of their service configuration. (#1419)
  • Add setitem support to StormTypes. (#1419)

Bugfixes

  • Fix a bug in the CryoCell __anit__ call signature. Fix the CryoCell getCellApi() method to use referneces to self instead of direct classes to resolved Telepath APIs. (#1426)
  • Fix variable scoping issue with Pure Storm commands. (#1419)
  • Fix variale scoping issue for Storm init and fini blocks by restricting them to only be runtsafe. (#1419)
  • Fix edit block order of operations issue. (#1419)

v0.1.36 - 2019-11-07

Features and Enhancements

  • Add support for single quoted strings to the Storm switch case labels. (#1424)
  • Add StormSvc addition and deletion hooks so that a StormSvc can define commands that are executed when a service is added to a Cortex or removed from a Cortex. (#1417)

Bugfixes

  • Protect the Snap.addFeedNodes() API from a function ctor which isn’t does not return an async generator. (#1421)
  • Fix an ambiguity in the Storm switch grammar which could have produced inconsistent results. (#1422)
  • Migrate stored trigger views to the new view iden. (#1423)

v0.1.35 - 2019-11-01

Features and Enhancements

  • Add :place secondary propeerties on forms with :latlong secondary properties to allow linking nodes to geo:place nodes. (#1416)
  • Add a geo:bbox type to the datamodel to record a rectangular latitude and longitude bounding box, and a :bbox secondary property to geo:place nodes. (#1416)
  • Add init and fini Storm blocks, which can contain Storm which is executed before and after any nodes would have been consumed. (#1418)
  • Add $lib.stats.tally() to Storm Types. This gets a Tally object that can currently be used to increment named counts. (#1418)

Bugfixes

  • Storm Service call timeouts produced an unclear TimeoutError for users. This has been replaced with a StormRuntimeError. (#1415)

v0.1.34 - 2019-10-30

Bugfixes

  • Fix an issue where Storm Edit blocks could encounter exponential processing time when performing Storm query parsing. (#1414)
  • Fix an issue where the Cortex.getView() function did not return the default View when the Cortex iden was provided as the iden argument. (#1414)

v0.1.33 - 2019-10-29

Features and Enhancements

  • Allow variables to be used when dereferencing values inside of Storm.and (#1405)
  • Add $lib.feed.list(), $lib.feed.ingest(), and $lib.feed.genr() to StormTypes. These expose ingest functions registered on a Cortex to Storm. The feed.list Storm command can be used to easily list feed functions. (#1408) (#1411)
  • Make the Cortex, View and Layer iden values unique. (#1402)
  • Allow objects (Views and Layers) to enforce permissions on themselves, as opposed to globally on a Cortex. (#1384)
  • Harmonized methods which take permissions - some took a tuple, some took *path arguments. Now, all methods take a tuple for permissions. (#1384)
  • Add support for the yield keyword in Storm to allow it to yield values which come from a binary buid, a Node iden, a raw Node object; or a an (async) generator which produces the previous values. This allows $lib.* functions to be written which can inject Nodes into the Storm pipeline. (#1409)

Bugfixes

  • Fix whitespace bug in Edit Parenthesis Storm grammer. (#1407)
  • Fix bug in the runt nodes representing triggers in a Cortex. (#1406)
  • Fix the Storm Edit Parenthesis behavior to allow the first EditNodeAdd AST element to support variables. (#1412)
  • Allow values referenced off of a Node, which are not set on the Node, to be emitted through the $lib.csv.emit() function. These will be serialied with the synapse.tools.csvtool as zero length strings. (#1413)
  • Allow synapse.tools.cellauth to work with older Synapse Cells which do not support the auth apis introduced in #1384. (#1410)

v0.1.32 - 2019-10-22

Features and Enhancements

  • Add some asyncio friendly multiprocessing helpers for future use. (#1397)
  • Add initial support for syn:cron runtime only nodes to represent Cron tasks configured on a Cortex. (#1401)
  • Add a editable doc field on Cron tasks. This can be edited via Storm edit syntax on syn:cron:doc properties. (#1401)

Bugfixes

  • Fix a Daemon issue where Link message coroutines were being scheduled on the Daemon, and not the Link object. This was preventing the proper cleanup of _onTaskV2Init coroutines for async generators when they were waiting for the next item and the link had been fini’d. Now, when a Link is fini’d, any free-running coroutines associated with the a given Link will be cancelled. (#1404)

v0.1.31 - 2019-10-11

Features and Enhancements

  • Allow a user to change their password via Telepath or HTTPAPI. (#1394)
  • Add the option to print rules related to a user’s roles to the synapse.tools.cellauth tool. (#1390)
  • Add initial model for crypto currency support. (#1393)
  • Add initial model support for X509 certificates. (#1374)
  • Add :parent secondary property to geo:place form to allow for hierarchical geo:place node creation. (#1399)
  • Update Tornado to version 6.0.3. (#1391)
  • Add $lib.vars to StormTypes to provide a CRUD interface to Runtime variables. (#1396)
  • Add methods to the $path StormType to provide a CRUD interface to Path variables. (#1396)

Bugfixes

  • HiveUser.setPasswd() now requires non-empty strings as input. (#1394)
  • Fix a bug related to the normalization of the inet:dns:name type which could introduce structural untruths into the graph when normalizing a pure integer string as a DNS name. (#1395)
  • Fix a index generation bug related to Hex types not being properly truncated. (#1398)

Improved Documentation

  • Update HTTP API documentation. (#1394)

v0.1.30 - 2019-10-04

Features and Enhancements

  • Make the Telepath Proxy link pool size configurable, and expose that as a configuration value on the Telepath Client. (#1389)
  • Add a Websocket based API For receiving Cortex watch events. (#1392)
  • Add initial support for multi-value properties. (#1361)

Bugfixes

  • Remove unused test assets and fix manifest so test assets are properly grafted into packages. (#1388)

v0.1.29 - 2019-09-26

Bugfixes

v0.1.28 - 2019-09-25

Features and Enhancements

  • Add a Cortex API for watching for tag changes on nodes in the Cortex. (#1383)
  • Require explicit permission for a user to add triggers or cron job. (#1380)
  • Add a CellApi get information about current Telepath client sessions on a Daemon. (#1364)

Bugfixes

  • Fix permissions checks added in #1371 for the Boss, Triggers and Cron subsystems. (#1385)

Improved Documentation

  • Add some high level Synapse architecture notes to the Developers guide. (#1382) (#1386)

v0.1.27 - 2019-09-18

Features and Enhancements

  • Add syn:prop runtime only nodes for extramodel properties. (#1372)
  • Add syn:tagprop runtime only nodes for tagprop properties. (#1372)
  • Add :person field to ps:persona form. (#1376)
  • Add reprs for tagprops to the packed node format, and add cmdr display for tags with tagprops. (#1373)
  • Add a scrape command to Storm to enable regex based scraping of node properties for easily identifiable forms. (#1368)
  • Add explicit permissions for interacting with the trigger, cron and boss operations. (#1371)
  • Add support for remote Telepath services in Storm. (#1352)
  • Add support for implementing Storm commands in pure Storm. (#1352)
  • Add persistent, durable queue objects to Storm backed by LMDB slabs. (#1352)
  • Add support for persistent Storm daemon loops. (#1352)
  • Add a Telepath Client object. The client object is heavier than a Proxy and supports reconnects and. (#1352)
  • Add StormType $lib.time.format() to allow for formatting a timestamp value value into an arbitrary string. (#1378)
  • Internal plumbing support for a Cortex managing multiple views. (#1348)

v0.1.26 - 2019-09-12

Features and Enhancements

  • Add :serial, :model, and :manu secondary properties to the it:host form. (#1358)

Bugfixes

  • Fix an issue in Storm where double quoted string values with backslash escaped characters in double quoted strings were not being properly escaped during syntax parsing. Double quoted strings are now being processed with ast.literal_eval(). This means that double quoted string values will be processed according to Python’s string literals as seen here https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals prior to passing them into any sort of model normalization routines. Single quoted string values parsing is not affected by this change. (#1366) (#1370)
  • Fix an issue where a Daemon Share object was being tracked twice on the server side. (#1363)
  • Fix an issue where Cron tasks could start prior to CoreModules loading being finalized. (#1367)
  • Fix an issue with inconsistent test coverage for synapse.lib.link. (#1365)

v0.1.25 - 2019-09-06

Features and Enhancements

  • Add $lib.inet.http.put() Stormtypes support to allow making HTTP PUT requests. (#1358)
  • Add $llib.base64 Stormtypes to allow for manipulation of base64 data in Storm. (#1358)
  • Add healthcheck tooling that can be used to implement heartbeat support for Synapse Cells. (#1344)

Bugfixes

  • Fix an issue where the geo:dist was missing comparator support. This was fixed by caussing it to inherit from the IntBase type. (#1362)

v0.1.24 - 2019-09-03

Features and Enhancements

  • Add a granular permission checking helper to the HTTPAPI Handler base. (#1346)
  • Allow retrieval of data from a LMDB SlabSeqn object by arbitrary index bytes. (#1342)
  • Add synapse.tools.hive.save and synapse.tools.hive.load to save an load arbitrary trees of a Hive. (#1340)
  • Add support to the Cell to preload hive on first boot via a hiveboot.yaml file containing a serialized Hive tree. (#1340)
  • Add POST support to the /api/v1/storm and /api/v1/storm/nodes HTTP APIs. (#1351)
  • Ensure that a Cortex always has an Axon available. By default, the Axon will be locally stored on disk in the Cortex cell directory. This can alternatively be configured to point to a Axon URL via the axon configuration option for a Cortex. (#1349)
  • Add Stormtypes $lib.bytes.put() to allow storing a Storm variable, representing bytes, in the Axon configured for a Cortex. (#1349)
  • Add support for storing arbitrary key value data on a node. (#1347)
  • Add geo:address type to record an arbitrary address string; add :address property to geo:place form. Convert ps:contact:address to be type geo:address. This does involve a automatic data migration during Cortex startup. (#1339)
  • Fix Axon permission handling for remote users to actually enforce permissions. (#1354)
  • Add a new form, inet:url:mirror, which represents URL content being mirror between two different URLs. (#1360)
  • Add support for user defined runtime properties. (#1350)
  • Add support for user defined secondary properties to be attached to a tag. (#1350)
  • Add support for defererencing a variable value in order to lift by a variable property name. (#1350)

Bugfixes

  • Fix an issue with the kill command failing when providing a purely numeric task identifier. (#1343)
  • Fix an with logging the incorrect user value when terminating a task. (#1343)
  • Replace asyncio.sleep() calls with self.waitfini() calls in loop retry code, to ensure that tasks do not end up retrying after the object has been torn down if the ioloop is still running. (#1353)
  • Remove codecov orb and use the codecov bash uploaded directly. (#1355) (#1357)
  • Make the Storm max command aware of Ival types, and pull the maximum value based on the right hand side of the interval. (#1359)

v0.1.23 - 2019-08-14

Features and Enhancements

  • Add a new Cortex configuration option, layer:lmdb:map_async, to enable asynchronous fsync calls in LMDB layers. (#1338)
  • Add asyncio.sleep(0) calls to Telepath generator loops to enable all Telepath generators to have fair scheduling on the server side. (#1341)

v0.1.22 - 2019-08-08

Features and Enhancements

  • Add :loc secondary prop to tel:mob:telem to record geopolitcal location of a mobile telemetry node. (#1337)
  • Add :spec secondary prop to mat:item to record the specification of an instance of an item. (#1337)

Bugfixes

  • Call seek() after truncating the file descriptor backing an Axon UpLoad context. (#1336)

v0.1.21 - 2019-08-08

Features and Enhancements

  • All the Axon UpLoad context manager to be re-used after calling .save() (#1333)
  • Add Stormtypes $lib.time.parse() to parse an arbitrary date string using datetime.strptime format rules. (#1334)
  • Make NoSuchProp exceptions more informative about Node form names if that data is relevant. (#1335)

Bugfixes

  • Allow two Base implementations to be used as mixins together without disrupting their underlying teardown and observable behaviors. (#1332)

v0.1.20 - 2019-08-06

Features and Enhancements

  • Refactor Axon to allow for easier subclassing. (#1327)
  • Miscellaneous Axon improvements. (#1331)

v0.1.19 - 2019-07-25

Features and Enhancements

  • Add a new Storm command, tee, that allows for executing multiple storm queries with the input node as the input to the queries, and rejoining their output as a new stream of nodes. (#1323)

Bugfixes

  • Fix a bug in HTTP API session handling which created duplicate sessions on the server side. (#1324)
  • Fix a documentation error in the quickstart guide regarding permissions. (#1326)

v0.1.18 - 2019-07-17

Features and Enhancements

  • Allow underscores in org:alias values. (#1320)
  • Allow plain variable references in tagnames and tagmatches in Storm. For example: +#aka.$var.t42 (#1322)

v0.1.17 - 2019-07-12

Features and Enhancements

  • Add type base data to show explicit type inheritance for data model types. (#1315)

Bugfixes

  • Fix rule deletion by the synapse.tools.cellauth tool. (#1319)

Improved Documentation

  • Add additional Storm documentation edit parenthesis, try statements, and type specific behavior. (#1316)

v0.1.16 - 2019-07-11

Features and Enhancements

  • In Cmdr, the time when a Storm query is being executed by the Cortex is now emitted to the user. (#1310)
  • Implement yield keyword. The keyword “yield” before a subquery causes the output nodes of the subquery to be merged into the output stream. (#1307)
  • Allow relative and universal properties to be specified from a variable in Storm. (#1305)
  • Allow parentheses in Storm editblocks. Edit operations in parentheses don’t receive incoming nodes from left of the parentheses. (#1303)
  • For Cron tasks, expose the Storm query and their iden in the Task data structure. (#1295)
  • Allow filtering inet:fqdn properties with * wildcards, such as +inet:fqdn=*.vertex.link. (#1292)
  • Add a Bytes object to StormTypes which allows for $gzip(), $gunzip(), $bzip(), $bunzip() and $json() decoding helpers. (#1291)

Bugfixes

  • The syn:prop runtime only nodes did not have :univ=1 set on universal properties which were pushed onto the form specific properties. They now have :univ=1 set on them. (#1313)
  • Fix invalid tool name references for synapse.tools.feed and synapse.tool.pullfile. (#1311)
  • Add a missing default share name for the Axon cell. (#1309)
  • Fix that non-runtsafe loops didn’t yield nodes, they now do. (#1307)
  • Fix that non-runtsafe loops that ran 0 times yielded the inbound node. They now yield no nodes. (#1307)
  • Fix synapse.tools.csvtool help description. (#1306)
  • Fix uses of s_common genfile where opened files weren’t being truncated, or in one case, appended to. (#1304)

Improved Documentation

  • Add additional Hive API documentation. (#1308)
  • Add additional type specific documentation for Storm. (#1302)
  • Add documentation for synapse.tools.csvtool, synapse.tools.pushfile, and synapse.tools.pullfile. (#1312)

v0.1.15 - 2019-07-01

Features and Enhancements

  • Add $lib.user.vars and $lib.globals Storm Types. These allow for persistent variable storage and retrieval inside of Storm across multiple queries. These use .set(), .get(), .pop() and .list() methods on the two new Storm Types. (#1287)
  • Add an optional try operator, ?=, to the Storm edit mode blocks. This allows for node creation and property setting to fail silently on BadTypeValu and BadPropValu errors. Example: [ inet:ipv4 ?= notAnIpAddress :asn?=NotAnAsn ]. (#1288)
  • Add while loop to Storm. (#1290)
  • Add :accuracy as a secondary property to the tel:mob:telem node, so a user can record the accuracy of the tel:mob:telem:latlong property. (#1294)
  • Always interpret numbers in expressions as numbers. (#1293)
  • Add a genr argument to iterStormQuery() to better facilitate nested Storm queries. (#1297)
  • Allow headers to be set when using $lib.inet.http() in Storm. (#1299)
  • Allow Storm variables to be used to make tag names in a edit block. (#1300)
  • Allow Storm variables with list values to be used to set multiple tags in a edit block, e.g. $foo=(tag1,tag2,tag3) [test:str=x +#$foo]. (#1300)
  • Allow quoted strings as variable names and fields. (#1298)

Bugfixes

  • Fix runtime safety scoping issue for variables in Storm. (#1296)

v0.1.14 - 2019-06-21

Features and Enhancements

  • Add sub-command aliases for the Cmdr hive and cron commands, so that similar subcommands like list and ls work across both commands. (#1281)
  • Simplify adding structured data to the cell Hive via Cmdr. (#1282)

Bugfixes

  • Fix an issue in Cmdr for hive get which could result in failing to properly overwrite files when saving a Hive value to disk. (#1282)

Improved Documentation

  • Add additional logging for ReadTheDocs documentation builds. (#1284)
  • Add additional Hive API docstrings. (#1285)

v0.1.13 - 2019-06-18

Features and Enhancements

  • Add syn:trigger runtime only nodes to the Cortex. These represent triggers which have been configured on a Cortex. (#1270)
  • Add a new packed node helper, synapse.lib.nodes.tagsnice(), to get all the leaf tags on a node and any tags which have a time interval associated with them. (#1271)
  • Add a err? column to the output of the cron list. This includes an X character in the column if the last execution of that Cron task encountered an error. (#1272)
  • Refactor the Boss commands in cmdr to their own file and improve test coverage for the Cortex storm command in Cmdr. (#1273)
  • Add $node.globtags() method to Storm which accepts a tag glob, and returns a list of the matching glob values. (#1275)
  • Add there remote Cortex API CoreApi.delNodeProp() to allow property deletion from a single node. (#1279)

Bugfixes

  • Update CellApi Hive functions to properly check permissions. (#1274)
  • Ensure that tearing down a Telepath generator via GeneratorExit from non-async code properly signals the generator to teardown on the ioloop. (#1278)
  • Fix an issue where Storm subquery variable assignments were being pushed to the global runtime, but were not properly available to the Path objects associated with inbound nodes. (#1280)

Improved Documentation

  • Improve inline API help for a few test helper functions. (#1273)
  • Update Cmdr reference documentation for trigger and cron updates. (#1277)

v0.1.12 - 2019-06-12

Features and Enhancements

  • Centralize the allowed() and _reqUserAllowed() function from the CoreApi class to the CellApi, making permission checking easier for CellApi implementers. (#1268)
  • Add the $path built-in Storm variable to the default variables populated in the Storm pipeline. (#1269)
  • Add a $path.trace() method to get a object which traces the pivots from a given Path object. The path idens can be obtained via trace.iden(). (#1269)
  • Add $lib.set() to Storm Types. This can be used to get a mutable set object. (#1269)

Bugfixes

  • Fix an issue where the Base link() API required the linking function to be a coroutine. (#1261)

Improved Documentation

  • Improve inline API help for a few functions. (#1268)

v0.1.11 - 2019-06-06

Features and Enhancements

  • Add an optional facility to lmdbslab to prevent its data from being swapped out of memory. Add a Cortex configuration option (in the cell.yaml file) named dedicated to enable this for the lmdb slabs that store the graph data in a Cortex. This is currently only supported on Linux. (#1254)

Bugfixes

  • Fix an issue where the Cmdr color awareness for error highlighting was preventing documentation from building properly. (#1261)
  • Fix an issue where the synapse.servers.cortex --mirror option was not properly mirroring realtime splices. (#1264)
  • Fix a runtsafe variable order bug in Storm. (#1265)
  • Work around an issue in prompt-toolkit’s print_formatted_text function. (#1266)
  • Fix an issue where color awareness was not available for Cmdr sessions launched via synapse.tools.csvtool and synapse.tools.feed. (#1267)

Improved Documentation

  • Update Storm lift documentation to include lifting by time intervals. (#1260)
  • Update ReadTheDocs build configuration to utilize a Docker container, instead of a conda environment. (#1262)

v0.1.10 - 2019-06-04

Features and Enhancements

  • Add $node.iden() method in Storm to expose the iden of a node. (#1257)
  • Add $lib.text() method in Storm Lib to add a mutable string formatting object. (#1258)

v0.1.9 - 2019-05-31

Features and Enhancements

  • Add colored error reporting in Cmdr when a BadSyntax exception is sent to the user. (#1248)
  • Expose the local Synapse version information in Cmdr via the locs command. (#1250)
  • Add reflected class names to the Telepath shareinfo. Expose this with the Proxy._getClasses() API. (#1250)
  • Add --file and --optsfile arguments to the Cmdr storm command. These, respectively, allow a user to provide a file containing a raw Storm query and variable arguments as a json file. (#1252)

Bugfixes

  • Fix an issue where the Cmdr log command did not clean up all of its settings. (#1249)
  • Fix an issue with the Storm switch statement handling of non-runtsafe values. (#1251)
  • Fix an issue with the Storm if statement handling of non-runtsafe values. (#1253)
  • Fix an issue with when connecting to a Cortex via Telepath for the default remote layer, which previously could have pointed to a layer which was not the correct layer for the default view. (#1255)

v0.1.8 - 2019-05-22

Features and Enhancements

  • Add if/elif/else statement. Add and/or/not inside dollar expressions. Have expressions always return an int. (#1235)
  • Add variable and expression filters. Test for and correct all known grammar ambiguities. Tag filters with a comparison, e.g. +#$foo=$bar, now don’t raise an exception (#1241)
  • Add ability to enable and disable cron jobs and triggers. (#1242)

Bugfixes

  • Fix a bug where a tag addition could cause a splice to be generated if the tag window being added was inside of the existing tag window. (#1243)
  • csvtool now correctly handles print events (#1245)

Improved Documentation

  • Update release process documentation. (#1244)

v0.1.7 - 2019-05-17

Features and Enhancements

  • Add the Synapse version information in the Telepath handshake. Expose this with the Proxy._getSynVers() API and in the Cmdr CLI via the locs command. (#1238)
  • Add a --save-nodes argument to the Storm command in Cmdr to do a one-shot record of nodes returned by a Storm query. (#1239)
  • Allow synapse.tools.cmdr to take a second argument and run that argument as a Cmdr command. (#1239)
  • Add $node.repr() to Storm types. This allows the user to get the repr of the primary property, or a secondary property, and assign it to a variable in storm. (#1222)
  • Add lib.csv.emit() to Storm types. This allows the user to emit a message during a Storm query which can easily be joined into a CSV. (#1236)
  • Add a --export option to synapse.tools.csvtool. This allows the user to create a CSV file from a query that uses the $lib.csv.emit() Storm function. (#1236)

Bugfixes

  • Resolve Storm grammar ambiguity between tag condition filters with value and left join. (#1237)
  • Resolve Storm grammar ambiguity to prevent reserved words from being identified as a Storm command. (#1240)

v0.1.6 - 2019-05-15

Bugfixes

  • Fix an ambuguity in the Storm grammer regarding quoted command arguments. (#1234)

v0.1.5 - 2019-05-15

Features and Enhancements

  • Make Ndef, Edge and TimeEdge repr implementations consistent. (#1217)
  • Add jsonl support the synapse.tools.feed tool. (#1220)
  • Add /api/v1/model API route for the Cortex HTTP API to expose the data model for a running Cortex. (#1221)
  • Add fire() function to Storm types to fire storm:fire messages during Storm command execution. (#1221)
  • Add $() expression syntax to Storm for mathematical operations, along with a new parsing engine built around Lark. (#1216)
  • Add a warning when Synapse is imported if the user is running Python with -OO optimizations, since that can degrade the library capabilities. (#1219)
  • Cleanup some exception chains so that type normalization errors do not result in large tracebacks on the server. (#1224)
  • Allow $lib.print() to accept curly brace {} formatted strings for using variable substitution when printing values in Storm. (#1227)

Bugfixes

  • Fix an issue in Storm with lifting or filtering nodes by tags when the tag value is a variable. (#1223)
  • Fix an issue which was preventing a tag variable value reference in Storm from behaving correctly. (#1228)
  • Fix a missing await statement which prevented properly setting layers for a Cortex View object. (#1231)

Improved Documentation

  • Fix some docstrings related to test code helpers. (#1230)

v0.1.4 - 2019-05-01

Features and Enhancements

  • Add POST support to the /api/v1/model/norm HTTP API endpoint. (#1207)
  • Add getPropNorm() and getTypeNorm() Telepath API endpoints to the Cortex and CoreApi. (#1207)
  • Add list length() and index() methods to Storm types. (#1208)
  • Add helper functions to synapse.lib.node for extracting repr values from packed nodes. (#1212)
  • Add --nodes-only to the Cmdr log command to only record raw nodes. (#1213)
  • Add guid(), min(), max() functions to Storm types. (#1215)
  • Add getStormEval() to the synapse.lib.storm.Cmd class. This helper can be used by Storm command implementers in resolving variables, full property, and relative property values off of the Storm runtime. (#1215)
  • The Storm min and max commands may now accept a relative property path, a full property path, or a variable. (#1215)
  • Add a --mirror to synapse.servers.cortex to allow easier mirroring of a backup Cortex from its source Cortex. (#1197)

Bugfixes

  • Fix an error in PropPivotOut and FormPivot where a None object could be yielded in the Storm pipeline. (#1210)
  • Shut down HTTP API servers on Cell fini(). (#1211)

Improved Documentation

  • Convert developer guide from static RST to Jupyter Notebook. (#1209)
  • Convert HTTP API guide from static RST to Jupyter Notebook. (#1211)
  • Add a note about backing up and restoring a cortex to the quickstart guide. (#1214)

v0.1.3 - 2019-04-17

Features and Enhancements

  • Add the ability to delete a role via HTTP API, as well as being able to mark a user as being archived. Archiving a user will also lock a user. (#1205)
  • Add support to archiving for user to the CellApi for use via Telepath. (#1206)

Bugfixes

  • Fix remote layer bug injected by previous optimization that would result in missing nodes from lifts when the node only resides in the distant layer. (#1203)

Improved Documentation

  • Fix error in the HTTP API documentation. (#1204)

v0.1.2 - 2019-04-10

Features and Enhancements

  • Automatically run unit tests for the master every day. (#1192)
  • Add test suite for synapse.lib.urlhelp. (#1195)
  • Improve multi-layer and single layer performance. This is a backwards-incompatible API change in that 0.1.2 cortex will not interoperate with 0.1.2 remote layers before version 0.1.2. Persistent storage format has not changed. (#1196)
  • Add skeleton for reverse engineering model. (#1198)

Bugfixes

  • When using synapse.tools.cmdr, issuing ctrl-c to cancel a running command in could result in the Telepath Proxy object being fini’d. This has been resolved by adding a signal handler to the synapse.lib.cli.Cli class which is registered by cmdr. (#1199)
  • Fix an issue where deleting a property which has no index failed. (#1200)
  • Single letter form and property names were improperly disallowed. They are now allowed. (#1201)

Improved Documentation

  • Add some example developer guide documentation. (#1193)

v0.1.1 - 2019-04-03

Features and Enhancements

  • Allow synapse.servers tools to specify a custom Telepath share name. (#1170)
  • Add $lib.print(), $lib.len(), $lib.min(), $lib.max(), and $lib.dict() Storm library functions. (#1179)
  • Add $lib.str.concat() and $lib.str.format() Storm library functions. (#1179)
  • Initial economic model for tracking purchases. (#1177)
  • Add progress logging for the (0, 1, 0) layer migration. (#1180)
  • Remove references to Cortex.layer as a Cortex level attribute. There was no guarantee that this was the correct write layer for a arbitrary view and could lead to incorrect usage. (#1181)
  • Optimize the snap.getNodesBy() API to shortcut true equality lift operations to become pure lifts by buid. (#1183)
  • Add a generic Cell server, synapse.servers.cell that can be used to launch any Cell by python class path and file path. This can be used to launch custom Cell objects. (#1182)
  • Add server side remote event processing to .storm() API calls. (#1171)
  • Add Telepath user proxying. (#1171)
  • Migrate Dockerhub docker container builds and pypi packaging and release processes to CircleCI. (#1185)
  • Improve performance. Add a small layer-level cache. Replace home-grown synapse.lib.cache.memoize implementation with standard one. Make layer microoptimizations. (#1191)

Bugfixes

  • Fixes for lmdblab.dropdb and lmdbslab.initdb mapfull safety. (#1174)
  • Graceful recovery for pre v0.1.0 database migrations for lmdbslab backed databases. (#1175)
  • Syntax parser did not allow for multiple dot hierarchies in universal properties. (#1178)
  • Fix for lmdbslab mapfull error during shutdown (#1184)
  • synapse.lib.reflect.getShareInfo() could return incorrect data depending on execution order and object type inheritance. (#1186)
  • Add missing test for Str types extracting named regular expression matches as subs. (#1187)

Improved Documentation

  • Minor documentation updates for permissions. (#1172)
  • Added docstring and test for synapse.lib.coro.executor(). (#1189)

v0.1.0 - 2019-03-19

  • Synapse version 0.1.0 released.