synapse.lib package
Subpackages
- synapse.lib.crypto package
- synapse.lib.platforms package
- synapse.lib.stormlib package
- Submodules
- synapse.lib.stormlib.aha module
- synapse.lib.stormlib.auth module
- synapse.lib.stormlib.backup module
- synapse.lib.stormlib.basex module
- synapse.lib.stormlib.cache module
- synapse.lib.stormlib.cell module
- synapse.lib.stormlib.compression module
- synapse.lib.stormlib.cortex module
- synapse.lib.stormlib.easyperm module
- synapse.lib.stormlib.env module
- synapse.lib.stormlib.ethereum module
- synapse.lib.stormlib.gen module
- synapse.lib.stormlib.gis module
- synapse.lib.stormlib.graph module
- synapse.lib.stormlib.hashes module
- synapse.lib.stormlib.hex module
- synapse.lib.stormlib.imap module
- synapse.lib.stormlib.index module
- synapse.lib.stormlib.infosec module
- synapse.lib.stormlib.ipv6 module
- synapse.lib.stormlib.iters module
- synapse.lib.stormlib.json module
- synapse.lib.stormlib.log module
- synapse.lib.stormlib.macro module
- synapse.lib.stormlib.math module
- synapse.lib.stormlib.mime module
- synapse.lib.stormlib.model module
- synapse.lib.stormlib.modelext module
LibModelExtLibModelExt.addEdge()LibModelExt.addExtModel()LibModelExt.addForm()LibModelExt.addFormProp()LibModelExt.addTagProp()LibModelExt.addType()LibModelExt.addUnivProp()LibModelExt.delEdge()LibModelExt.delForm()LibModelExt.delFormProp()LibModelExt.delTagProp()LibModelExt.delType()LibModelExt.delUnivProp()LibModelExt.getExtModel()LibModelExt.getObjLocals()
- synapse.lib.stormlib.notifications module
- synapse.lib.stormlib.oauth module
- synapse.lib.stormlib.pack module
- synapse.lib.stormlib.project module
- synapse.lib.stormlib.random module
- synapse.lib.stormlib.scrape module
- synapse.lib.stormlib.smtp module
- synapse.lib.stormlib.spooled module
- synapse.lib.stormlib.stats module
- synapse.lib.stormlib.stix module
- synapse.lib.stormlib.storm module
- synapse.lib.stormlib.tabular module
- synapse.lib.stormlib.utils module
- synapse.lib.stormlib.vault module
- synapse.lib.stormlib.version module
- synapse.lib.stormlib.xml module
- synapse.lib.stormlib.yaml module
Submodules
synapse.lib.agenda module
- class synapse.lib.agenda.Agenda[source]
Bases:
BaseOrganize and execute all the scheduled storm queries in a cortex.
- async add(cdef)[source]
Persistently adds an appointment
- Parameters:
cdef (dict) – Dictionary containing the Cron definition.
Notes
The cron definition may contain the following keys:
- creator (str)
Iden of the creating user.
- iden (str)
Iden of the appointment.
- storm (str)
The Storm query to run.
- reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…])
One or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.
- incunit (Union[None, TimeUnit])
The unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.
- incvals (Union[None, int, Iterable[int])
Count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)
If the values for req and incvals are both lists, all combinations of all values (the product) are used.
- Returns:
Packed appointment definition
- class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)[source]
Bases:
objectRepresents a single element of a single combination of an appointment
synapse.lib.aha module
- class synapse.lib.aha.AhaApi[source]
Bases:
CellApi- async addAhaSvc(name, info, network=None)[source]
Register a service with the AHA discovery server.
- NOTE: In order for the service to remain marked “up” a caller
must maintain the telepath link.
- addAhaSvcProv(name, provinfo=None)[source]
Provision the given relative service name within the configured network name.
- addAhaUserEnroll(name, userinfo=None, again=False)[source]
Create and return a one-time user enroll key.
- async getAhaSvc(name, filters=None)[source]
Return an AHA service description dictionary for a service name.
- async getAhaSvcMirrors(name)[source]
Return list of AHA svcinfo dictionaries for mirrors of a service.
- class synapse.lib.aha.AhaCell[source]
Bases:
Cell- confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'auth:passwd:policy': {'description': 'Specify password policy/complexity requirements.', 'type': 'object'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'health:sysctl:checks': {'default': True, 'description': 'Enable sysctl parameter checks and warn if values are not optimal.', 'type': 'boolean'}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:parse:proxy:remoteip': {'default': False, 'description': 'Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.', 'type': 'boolean'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'max:users': {'default': 0, 'description': 'Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).', 'minimum': 0, 'type': 'integer'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': False, 'hidedocs': False, 'type': ['string', 'null']}, 'nexslog:async': {'default': True, 'description': 'Deprecated. This option ignored.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.', 'type': 'boolean'}}
- confdefs = {'aha:urls': {'description': 'Deprecated. AHA servers can now manage this automatically.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'clone': {'description': 'Bootstrap a clone from the AHA clone URL.', 'hidecmdl': True, 'type': ['string', 'null']}, 'dns:name': {'description': 'The registered DNS name used to reach the AHA service.', 'type': ['string', 'null']}, 'provision:listen': {'description': 'A telepath URL for the AHA provisioning listener.', 'type': ['string', 'null']}}
- class synapse.lib.aha.AhaProvisionServiceV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.aha.AhaServicesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
synapse.lib.ast module
- class synapse.lib.ast.AndCond(astinfo, kids=())[source]
Bases:
Cond<cond> and <cond>
- class synapse.lib.ast.AstNode(astinfo, kids=())[source]
Bases:
objectBase class for all nodes in the Storm abstract syntax tree.
- runtopaque = False
- class synapse.lib.ast.Cond(astinfo, kids=())[source]
Bases:
ValueA condition that is evaluated to filter nodes.
- class synapse.lib.ast.DollarExpr(astinfo, kids=())[source]
Bases:
ValueTop level node for $(…) expressions
- class synapse.lib.ast.EditNodeAdd(astinfo, kids=())[source]
Bases:
Edit
- class synapse.lib.ast.EmptyBlock(astinfo, kids=())[source]
Bases:
AstNodeAn AST node that only runs if there are not inbound nodes in the pipeline. It is capable of yielding nodes into the pipeline.
Example
Using an empty block:
empty { // the pipeline is empty so this block will execute } [foo:bar=*] empty { // there is a node in the pipeline so this block will not run }
- class synapse.lib.ast.ExprNode(astinfo, kids=())[source]
Bases:
ValueA binary (i.e. two argument) expression node
- class synapse.lib.ast.FiniBlock(astinfo, kids=())[source]
Bases:
AstNodeAn AST node that runs only once after all nodes have been consumed.
Example
Using a fini block:
fini { // stuff here runs *once* after the last node yield (even if there are no nodes) }
Notes
A fini block must be runtsafe.
- class synapse.lib.ast.FormPivot(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper-> foo:bar
- class synapse.lib.ast.FuncArgs(astinfo, kids=())[source]
Bases:
AstNodeRepresents the function arguments in a function definition
- class synapse.lib.ast.Function(astinfo, kids=())[source]
Bases:
AstNode( name, args, body )
// use args/kwargs syntax function bar(x, v=$(30)) { }
# we auto-detect the behavior of the target function
# return a value function bar(x, y) { return ($(x + y)) }
# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }
$foo = $bar(10, v=20)
- async callfunc(runt, argdefs, args, kwargs, funcpath)[source]
Execute a function call using the given runtime.
This function may return a value / generator / async generator
- runtopaque = True
- class synapse.lib.ast.HasRelPropCond(astinfo, kids=())[source]
Bases:
Cond
- class synapse.lib.ast.InitBlock(astinfo, kids=())[source]
Bases:
AstNodeAn AST node that runs only once before yielding nodes.
Example
Using a init block:
init { // stuff here runs *once* before the first node yield (even if there are no nodes) }
- class synapse.lib.ast.LiftByArray(astinfo, kids=())[source]
Bases:
LiftOper:prop*[range=(200, 400)]
- class synapse.lib.ast.LiftFormTagProp(astinfo, kids=())[source]
Bases:
LiftOperhehe:haha#foo.bar:baz [ = x ]
- class synapse.lib.ast.Lookup(astinfo, kids, autoadd=False)[source]
Bases:
QueryWhen storm input mode is “lookup”
- class synapse.lib.ast.PivotInFrom(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper<- foo:edge
- class synapse.lib.ast.PivotToTags(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present
- class synapse.lib.ast.PropPivot(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper:foo -> bar:foo
- class synapse.lib.ast.PropPivotOut(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper:prop -> *
- class synapse.lib.ast.RawPivot(astinfo, kids=(), isjoin=False)[source]
Bases:
PivotOper-> { <varsfrompath> }
- class synapse.lib.ast.RelPropCond(astinfo, kids=())[source]
Bases:
Cond(:foo:bar or .univ) <cmpr> <value>
- class synapse.lib.ast.SetItemOper(astinfo, kids=())[source]
Bases:
Oper$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz
- class synapse.lib.ast.SubGraph(rules)[source]
Bases:
objectAn Oper like object which generates a subgraph.
Notes
The rules format for the subgraph is shaped like the following:
rules = { 'degrees': 1, 'edges': True, 'edgelimit': 3000, 'filterinput': True, 'yieldfiltered': False, 'filters': [ '-(#foo or #bar)', '-(foo:bar or baz:faz)', ], 'pivots': [ '-> * | limit 100', '<- * | limit 100', ] 'forms': { 'inet:fqdn':{ 'filters': [], 'pivots': [], } '*': { 'filters': [], 'pivots': [], }, }, }
Nodes which were original seeds have path.meta(‘graph:seed’).
All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.
- class synapse.lib.ast.SubQuery(astinfo, kids=())[source]
Bases:
Oper
- class synapse.lib.ast.TagCond(astinfo, kids=())[source]
Bases:
Cond#foo.bar
- class synapse.lib.ast.TagMatch(astinfo, kids=())[source]
Bases:
TagNameLike TagName, but can have asterisks
- class synapse.lib.ast.UnaryExprNode(astinfo, kids=())[source]
Bases:
ValueA unary (i.e. single-argument) expression node
- class synapse.lib.ast.Value(astinfo, kids=())[source]
Bases:
AstNodeThe base class for all values and value expressions.
- class synapse.lib.ast.VarEvalOper(astinfo, kids=())[source]
Bases:
OperFacilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)
synapse.lib.auth module
- class synapse.lib.auth.Auth[source]
Bases:
PusherAuth is a user authentication and authorization stored in a Slab. Users correspond to separate logins with different passwords and potentially different privileges.
Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).
Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.
AuthGates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.
Layout:
Auth root (passed into constructor) ├ roles │ ├ <role iden 1> │ ├ ... │ └ last role ├ users │ ├ <user iden 1> │ ├ ... │ └ last user └ authgates ├ <iden 1> │ ├ roles │ │ ├ <role iden 1> │ │ ├ ... │ │ └ last role │ └ users │ ├ <user iden 1> │ ├ ... │ └ last user ├ <iden 2> │ ├ ... └ ... last authgate- async addAuthGate(iden, authgatetype)[source]
Retrieve AuthGate by iden. Create if not present.
Note
Not change distributed
- Returns:
(AuthGate)
- async addRole(name, iden=None)[source]
Add a Role to the Auth system.
- Parameters:
name (str) – The name of the role.
iden (str) – A optional iden to use as the role iden.
- Returns:
A Role.
- Return type:
- async addUser(name, passwd=None, email=None, iden=None)[source]
Add a User to the Auth system.
- Parameters:
name (str) – The name of the User.
passwd (str) – A optional password for the user.
email (str) – A optional email for the user.
iden (str) – A optional iden to use as the user iden.
- Returns:
A User.
- Return type:
- checkUserLimit()[source]
Check if we’re at the specified user limit.
This should be called right before adding/unlocking/unarchiving a user.
Raises: s_exc.HitLimit if the number of active users is at the maximum.
- class synapse.lib.auth.AuthGate(info, auth)[source]
Bases:
objectThe storage object for object specific rules for users/roles.
- class synapse.lib.auth.Role(info, auth)[source]
Bases:
RulerA role within the authorization subsystem.
A role in Auth exists to bundle rules together so that the same set of rules can be applied to multiple users.
- class synapse.lib.auth.Ruler(info, auth)[source]
Bases:
objectAn object that holds a list of rules. This includes Users, Roles, and the AuthGate variants of those
- class synapse.lib.auth.User(info, auth)[source]
Bases:
RulerA user (could be human or computer) of the system within Auth.
Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.
- allowed(perm: tuple[str, ...], default: str | None = None, gateiden: str | None = None, deepdeny: bool = False) bool | None[source]
Check if a user is allowed a given permission.
- Parameters:
perm – The permission tuple to check.
default – The default rule value if there is no match.
gateiden – The gate iden to check against.
deepdeny – If True, give precedence for checking deny rules which are more specific than the requested permission.
Notes
The use of the deepdeny argument is intended for checking a less-specific part of a permissions tree, in order to know about possible short circuit options. Using it to check a more specific part may have unintended results.
- Returns:
The allowed value of the permission.
- getAllowedReason(perm, default=None, gateiden=None)[source]
A routine which will return a tuple of (allowed, info).
- async setRoles(roleidens)[source]
Replace all the roles for a given user with a new list of roles.
- Parameters:
roleidens (list) – A list of roleidens.
Notes
The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.
- Returns:
None
synapse.lib.autodoc module
- synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)[source]
Process a list of StormTypes doc information to add them to an RstHelp object.
- Notes
This will create internal hyperlink link targets for each header item. The link prefix string must be given with the
linkprefixargument.
- Parameters:
page (RstHelp) – The RST page to add .
docinfo (dict) – A Stormtypes Doc.
linkprefix (str) – The RST link prefix string to use.
islib (bool) – Treat the data as a library. This will preface the header and attribute values with
$and use full paths for attributes.lvl (int) – The base header level to use when adding headers to the page.
- Returns:
None
- synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None, isstor=False)[source]
- synapse.lib.autodoc.ljuster(ilines)[source]
Helper to lstrip lines of whitespace an appropriate amount.
- synapse.lib.autodoc.runtimeDocStormTypes(page, docinfo, islib=False, lvl=1, oneline=False, addheader=True, preamble=None)[source]
Process a list of StormTypes doc information to add them to a RstHelp object.
Used for Storm runtime help generation.
- Parameters:
page (RstHelp) – The RST page to add .
docinfo (dict) – A Stormtypes Doc.
linkprefix (str) – The RST link prefix string to use.
islib (bool) – Treat the data as a library. This will preface the header and attribute values with
$and use full paths for attributes.lvl (int) – The base header level to use when adding headers to the page.
oneline (bool) – Only display the first line of description. Omits local headers.
preamble (list) – Lines added after the header; and before locls.
- Returns:
None
synapse.lib.base module
- class synapse.lib.base.Base[source]
Bases:
objectBase class for Synapse objects.
Acts as an observable, enables async init and fini.
Example
class Foo(Base):
async def __anit__(self, x, y):
await Base.__anit__(self)
await stuff(x, y)
foo = await Foo.anit(10)
Note
One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.
- async addSignalHandlers()[source]
Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.
- async dist(mesg)[source]
Distribute an existing event tuple.
- Parameters:
mesg ((str,dict)) – An event tuple.
Example
await base.dist( (‘foo’,{‘bar’:’baz’}) )
- async enter_context(item)[source]
Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.
- Returns:
The result of item’s own __aenter__ or __enter__() method.
- async fini()[source]
Shut down the object and notify any onfini() coroutines.
- Returns:
Remaining ref count
- async fire(evtname, **info)[source]
Fire the given event name on the Base. Returns a list of the return values of each callback.
Example
- for ret in d.fire(‘woot’,foo=’asdf’):
print(‘got: %r’ % (ret,))
- incref()[source]
Increment the reference count for this base. This API may be optionally used to control fini().
- link(func)[source]
Add a callback function to receive all events.
Example
base1 = Base() base2 = Base()
base1.link( base2.dist )
# all events on base1 are also propagated on base2
- async main()[source]
Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )
Note
This API may only be used when the ioloop is also the main thread.
- off(evnt, func)[source]
Remove a previously registered event handler function.
Example
base.off( ‘foo’, onFooFunc )
- on(evnt, func, base=None)[source]
Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.
- Parameters:
evnt (str) – An event name
func (function) – A callback function to receive event tufo
Examples
Add a callback function and fire it:
- async def baz(event):
x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y
d.on(‘foo’, baz)
# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)
- Return type:
None
- onWith(evnt, func)[source]
A context manager which can be used to add a callback and remove it when using a
withstatement.- Parameters:
evnt (str) – An event name
func (function) – A callback function to receive event tufo
- onfini(func)[source]
Add a function/coroutine/Base to be called on fini().
- The rules around how to register function/coroutine/Base to be called:
Call this method with an instance of Base (this class) if holding a reference to a bound method of the instance (such as a fini() method) would cause the object to be leaked. This is appropriate for ephemeral objects that may be constructed/destroyed multiple times over the lifetime of a process.
Call this method with an instance method if you want the object to have a lifetime as long as the thing being fini’d.
- async postAnit()[source]
Method called after self.__anit__() has completed, but before anit() returns the object to the caller.
- schedCallSafe(func, *args, **kwargs)[source]
Schedule a function to run as soon as possible on the same event loop that this Base is running on.
This function does not pend on the function completion.
- Parameters:
func
*args
**kwargs
Notes
This method may be called from outside of the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.
- Returns:
A Future representing the eventual function execution.
- Return type:
concurrent.futures.Future
- schedCoro(coro)[source]
Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.
- Parameters:
coro – The coroutine to schedule.
Notes
This function is not threadsafe and must be run on the Base’s event loop. Tasks created by this function do inherit the synapse.lib.scope Scope from the current task.
- Returns:
An asyncio.Task object.
- Return type:
asyncio.Task
- schedCoroSafe(coro)[source]
Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.
This function does not pend on coroutine completion.
Notes
This method may be run outside the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.
- Returns:
A Future representing the eventual coroutine execution.
- Return type:
concurrent.futures.Future
- schedCoroSafePend(coro)[source]
Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on
Note
This method may not be run inside an event loop
- unlink(func)[source]
Remove a callback function previously added with link()
Example
base.unlink( callback )
- waiter(count, *names, timeout=None)[source]
Construct and return a new Waiter for events on this base.
Example
# wait up to 3 seconds for 10 foo:bar events…
waiter = base.waiter(10,’foo:bar’)
# .. fire task that will cause foo:bar events
events = await waiter.wait(timeout=3)
- if events == None:
# handle the timeout case…
- for event in events:
# parse the events if you need…
Note
Use this with caution. It’s easy to accidentally construct race conditions with this mechanism ;)
- class synapse.lib.base.BaseRef[source]
Bases:
BaseAn object for managing multiple Base instances by name.
- async gen(name)[source]
Atomically get/gen a Base and incref. (requires ctor during BaseRef init)
- Parameters:
name (str) – The name/iden of the Base instance.
- get(name)[source]
Retrieve a Base instance by name.
- Parameters:
name (str) – The name/iden of the Base
- Returns:
The Base instance (or None)
- Return type:
(Base)
- pop(name)[source]
Remove and return a Base from the BaseRef.
- Parameters:
name (str) – The name/iden of the Base instance
- Returns:
The named base ( or None )
- Return type:
(Base)
synapse.lib.boss module
- class synapse.lib.boss.Boss[source]
Bases:
BaseAn object to track “promoted” async tasks.
Promoted tasks are asyncio tasks, wrapped in a synapse task (
s_task.Task), that are visible to storm users via the task tracking libs/commands such asps.listand$lib.ps.list().- async execute(coro, name, user, info=None, iden=None)[source]
Create a synapse task from the given coroutine.
- async promote(name, user, info=None, taskiden=None)[source]
Promote the currently running task.
- Parameters:
name (str) – The name of the task.
user – The User who owns the task.
taskiden – An optional GUID for the task.
info – An optional information dictionary containing information about the task.
- Returns:
The Synapse Task object.
- Return type:
s_task.Task
synapse.lib.cache module
A few speed optimized (lockless) cache helpers. Use carefully.
- class synapse.lib.cache.LruDict(size=10000)[source]
Bases:
MutableMappingMaintains the last n accessed keys
- class synapse.lib.cache.TagGlobs[source]
Bases:
objectAn object that manages multiple tag globs and values for caching.
- synapse.lib.cache.memoizemethod(size=16384)[source]
A version of memoize that doesn’t cause GC cycles when applied to a method.
- synapse.lib.cache.regexizeTagGlob(tag)[source]
- Returns:
a regular expression string with ** and * interpreted as tag globs
- Precondition:
tag is a valid tagmatch
Notes
A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.
The returned string does not contain a starting ‘^’ or trailing ‘$’.
synapse.lib.cell module
- class synapse.lib.cell.Cell[source]
-
A Cell() implements a synapse micro-service.
- A Cell has 5 phases of startup:
Universal cell data structures
Service specific storage/data (pre-nexs)
Nexus subsystem initialization
Service specific startup (with nexus)
Networking and mirror services
- BACKUP_SPAWN_TIMEOUT = 60.0
- COMMIT = ''
- FREE_SPACE_CHECK_FREQ = 60.0
- LOGGED_HTTPAPI_HEADERS = ('User-Agent',)
- SYSCTL_CHECK_FREQ = 60.0
- SYSCTL_VALS = {'vm.dirty_expire_centisecs': 20, 'vm.dirty_writeback_centisecs': 20}
- VERSION = (2, 210, 0)
- VERSTRING = '2.210.0'
- addActiveCoro(func, iden=None, base=None)[source]
Add a function callback to be run as a coroutine when the Cell is active.
- Parameters:
func (coroutine function) – The function run as a coroutine.
iden (str) – The iden to use for the coroutine.
base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d
- Returns:
A GUID string that identifies the coroutine for delActiveCoro()
- Return type:
str
Note
This will re-fire the coroutine if it exits and the Cell is still active.
- async addDrivePath(path, perm=None, reldir='00000000000000000000000000000000')[source]
Create the given path using the specified permissions.
The specified permissions are only used when creating new directories.
- NOTE: We must do this outside the Drive class to allow us to generate
iden and tick but remain nexus compatible.
- async addHttpsPort(port, host='0.0.0.0', sslctx=None)[source]
Add a HTTPS listener to the Cell.
- Parameters:
port (int) – The listening port to bind.
host (str) – The listening host.
sslctx (ssl.SSLContext) – An externally managed SSL Context.
Note
If the SSL context is not provided, the Cell will assume it manages the SSL context it creates for a given listener and will add a reload handler named https:certs to enabled reloading the SSL certificates from disk.
- addReloadableSystem(name: str, func: callable)[source]
Add a reloadable system. This may be dynamically called at at time.
- Parameters:
name (str) – Name of the system.
func – The callable for reloading a given system.
Note
Reload functions take no arguments when they are executed. Values returned by the reload function must be msgpack friendly.
- Returns:
None
- async addSignalHandlers()[source]
Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.
- async addUserApiKey(useriden, name, duration=None)[source]
Add an API key for a user.
Notes
The secret API key is only available once.
- Parameters:
useriden (str) – User iden value.
name (str) – Name of the API key.
duration (int or None) – Duration of time for the API key to be valid ( in milliseconds ).
- Returns:
A tuple of the secret API key value and the API key metadata information.
- Return type:
tuple
- async callPeerApi(todo, timeout=None)[source]
Yield responses from our peers via the AHA gather call API.
- async callPeerGenr(todo, timeout=None)[source]
Yield responses from invoking a generator via the AHA gather API.
- async checkUserApiKey(key)[source]
Check if a user API key is valid.
Notes
If the key is not valid, the dictionary will contain a
mesgkey. If the key is valid, the dictionary will contain the user def in audefkey, and the key metadata in akdefkey.- Parameters:
key (str) – The API key to check.
- Returns:
Tuple of two items, a boolean if the key is valid and a dictionary.
- Return type:
tuple
- confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'auth:passwd:policy': {'description': 'Specify password policy/complexity requirements.', 'type': 'object'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'health:sysctl:checks': {'default': True, 'description': 'Enable sysctl parameter checks and warn if values are not optimal.', 'type': 'boolean'}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:parse:proxy:remoteip': {'default': False, 'description': 'Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.', 'type': 'boolean'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'max:users': {'default': 0, 'description': 'Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).', 'minimum': 0, 'type': 'integer'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': True, 'hidedocs': True, 'type': ['string', 'null']}, 'nexslog:async': {'default': True, 'description': 'Deprecated. This option ignored.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.', 'type': 'boolean'}}
- confdefs = {}
- async delActiveCoro(iden)[source]
Remove an Active coroutine previously added with addActiveCoro().
- Parameters:
iden (str) – The iden returned by addActiveCoro()
- async delUserApiKey(iden)[source]
Delete an existing API key.
- Parameters:
iden (str) – The iden of the API key to delete.
- Returns:
True indicating the key was deleted.
- Return type:
bool
- async classmethod execmain(argv, outp=None)[source]
The main entry point for running the Cell as an application.
- Parameters:
argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.
Notes
This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.
- Returns:
None.
- async feedBeholder(name, info, gates=None, perms=None)[source]
Feed a named event onto the
cell:beholdermessage bus that will sent to any listeners.- Parameters:
info (dict) – An information dictionary to be sent to any consumers.
gates (list) – List of gate idens, whose details will be added to the outbound message(s).
perms (list) – List of permission names, whose details will be added to the outbound message(s).
- Returns:
None
- classmethod getArgParser(conf=None)[source]
Get an
argparse.ArgumentParserfor the Cell.- Parameters:
conf (s_config.Config) – Optional, a Config object which
Notes
Boot time configuration data is placed in the argument group called
config. This adds defaultdirn,--telepath,--httpsand--namearguements to the argparser instance. Configuration values which have thehideconforhidecmdlvalue set to True are not added to the argparser instance.- Returns:
A ArgumentParser for the Cell.
- Return type:
argparse.ArgumentParser
- async getCellApi(link, user, path)[source]
Get an instance of the telepath Client object for a given user, link and path.
- Parameters:
link (s_link.Link) – The link object.
user (s_auth.User) – The heavy user object.
path (str) – The path requested.
Notes
This defaults to the self.cellapi class. Implementors may override the default class attribute for cellapi to share a different interface.
- Returns:
The shared object for this cell.
- Return type:
object
- async getCellInfo()[source]
Return metadata specific for the Cell.
Notes
By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.
It is expected that implementers override the following Class attributes in order to provide meaningful version information:
COMMIT- A Git CommitVERSION- A Version tuple.VERSTRING- A Version string.- Returns:
A Dictionary of metadata.
- Return type:
- getDmonUser()[source]
Return the user IDEN of a telepath caller who invoked the current task. ( defaults to returning current root user )
- async getDriveData(iden, vers=None)[source]
Return the data associated with the drive item by iden. If vers is specified, return that specific version.
- async getDrivePath(path, reldir='00000000000000000000000000000000')[source]
Return a list of drive info elements for each step in path.
This may be used as a sort of “open” which returns all the path info entries. You may then operate directly on drive iden entries and/or check easyperm entries on them before you do…
- async getLogExtra(**kwargs)[source]
Get an extra dictionary for structured logging which can be used as a extra argument for loggers.
- Parameters:
**kwargs – Additional key/value items to add to the log.
- Returns:
A dictionary
- Return type:
- async getSystemInfo()[source]
Get info about the system in which the cell is running
- Returns:
volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
cellstarttime - Cell start time in epoch milliseconds
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system
cpucount - Number of CPUs on system
tmpdir - The temporary directory interpreted by the Python runtime.
- Return type:
A dictionary with the following keys. All size values are in bytes
- async getTeleApi(link, mesg, path)[source]
Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)
- async getUserApiKey(iden)[source]
Get a user API key via iden.
Notes
This contains the raw value. Callers are responsible for removing the
shadowkey.- Parameters:
iden (str) – The key iden.
- Returns:
The key dictionary; or none.
- Return type:
dict
- async handoff(turl, timeout=30)[source]
Hand off leadership to a mirror in a transactional fashion.
- classmethod initCellConf(conf=None)[source]
Create a Config object for the Cell.
- Parameters:
conf (s_config.Config) – An optional config structure. This has _opts_data taken from it.
Notes
The Config object has a
envar_prefixset according to the results ofcls.getEnvPrefix().- Returns:
A Config helper object.
- Return type:
s_config.Config
- async classmethod initFromArgv(argv, outp=None)[source]
Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.
- Parameters:
argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.
Notes
- This does the following items:
Create a Config object from the Cell class.
Creates an Argument Parser from the Cell class and Config object.
Parses the provided arguments.
Loads configuration data from the parsed options and environment variables.
Sets logging for the process.
Creates the Cell from the Cell Ctor.
Adds a Telepath listener, HTTPs port listeners and Telepath share names.
Returns the Cell.
- Returns:
This returns an instance of the Cell.
- Return type:
- async listUserApiKeys(useriden)[source]
Get all the API keys for a user.
- Parameters:
useriden (str) – The user iden.
- Returns:
A list of kdef values.
- Return type:
list
- modCellConf(conf)[source]
Modify the Cell’s ondisk configuration overrides file and runtime configuration.
- Parameters:
conf (dict) – A dictionary of items to set.
Notes
This does require the data being set to be schema valid.
- Returns:
None.
- async modUserApiKey(iden, key, valu)[source]
Update a value in the user API key metadata.
- Parameters:
iden (str) – Iden of the key to update.
key (str) – Name of the valu to update.
valu – The new value.
- Returns:
An updated key metadata dictionary.
- Return type:
dict
- popCellConf(name)[source]
Remove a key from the Cell’s ondisk configuration overrides file and runtime configuration.
- Parameters:
name (str) – Name of the value to remove.
Notes
This does not modify the cell.yaml file. This does re-validate the configuration after removing the value, so if the value removed had a default populated by schema, that default would be reset.
- Returns:
None
- async popHiveKey(path)[source]
Remove and return the value of a key in the cell default hive.
Note: this is for expert emergency use only.
- async promote(graceful=False)[source]
Transform this cell from a passive follower to an active cell that writes changes locally.
- class synapse.lib.cell.CellApi[source]
Bases:
Base- async allowed(perm, default=None)[source]
Check if the user has the requested permission.
- Parameters:
perm – permission path components to check
default – Value returned if no value stored
Examples
Form a path and check the permission from a remote proxy:
perm = ('node', 'add', 'inet:ipv4') allowed = await prox.allowed(perm) if allowed: dostuff()
- Returns:
True if the user has permission, False if explicitly denied, None if no entry
- Return type:
Optional[bool]
- cullNexsLog(offs)[source]
Remove Nexus log entries up to (and including) the given offset.
Note
If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.
Only rotated logs where the last index is less than the provided offset will be removed from disk.
- Parameters:
offs (int) – The offset to remove entries up to.
- Returns:
Whether the cull was executed
- Return type:
bool
- delBackup(name)[source]
Delete a backup by name.
- Parameters:
name (str) – The name of the backup to delete.
- getAuthUsers(archived=False)[source]
- Parameters:
archived (bool) – If true, list all users, else list non-archived users
- getBackupInfo()[source]
Get information about recent backup activity.
- Returns:
currduration - If backup currently running, time in ms since backup started, otherwise None
laststart - Last time (in epoch milliseconds) a backup started
lastend - Last time (in epoch milliseconds) a backup ended
lastduration - How long last backup took in ms
lastsize - Disk usage of last backup completed
lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive
lastexception - Tuple of exception information if last backup failed, otherwise None
- Return type:
(dict) It has the following keys
Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.
- getBackups()[source]
Retrieve a list of backups.
- Returns:
A list of backup names.
- Return type:
list[str]
- getGcInfo()[source]
For diagnostic purposes only!
NOTE: This API is not supported and can be removed at any time!
- getSystemInfo()[source]
Get info about the system in which the cell is running
- Returns:
volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system
- Return type:
A dictionary with the following keys. All size values are in bytes
- iterBackupArchive(name)[source]
Retrieve a backup by name as a compressed stream of bytes.
Note: Compression and streaming will occur from a separate process.
- Parameters:
name (str) – The name of the backup to retrieve.
- iterNewBackupArchive(name=None, remove=False)[source]
Run a new backup and return it as a compressed stream of bytes.
Note: Compression and streaming will occur from a separate process.
- Parameters:
name (str) – The name of the backup to retrieve.
remove (bool) – Delete the backup after streaming.
- rotateNexsLog()[source]
Rotate the Nexus log at the current offset.
- Returns:
The starting index of the active Nexus log
- Return type:
int
- runBackup(name=None, wait=True)[source]
Run a new backup.
- Parameters:
name (str) – The optional name of the backup.
wait (bool) – On True, wait for backup to complete before returning.
- Returns:
The name of the newly created backup.
- Return type:
str
- runGcCollect(generation=2)[source]
For diagnostic purposes only!
NOTE: This API is not supported and can be removed at any time!
- setCellUser(iden)[source]
Switch to another user (admin only).
This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.
- trimNexsLog(consumers=None, timeout=60)[source]
Rotate and cull the Nexus log (and those of any consumers) at the current offset.
Note
If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.
- Parameters:
consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.
timeout (int) – Time in seconds to wait for downstream consumers to be caught up.
- Returns:
The offset that the Nexus log was culled up to and including.
- Return type:
int
- synapse.lib.cell.SSLCTX_CACHE_SIZE = 64
Base classes for the synapse “cell” microservice architecture.
synapse.lib.certdir module
- class synapse.lib.certdir.CertDir(path: str | None = None)[source]
Bases:
objectCertificate loading/generation/signing utilities.
- Features:
Locates and load certificates, keys, and certificate signing requests (CSRs).
Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.
Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).
Signs certificate signing requests (CSRs).
Generates PKCS#12 archives for use in browser.
- Parameters:
path (str) – Optional path which can override the default path directory.
Notes
All certificates will be loaded from and written to ~/.syn/certs by default. Set the environment variable SYN_CERT_DIR to override.
All certificate generation methods create 4096 bit RSA keypairs.
All certificate signing methods use sha256 as the signature algorithm.
CertDir does not currently support signing CA CSRs.
- genCaCert(name: str, signas: str | None = None, outp: OutPut | None = None, save: bool = True) Tuple[RSAPrivateKey, Certificate][source]
Generates a CA keypair.
- Parameters:
name – The name of the CA keypair.
signas – The CA keypair to sign the new CA with.
outp – The output buffer.
save – Save the certificate and key to disk.
Examples
Make a CA named “myca”:
mycakey, mycacert = cdir.genCaCert('myca')
- Returns:
Tuple containing the private key and certificate objects.
- genCaCrl(name: str) Crl[source]
Get the CRL for a given CA.
- Parameters:
name – The CA name.
- Returns:
The CRL object.
- genClientCert(name: str, outp: OutPut | None = None) None[source]
Generates a user PKCS #12 archive.
Please note that the resulting file will contain private key material.
- Parameters:
name (str) – The name of the user keypair.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Make the PKC12 object for user “myuser”:
myuserpkcs12 = cdir.genClientCert('myuser')
- Returns:
None
- genCodeCert(name: str, signas: str | None = None, outp: OutPut | None = None, save: bool = True) Tuple[RSAPrivateKey, Certificate][source]
Generates a code signing keypair.
- Parameters:
name – The name of the code signing cert.
signas – The CA keypair to sign the new code keypair with.
outp – The output buffer.
Examples
Generate a code signing cert for the name “The Vertex Project”:
myuserkey, myusercert = cdir.genCodeCert('The Vertex Project')
- Returns:
Tuple containing the key and certificate objects.
- genHostCert(name: str, signas: str | None = None, outp: OutPut | None = None, csr: DHPublicKey | DSAPublicKey | RSAPublicKey | EllipticCurvePublicKey | Ed25519PublicKey | Ed448PublicKey | X25519PublicKey | X448PublicKey | None = None, sans: str | None = None, save: bool = True) Tuple[RSAPrivateKey | None, Certificate][source]
Generates a host keypair.
- Parameters:
name – The name of the host keypair.
signas – The CA keypair to sign the new host keypair with.
outp – The output buffer.
csr – The CSR public key when generating the keypair from a CSR.
sans – String of comma separated alternative names.
Examples
Make a host keypair named “myhost”:
myhostkey, myhostcert = cdir.genHostCert('myhost')
- Returns:
Tuple containing the private key and certificate objects. Private key may be None when signing a CSR.
- genHostCsr(name: str, outp: OutPut | None = None) bytes[source]
Generates a host certificate signing request.
- Parameters:
name – The name of the host CSR.
outp – The output buffer.
Examples
Generate a CSR for the host key named “myhost”:
cdir.genHostCsr('myhost')
- Returns:
The bytes of the CSR.
- genUserCert(name: str, signas: str | None = None, outp: OutPut | None = None, csr: DHPublicKey | DSAPublicKey | RSAPublicKey | EllipticCurvePublicKey | Ed25519PublicKey | Ed448PublicKey | X25519PublicKey | X448PublicKey | None = None, save: bool = True) Tuple[RSAPrivateKey | None, Certificate][source]
Generates a user keypair.
- Parameters:
name – The name of the user keypair.
signas – The CA keypair to sign the new user keypair with.
outp – The output buffer.
csr – The CSR public key when generating the keypair from a CSR.
Examples
Generate a user cert for the user “myuser”:
myuserkey, myusercert = cdir.genUserCert('myuser')
- Returns:
Tuple containing the key and certificate objects.
- genUserCsr(name: str, outp: OutPut | None = None) bytes[source]
Generates a user certificate signing request.
- Parameters:
name – The name of the user CSR.
outp – The output buffer.
Examples
Generate a CSR for the user “myuser”:
cdir.genUserCsr('myuser')
- Returns:
The bytes of the CSR.
- getCaCert(name: str) Certificate | None[source]
Loads the X509 object for a given CA.
- Parameters:
name – The name of the CA keypair.
Examples
Get the certificate for the CA “myca”:
mycacert = cdir.getCaCert('myca')
- Returns:
The certificate, if exists.
- getCaCertPath(name: str) str | None[source]
Gets the path to a CA certificate.
- Parameters:
name – The name of the CA keypair.
Examples
Get the path to the CA certificate for the CA “myca”:
mypath = cdir.getCACertPath('myca')
- Returns:
The path, if exists.
- getCaCerts() List[Certificate][source]
Return a list of CA certs from the CertDir.
- Returns:
List of CA certificates.
- getCaKey(name) RSAPrivateKey | DSAPrivateKey | None[source]
Loads the PKey object for a given CA keypair.
- Parameters:
name – The name of the CA keypair.
Examples
Get the private key for the CA “myca”:
mycakey = cdir.getCaKey('myca')
- Returns:
The private key, if exists.
- getCaKeyPath(name: str) str | None[source]
Gets the path to a CA key.
- Parameters:
name – The name of the CA keypair.
Examples
Get the path to the private key for the CA “myca”:
mypath = cdir.getCAKeyPath('myca')
- Returns:
The path, if exists.
- getClientCert(name: str) PKCS12KeyAndCertificates | None[source]
Loads the PKCS12 archive object for a given user keypair.
- Parameters:
name – The name of the user keypair.
Examples
Get the PKCS12 object for the user “myuser”:
mypkcs12 = cdir.getClientCert('myuser')
Notes
The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool
- Returns:
The PKCS12 archive, if exists.
- getClientCertPath(name: str) str | None[source]
Gets the path to a client certificate.
- Parameters:
name – The name of the client keypair.
Examples
Get the path to the client certificate for “myuser”:
mypath = cdir.getClientCertPath('myuser')
- Returns:
The path, if exists.
- getClientSSLContext(certname: str | None = None) SSLContext[source]
Returns an ssl.SSLContext appropriate for initiating a TLS session
- Parameters:
certname – If specified, use the user certificate with the matching name to authenticate to the remote service.
- Returns:
A SSLContext object.
- getHostCaPath(name: str) str | None[source]
Gets the path to the CA certificate that issued a given host keypair.
- Parameters:
name – The name of the host keypair.
Examples
Get the path to the CA cert which issue the cert for “myhost”:
mypath = cdir.getHostCaPath('myhost')
- Returns:
The path, if exists.
- getHostCert(name: str) Certificate | None[source]
Loads the X509 object for a given host keypair.
- Parameters:
name – The name of the host keypair.
Examples
Get the certificate object for the host “myhost”:
myhostcert = cdir.getHostCert('myhost')
- Returns:
The certificate, if exists.
- getHostCertPath(name: str) str | None[source]
Gets the path to a host certificate.
- Parameters:
name – The name of the host keypair.
Examples
Get the path to the host certificate for the host “myhost”:
mypath = cdir.getHostCertPath('myhost')
- Returns:
The path, if exists.
- getHostKey(name: str) RSAPrivateKey | DSAPrivateKey | None[source]
Loads the PKey object for a given host keypair.
- Parameters:
name – The name of the host keypair.
Examples
Get the private key object for the host “myhost”:
myhostkey = cdir.getHostKey('myhost')
- Returns:
The private key, if exists.
- getHostKeyPath(name: str) str | None[source]
Gets the path to a host key.
- Parameters:
name – The name of the host keypair.
Examples
Get the path to the host key for the host “myhost”:
mypath = cdir.getHostKeyPath('myhost')
- Returns:
The path if exists.
- Return type:
str
- getServerSSLContext(hostname: str | None = None, caname: str | None = None) SSLContext[source]
Returns an ssl.SSLContext appropriate to listen on a socket
- Parameters:
hostname – If None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory.
caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.
- Returns:
A SSLContext object.
- getUserCaPath(name: str) str | None[source]
Gets the path to the CA certificate that issued a given user keypair.
- Parameters:
name – The name of the user keypair.
Examples
Get the path to the CA cert which issue the cert for “myuser”:
mypath = cdir.getUserCaPath('myuser')
- Returns:
The path, if exists.
- getUserCert(name: str) Certificate | None[source]
Loads the X509 object for a given user keypair.
- Parameters:
name – The name of the user keypair.
Examples
Get the certificate object for the user “myuser”:
myusercert = cdir.getUserCert('myuser')
- Returns:
The certificate, if exists.
- getUserCertPath(name: str) str | None[source]
Gets the path to a user certificate.
- Parameters:
name (str) – The name of the user keypair.
Examples
Get the path for the user cert for “myuser”:
mypath = cdir.getUserCertPath('myuser')
- Returns:
The path, if exists.
- getUserForHost(user: str, host: str) str | None[source]
Gets the name of the first existing user cert for a given user and host.
- Parameters:
user – The name of the user.
host – The name of the host.
Examples
Get the name for the “myuser” user cert at “cool.vertex.link”:
usercertname = cdir.getUserForHost('myuser', 'cool.vertex.link')
- Returns:
The cert name, if exists.
- Return type:
str
- getUserKey(name: str) RSAPrivateKey | DSAPrivateKey | None[source]
Loads the PKey object for a given user keypair.
- Parameters:
name – The name of the user keypair.
Examples
Get the key object for the user key for “myuser”:
myuserkey = cdir.getUserKey('myuser')
- Returns:
The private key, if exists.
- getUserKeyPath(name: str) str | None[source]
Gets the path to a user key.
- Parameters:
name – The name of the user keypair.
Examples
Get the path to the user key for “myuser”:
mypath = cdir.getUserKeyPath('myuser')
- Returns:
The path, if exists.
- importFile(path: str, mode: str, outp: OutPut | None = None) None[source]
Imports certs and keys into the Synapse cert directory
- Parameters:
path – The path of the file to be imported.
mode – The certdir subdirectory to import the file into.
Examples
Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory:
certdir.importFile('mycoolca.crt', 'cas')
Notes
importFile does not perform any validation on the files it imports.
- Returns:
None
- isCaCert(name: str) bool[source]
Checks if a CA certificate exists.
- Parameters:
name – The name of the CA keypair.
Examples
Check if the CA certificate for “myca” exists:
exists = cdir.isCaCert('myca')
- Returns:
True if the certificate is present, False otherwise.
- isClientCert(name: str) bool[source]
Checks if a user client certificate (PKCS12) exists.
- Parameters:
name – The name of the user keypair.
Examples
Check if the client certificate “myuser” exists:
exists = cdir.isClientCert('myuser')
- Returns:
True if the certificate is present, False otherwise.
- isCodeCert(name: str) bool[source]
Checks if a code certificate exists.
- Parameters:
name – The name of the code keypair.
Examples
Check if the code cert “mypipeline” exists:
exists = cdir.isCodeCert('mypipeline')
- Returns:
True if the certificate is present, False otherwise.
- isHostCert(name: str) bool[source]
Checks if a host certificate exists.
- Parameters:
name – The name of the host keypair.
Examples
Check if the host cert “myhost” exists:
exists = cdir.isUserCert('myhost')
- Returns:
True if the certificate is present, False otherwise.
- isUserCert(name: str) bool[source]
Checks if a user certificate exists.
- Parameters:
name – The name of the user keypair.
Examples
Check if the user cert “myuser” exists:
exists = cdir.isUserCert('myuser')
- Returns:
True if the certificate is present, False otherwise.
- loadCertByts(byts: bytes) Certificate[source]
Load a X509 certificate from its PEM encoded bytes.
- Parameters:
byts – The PEM encoded bytes of the certificate.
- Returns:
The X509 certificate.
- Raises:
BadCertBytes – If the certificate bytes are invalid.
- saveCertPem(cert: Certificate, path: str) None[source]
Save a certificate in PEM format to a file outside the certdir.
- savePkeyPem(pkey: DHPrivateKey | Ed25519PrivateKey | Ed448PrivateKey | RSAPrivateKey | DSAPrivateKey | EllipticCurvePrivateKey | X25519PrivateKey | X448PrivateKey, path: str) None[source]
Save a private key in PEM format to a file outside the certdir.
- selfSignCert(builder: CertificateBuilder, pkey: RSAPrivateKey | DSAPrivateKey) Certificate[source]
Self-sign a certificate.
- Parameters:
cert – The certificate to sign.
pkey – The PKey with which to sign the certificate.
Examples
Sign a given certificate with a given private key:
cdir.selfSignCert(mycert, myotherprivatekey)
- Returns:
None
- signCertAs(builder: CertificateBuilder, signas: str) Certificate[source]
Signs a certificate with a CA keypair.
- Parameters:
cert – The certificate to sign.
signas – The CA keypair name to sign the new keypair with.
Examples
Sign a certificate with the CA “myca”:
cdir.signCertAs(mycert, 'myca')
- Returns:
None
- signHostCsr(xcsr: CertificateSigningRequest, signas: str, outp: OutPut | None = None, sans: str | None = None, save: bool = True) Tuple[RSAPrivateKey | None, Certificate][source]
Signs a host CSR with a CA keypair.
- Parameters:
xcsr – The certificate signing request.
signas – The CA keypair name to sign the CSR with.
outp – The output buffer.
sans – List of subject alternative names.
Examples
Sign a host key with the CA “myca”:
cdir.signHostCsr(mycsr, 'myca')
- Returns:
Tuple containing the public key and certificate objects.
- signUserCsr(xcsr: CertificateSigningRequest, signas: str, outp: OutPut | None = None, save: bool = True) Tuple[RSAPrivateKey | None, Certificate][source]
Signs a user CSR with a CA keypair.
- Parameters:
xcsr – The certificate signing request.
signas – The CA keypair name to sign the CSR with.
outp – The output buffer.
Examples
Sign a user CSR with “myca”:
cdir.signUserCsr(mycsr, 'myca')
- Returns:
Tuple containing the public key and certificate objects.
- Return type:
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- valCodeCert(byts: bytes) Certificate[source]
Verify a code cert is valid according to certdir’s available CAs and CRLs.
- Parameters:
byts – The certificate bytes.
- Raises:
s_exc.BadCertVerify if we are unable to verify the certificate. –
- Returns:
The certificate.
- valUserCert(byts: bytes, cacerts: List[Certificate] | None = None) Certificate[source]
Validate the PEM encoded x509 user certificate bytes and return it.
- Parameters:
byts – The bytes for the User Certificate.
cacerts – A tuple of CA Certificates to use for validating the user cert.
- Raises:
BadCertVerify – If the certificate is not valid.
- Returns:
The certificate, if it is valid.
- synapse.lib.certdir.getCertDir() CertDir[source]
Get the singleton CertDir instance.
- Returns:
A certdir object.
- Return type:
- synapse.lib.certdir.getCertDirn() str[source]
Get the expanded default path used by the singleton CertDir instance.
- Returns:
The path string.
- Return type:
str
- synapse.lib.certdir.getServerSSLContext() SSLContext[source]
Get a server SSLContext object.
This object has a minimum TLS version of 1.2, a subset of ciphers in use, and disabled client renegotiation.
This object has no certificates loaded in it.
- Returns:
The context object.
- Return type:
ssl.SSLContext
synapse.lib.chop module
- synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)
Shared primitive routines for chopping up strings and values.
- synapse.lib.chop.cvss_validate(vect, vers)[source]
- Validate (as best as possible) the CVSS vector string. Look for issues such as:
No duplicated metrics
Invalid metrics
Invalid metric values
Missing mandatory metrics
Returns a dictionary with the parsed metric:value pairs.
- synapse.lib.chop.hexstr(text)[source]
Ensure a string is valid hex.
- Parameters:
text (str) – String to normalize.
Examples
Norm a few strings:
hexstr(‘0xff00’) hexstr(‘ff00’)
Notes
Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.
- Returns:
Normalized hex string.
- Return type:
str
- synapse.lib.chop.replaceUnicodeDashes(valu)[source]
Replace unicode dashes in a string with regular dashes.
- Parameters:
valu (str) – A string.
- Returns:
A new string with replaced dashes.
- Return type:
str
- synapse.lib.chop.stormstring(s)[source]
Make a string storm safe by escaping backslashes and double quotes.
- Parameters:
s (str) – String to make storm safe.
Notes
This does not encapsulate a string in double quotes.
- Returns:
A string which can be embedded directly into a storm query.
- Return type:
str
- synapse.lib.chop.uncnorm(valu)[source]
Validate and normalize the UNC path passed in valu into a URI.
This function will accept @SSL and @<port> as part of the host name to indicate SSL (https) or a specific port number. It can also accept IPv6 addresses in the host name even though those are non-standard according to the spec.
synapse.lib.cli module
- class synapse.lib.cli.Cli[source]
Bases:
BaseA modular / event-driven CLI base object.
- async addSignalHandlers()[source]
Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task. Removes the handler when the cli is fini’d.
- histfile = 'cmdr_history'
- async runCmdLine(line)[source]
Run a single command line.
- Parameters:
line (str) – Line to execute.
Examples
Execute the ‘woot’ command with the ‘help’ switch:
await cli.runCmdLine(‘woot –help’)
- Returns:
Arbitrary data from the cmd class.
- Return type:
object
- class synapse.lib.cli.Cmd(cli, **opts)[source]
Bases:
objectBase class for modular commands in the synapse CLI.
- getCmdOpts(text)[source]
Use the _cmd_syntax def to split/parse/normalize the cmd line.
- Parameters:
text (str) – Command to process.
Notes
This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type
- Returns:
An opts dictionary.
- Return type:
dict
- class synapse.lib.cli.CmdHelp(cli, **opts)[source]
Bases:
CmdList commands and display help output.
Example
help foocmd
synapse.lib.cmd module
synapse.lib.cmdr module
- async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)[source]
Construct and return a cmdr for the given remote cell.
- Parameters:
cell – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.
Examples
Get the cmdr for a proxy:
cmdr = await getItemCmdr(foo)
- Returns:
A Cli instance with Cmds loaeded into it.
- Return type:
s_cli.Cli
- async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)[source]
Create a cmdr for the given item and run the cmd loop.
- Parameters:
item – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.
Notes
This function does not return while the command loop is run.
Examples
Run the Cmdr for a proxy:
await runItemCmdr(foo)
- Returns:
This function returns None.
- Return type:
None
synapse.lib.config module
- class synapse.lib.config.Config(schema, conf=None, envar_prefixes=None)[source]
Bases:
MutableMappingSynapse configuration helper based on JSON Schema.
- Parameters:
schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.
conf (dict) – Optional, a set of configuration data to preload.
envar_prefixes (list) – Optional, a list of prefix strings used when collecting configuration data from environment variables.
Notes
This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.
The default values provided in the schema must be able to be recreated from the repr() of their Python value.
Default values are not loaded into the configuration data until the
reqConfValid()method is called.- asDict()[source]
Get a copy of configuration data.
- Returns:
A copy of the configuration data.
- Return type:
dict
- classmethod getConfFromCell(cell, conf=None, envar_prefixes=None)[source]
Get a Config object from a Cell directly (either the ctor or the instance thereof).
- Returns:
A Config object.
- Return type:
- getEnvarMapping(prefix=None)[source]
Get a mapping of config values to envars.
Configuration values which have the
hideconfvalue set to True are not resolved from environment variables.
- req(key)[source]
Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.
- Parameters:
key (str) – The key to require.
- Returns:
The requested value.
- reqConfValid()[source]
Validate that the loaded configuration data is valid according to the schema.
Notes
The validation set does set any default values which are not currently set for configuration options.
- Returns:
This returns nothing.
- Return type:
None
- reqKeyValid(key, value)[source]
Test if a key is valid for the provided schema it is associated with.
- Parameters:
key (str) – Key to check.
value – Value to check.
- Raises:
BadArg – If the key has no associated schema.
BadConfValu – If the data is not schema valid.
- Returns:
None when valid.
- setConfFromEnvs()[source]
Set configuration options from environment variables.
Notes
Environment variables are resolved from configuration options after doing the following transform:
Replace
:characters with_.Add a config provided prefix, if set.
Uppercase the string.
Resolve the environment variable
If the environment variable is set, set the config value to the results of
yaml.yaml_safeload()on the value.
Configuration values which have the
hideconfvalue set to True are not resolved from environment variables.Examples
For the configuration value
auth:passwd, the environment variable is resolved asAUTH_PASSWD. With the prefixcortex, the the environment variable is resolved asCORTEX_AUTH_PASSWD.- Returns:
Returns a dictionary of values which were set from enviroment variables.
- Return type:
dict
- synapse.lib.config.getJsSchema(confbase, confdefs)[source]
Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.
- Parameters:
confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.
confdefs (dict) – A JSON Schema dictionary of properties for the object.
Notes
This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify
- Returns:
A complete JSON schema.
- Return type:
dict
- synapse.lib.config.getJsValidator(schema, use_default=True, handlers={'http': <function localSchemaRefHandler>, 'https': <function localSchemaRefHandler>})[source]
Get a fastjsonschema callable.
- Parameters:
schema (dict) – A JSON Schema object.
use_default (bool) – Whether to insert “default” key arguments into the validated data structure.
- Returns:
A callable function that can be used to validate data against the json schema.
- Return type:
callable
- synapse.lib.config.localSchemaRefHandler(uri)[source]
This function parses the given URI to get the path component and then tries to resolve the referenced schema from the ‘jsonschemas’ directory of synapse.data.
- synapse.lib.config.make_envar_name(key, prefix=None)[source]
Convert a colon delimited string into an uppercase, underscore delimited string.
- Parameters:
key (str) – Config key to convert.
prefix (str) – Optional string prefix to prepend the the config key.
- Returns:
The string to lookup against a envar.
- Return type:
str
synapse.lib.const module
synapse.lib.coro module
Async/Coroutine related utilities.
- async synapse.lib.coro.agen(item)[source]
Wrap an async_generator or generator in an async_generator.
Notes
Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.
- async synapse.lib.coro.event_wait(event: Event, timeout=None)[source]
Wait on an an asyncio event with an optional timeout
- Returns:
true if the event got set, False if timed out
- synapse.lib.coro.executor(func, *args, **kwargs)[source]
Execute a non-coroutine function in the ioloop executor pool.
- Parameters:
func – Function to execute.
*args – Args for the function.
**kwargs – Kwargs for the function.
Examples
Execute a blocking API call in the executor pool:
import requests def block(url, params=None): return requests.get(url, params=params).json() fut = s_coro.executor(block, 'http://some.tld/thign') resp = await fut
- Returns:
An asyncio future.
- Return type:
asyncio.Future
- async synapse.lib.coro.forked(func, *args, **kwargs)[source]
Execute a target function in the shared forked process pool and fallback to running in a spawned process if the pool is unavailable.
- Parameters:
func – The target function.
*args – Function positional arguments.
**kwargs – Function keyword arguments.
- Returns:
The target function return.
- async synapse.lib.coro.ornot(func, *args, **kwargs)[source]
Calls func and awaits it if a returns a coroutine.
Note
This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.
This is also useful when calling a callback that might either be a coroutine function or a regular function.
- Usage:
ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)
- async synapse.lib.coro.semafork(func, *args, **kwargs)[source]
Execute a target function in the shared forked process pool gated by a semaphore to ensure there are workers reserved for the Storm parser.
- Parameters:
func – The target function.
*args – Function positional arguments.
**kwargs – Function keyword arguments.
- Returns:
The target function return.
- async synapse.lib.coro.spawn(todo, timeout=None, ctx=None, log_conf=None)[source]
Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.
- Parameters:
todo (tuple) – A tuple of function,
*args, and**kwargs.timeout (int) – The timeout to wait for the todo function to finish.
ctx (multiprocess.Context) – A optional multiprocessing context object.
log_conf (dict) – An optional logging configuration for the spawned process.
Notes
The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.
- Returns:
The return value of executing the todo function.
synapse.lib.datfile module
Utilities for handling data files embedded within python packages.
synapse.lib.drive module
- class synapse.lib.drive.Drive[source]
Bases:
BaseDrive is a hierarchical storage abstraction which:
Provides enveloping which includes meta data for each item: * creator iden / time * updated iden / time / version * number of children * data type for the item * easy perms (enforcement is up to the caller)
Enforces schemas for data
Allows storage of historical versions of data
Provides a “path traversal” based API
Provides an iden based API that does not require traversal
- async addItemInfo(info, path=None, reldir='00000000000000000000000000000000')[source]
Add a new item at the specified path relative to reldir.
- getItemData(iden, vers=None)[source]
Return a (versinfo, data) tuple for the given iden. If version is not specified, the current version is returned.
- async getPathInfo(path, reldir='00000000000000000000000000000000')[source]
Return a list of item info for each step in the given path relative to rootdir.
This API is designed to allow the caller to retrieve the path info and potentially check permissions on each level to control access.
synapse.lib.dyndeps module
- synapse.lib.dyndeps.getDynLocal(name)[source]
Dynamically import a python module and return a local.
Example
cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()
- synapse.lib.dyndeps.getDynMod(name)[source]
Dynamically import a python module and return a ref (or None).
Example
mod = getDynMod(‘foo.bar’)
- synapse.lib.dyndeps.runDynTask(task)[source]
Run a dynamic task and return the result.
Example
foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )
- synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)[source]
Dynamically import a module and call a function or raise an exception.
synapse.lib.encoding module
- synapse.lib.encoding.decode(name, byts, **opts)[source]
Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.
Example
byts = s_encoding.decode(‘base64’,byts)
- Note: Decoder names may also be prefixed with +
to encode for that name/layer.
- synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)[source]
Iterate through the data provided by a file like object.
Optional parameters may be used to control how the data is deserialized.
Examples
The following example show use of the iterdata function.:
with open('foo.csv','rb') as fd: for row in iterdata(fd, format='csv', encoding='utf8'): dostuff(row)
- Parameters:
fd (file) – File like object to iterate over.
close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.
**opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.
- Yields:
An item to process. The type of the item is dependent on the format parameters.
synapse.lib.gis module
- synapse.lib.gis.bbox(lat, lon, dist)[source]
Calculate a min/max bounding box for the circle defined by lat/lon/dist.
- Parameters:
lat (float) – The latitude in degrees
lon (float) – The longitude in degrees
dist (int) – A distance in geo:dist base units (mm)
- Returns:
(latmin, latmax, lonmin, lonmax)
- Return type:
(float,float,float,float)
- synapse.lib.gis.dms2dec(degs, mins, secs)[source]
Convert degrees, minutes, seconds lat/long form to degrees float.
- Parameters:
degs (int) – Degrees
mins (int) – Minutes
secs (int) – Seconds
- Returns:
Degrees
- Return type:
(float)
- synapse.lib.gis.haversine(px, py, r=6371008800.0)[source]
Calculate the haversine distance between two points defined by (lat,lon) tuples.
- Parameters:
px ((float,float)) – lat/long position 1
py ((float,float)) – lat/long position 2
r (float) – Radius of sphere
- Returns:
Distance in mm.
- Return type:
(int)
- synapse.lib.gis.latlong(text)[source]
Chop a latlong string and return (float,float). Does not perform validation on the coordinates.
- Parameters:
text (str) – A longitude,latitude string.
- Returns:
A longitude, latitude float tuple.
- Return type:
(float,float)
- synapse.lib.gis.near(point, dist, points)[source]
Determine if the given point is within dist of any of points.
- Parameters:
point ((float,float)) – A latitude, longitude float tuple.
dist (int) – A distance in mm ( base units )
points (list) – A list of latitude, longitude float tuples to compare against.
synapse.lib.grammar module
synapse.lib.hashitem module
synapse.lib.hashset module
synapse.lib.health module
- class synapse.lib.health.HealthCheck(iden)[source]
Bases:
object- update(name, status, mesg='', data=None)[source]
Append a new component to the Healcheck object.
- Parameters:
name (str) – Name of the reported component.
status (str) – nomdinal/degraded/failed status code.
mesg (str) – Optional message about the component status.
data (dict) – Optional arbitrary dictionary of additional metadata about the component.
- Returns:
None
synapse.lib.hive module
- class synapse.lib.hive.Hive[source]
Bases:
PusherAn optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.
- async dict(full, nexs=False)[source]
Open a HiveDict at the given full path.
- Parameters:
full (tuple) – A full path tuple.
- Returns:
A HiveDict for the full path.
- Return type:
- dir(full)[source]
List subnodes of the given Hive path.
- Parameters:
full (tuple) – A full path tuple.
Notes
This returns None if there is not a node at the path.
- Returns:
A list of tuples. Each tuple contains the name, node value, and the number of children nodes.
- Return type:
list
- async get(full, defv=None)[source]
Get the value of a node at a given path.
- Parameters:
full (tuple) – A full path tuple.
- Returns:
Arbitrary node value.
- async open(full)[source]
Open and return a hive Node().
- Parameters:
full (tuple) – A full path tuple.
- Returns:
A Hive node.
- Return type:
- class synapse.lib.hive.Node[source]
Bases:
BaseA single node within the Hive tree.
- async dict(nexs=False)[source]
Get a HiveDict for this Node.
- Returns:
A HiveDict for this Node.
- Return type:
synapse.lib.httpapi module
- class synapse.lib.httpapi.ActiveV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthAddRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthAddUserV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthDelRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthGrantV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler/api/v1/auth/grant?user=iden&role=iden
- class synapse.lib.httpapi.AuthRevokeV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler/api/v1/auth/grant?user=iden&role=iden
- class synapse.lib.httpapi.AuthRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthRolesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthUserPasswdV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthUserV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.AuthUsersV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.BeholdSockV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
WebSocket
- class synapse.lib.httpapi.CoreInfoV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler/api/v1/core/info
- class synapse.lib.httpapi.ExtApiHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler/api/ext/.*
- compute_etag()[source]
Computes the etag header to be used for this request.
By default uses a hash of the content written so far.
May be overridden to provide custom etag implementations, or may return None to disable tornado’s default etag support.
- set_default_headers()[source]
Override this to set HTTP headers at the beginning of the request.
For example, this is the place to set a custom
Serverheader. Note that setting such headers in the normal flow of request processing may not do what you want, since headers may be reset during error handling.
- storm_prefix = 'init { $request = $lib.cortex.httpapi.response($_http_request_info) }'
- class synapse.lib.httpapi.FeedV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler/api/v1/feed
Examples
Example data:
{ 'name': 'syn.nodes', 'view': null, 'items': [...], }
- class synapse.lib.httpapi.Handler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
HandlerBase,RequestHandler- on_connection_close()[source]
Called in async handlers if the client closed the connection.
Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.
Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.
- prepare()[source]
Called at the beginning of a request before get/post/etc.
Override this method to perform common initialization regardless of the request method.
Asynchronous support: Use
async defor decorate this method with .gen.coroutine to make it asynchronous. If this method returns anAwaitableexecution will not proceed until theAwaitableis done.Added in version 3.1: Asynchronous support.
- class synapse.lib.httpapi.HandlerBase[source]
Bases:
object- async allowed(perm, default=False, gateiden=None)[source]
Check if the authenticated user has the given permission.
- Parameters:
perm (tuple) – The permission tuple to check.
default (boolean) – The default value for the permission.
gateiden (str) – The gateiden to check the permission against.
Notes
This API sets up HTTP response values if it returns False.
- Returns:
True if the user has the requested permission.
- Return type:
bool
- async authenticated()[source]
Check if the request has an authenticated user or not.
- Returns:
True if the request has an authenticated user, false otherwise.
- Return type:
bool
- async getUseridenBody(validator=None)[source]
Helper function to confirm that there is an auth user and a valid JSON body in the request.
- Parameters:
validator – Validator function run on the deserialized JSON body.
- Returns:
The user definition and body of the request as deserialized JSON, or a tuple of s_common.novalu objects if there was no user or json body.
- Return type:
(str, object)
- async handleBasicAuth()[source]
Handle basic authentication in the handler.
Notes
Implementors may override this to disable or implement their own basic auth schemes. This is expected to set web_useriden and web_username upon successful authentication.
- Returns:
The user iden of the logged in user.
- Return type:
str
- async isUserAdmin()[source]
Check if the current authenticated user is an admin or not.
- Returns:
True if the user is an admin, false otherwise.
- Return type:
bool
- logAuthIssue(mesg=None, user=None, username=None, level=30)[source]
Helper to log issues related to request authentication.
- Parameters:
mesg (str) – Additional message to log.
user (str) – User iden, if available.
username (str) – Username, if available.
level (int) – Logging level to log the message at. Defaults to logging.WARNING.
- Returns:
None
- async reqAuthAdmin()[source]
Require the current authenticated user to be an admin.
Notes
If this returns False, an error message has already been sent and no additional processing for the request should be done.
- Returns:
True if the user is an admin, false otherwise.
- Return type:
bool
- sendRestErr(code: str, mesg: str, *, status_code: int | HTTPStatus | None = None) None[source]
Send a JSON REST error message with a code and message.
- Parameters:
code – The error code.
mesg – The error message.
status_code – The HTTP status code. This is optional.
Notes
If the status code is not provided or set prior to calling this API, the response will have an HTTP status code of 200 (OK).
This does write the response stream. No further content should be written in the response after calling this.
- sendRestExc(e: Exception, *, status_code: int | HTTPStatus | None = None) None[source]
Send a JSON REST error message based on the exception.
- Parameters:
e – The exception to send. The exception class name will be used as the error code.
status_code – The HTTP status code. This is optional.
Notes
If the status code is not provided or set prior to calling this API, the response will have an HTTP status code of 200.
This does write the response stream. No further content should be written in the response after calling this.
- sendRestRetn(valu, *, status_code: int | HTTPStatus | None = None) None[source]
Send a successful JSON REST response.
- Parameters:
valu – The JSON compatible value to send.
status_code – The HTTP status code. This is optional.
Notes
If the status code is not provided or set prior to calling this API, the response will have an HTTP status code of 200.
This does write the response stream. No further content should be written in the response after calling this.
- async sess(gen=True)[source]
Get the heavy Session object for the request.
- Parameters:
gen (bool) – If set to True, generate a new session if there is no sess cookie.
Notes
This stores the identifier in the
sesscookie for with a 14 day expiration, stored in the Cell. Valid requests with thatsesscookie will resolve to the same Session object.- Returns:
A heavy session object. If the sess cookie is invalid or gen is false, this returns None.
- Return type:
- class synapse.lib.httpapi.HealthCheckV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.LoginV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.LogoutV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.ModelNormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.ModelV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.OnePassIssueV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler/api/v1/auth/onepass/issue
- class synapse.lib.httpapi.ReqValidStormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler
- class synapse.lib.httpapi.RobotHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
HandlerBase,RequestHandler
- class synapse.lib.httpapi.StormCallV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler
- class synapse.lib.httpapi.StormExportV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler
- class synapse.lib.httpapi.StormHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.StormNodesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler
- class synapse.lib.httpapi.StormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
StormHandler
- class synapse.lib.httpapi.StormVarsGetV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.StormVarsPopV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.StormVarsSetV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
Handler
- class synapse.lib.httpapi.StreamHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
HandlerSubclass for Tornado streaming uploads.
Notes
Async method prepare() is called after headers are read but before body processing.
Sync method on_finish() can be used to cleanup after a request.
Sync method on_connection_close() can be used to cleanup after a client disconnect.
Async methods post(), put(), etc are called after the streaming has completed.
- class synapse.lib.httpapi.WebSocket(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]
Bases:
HandlerBase,WebSocketHandler
synapse.lib.ingest module
synapse.lib.interval module
A few utilities for dealing with intervals.
- synapse.lib.interval.fold(*vals)[source]
Initialize a new (min,max) tuple interval from values.
- Parameters:
*vals ([int,...]) – A list of values (or Nones)
- Returns:
A (min,max) interval tuple or None
- Return type:
((int,int))
synapse.lib.json module
- synapse.lib.json.dump(obj: Any, fp: BinaryIO, sort_keys: bool = False, indent: bool = False, default: Callable | None = None, newline: bool = False) None[source]
Serialize a python object to a file-like object opened in binary mode.
Similar to the standard library json.dump().
- Parameters:
obj (object) – The python object to serialize.
fp (file) – The python file pointer to write the serialized data to.
sort_keys (bool) – Sort dictionary keys. Default: False.
indent (bool) – Include 2 spaces of indentation. Default: False.
default (Optional[Callable]) – Callback for serializing unknown object types. Default: None.
newline (bool) – Append a newline to the end of the serialized data. Default: False.
Returns: None
- Raises:
synapse.exc.MustBeJsonSafe – This exception is raised when a python object cannot be serialized.
- synapse.lib.json.dumps(obj: Any, sort_keys: bool = False, indent: bool = False, default: Callable | None = None, newline: bool = False) bytes[source]
Serialize a python object to byte string.
Similar to the standard library json.dumps().
- Parameters:
obj (object) – The python object to serialize.
sort_keys (bool) – Sort dictionary keys. Default: False.
indent (bool) – Include 2 spaces of indentation. Default: False.
default (Optional[Callable]) – Callback for serializing unknown object types. Default: None.
newline (bool) – Append a newline to the end of the serialized data. Default: False.
- Returns:
The JSON serialized python object.
- Return type:
(bytes)
- Raises:
synapse.exc.MustBeJsonSafe – This exception is raised when a python object cannot be serialized.
- synapse.lib.json.jslines(*paths: list[str]) Iterator[Any][source]
Deserialize the JSON lines data at *paths.
- Parameters:
*paths – The file path parts to load the data from.
- Yields:
(object) – The deserialized JSON data from each line.
- Raises:
synapse.exc.BadJsonText – This exception is raised when there is an error deserializing the provided data.
- synapse.lib.json.jsload(*paths: str) Any[source]
Deserialize the JSON data at *paths.
- Parameters:
*paths – The file path parts to load the data from.
- Returns:
The deserialized JSON data.
- Return type:
(object)
- Raises:
synapse.exc.BadJsonText – This exception is raised when there is an error deserializing the provided data.
- synapse.lib.json.jssave(js: Any, *paths: list[str]) None[source]
Serialize the python object to a file.
- Parameters:
js – The python object to serialize.
*paths – The file path parts to save the data to.
Returns: None
- Raises:
synapse.exc.MustBeJsonSafe – This exception is raised when a python
object cannot be serialized. –
- synapse.lib.json.load(fp: BinaryIO) Any[source]
Deserialize JSON data from a file.
Similar to the standard library json.load().
- Parameters:
fp (file) – The python file pointer to read the data from.
- Returns:
The deserialized JSON data.
- Return type:
(object)
- Raises:
synapse.exc.BadJsonText – This exception is raised when there is an error deserializing the provided data.
- synapse.lib.json.loads(s: str | bytes) Any[source]
Deserialize a JSON string.
Similar to the standard library json.loads().
- Parameters:
s (str | bytes) – The JSON data to be deserialized.
- Returns:
The deserialized JSON data.
- Return type:
(object)
- Raises:
synapse.exc.BadJsonText – This exception is raised when there is an error deserializing the provided data.
- synapse.lib.json.reqjsonsafe(item: Any, strict: bool = False) None[source]
Check if a python object is safe to be serialized to JSON.
Uses default type coercion from synapse.lib.json.dumps.
- Parameters:
item (any) – The python object to check.
strict (bool) – If specified, do not fallback to python json library which is more permissive of unicode strings. Default: False
Returns: None if item is json serializable, otherwise raises an exception.
- Raises:
synapse.exc.MustBeJsonSafe – This exception is raised when the item
cannot be serialized. –
synapse.lib.jsonstor module
- class synapse.lib.jsonstor.JsonStor[source]
Bases:
BaseA filesystem like storage mechanism that allows hirarchical lookup of reference counted “objects” that have individually editable properties.
#TODO json validation by path glob matches? (persists?) #TODO GUID ACCESS with index generation by type #TODO registered types jsonschema with optional write-back validation
- async setPathLink(srcpath, dstpath)[source]
Add a link from the given srcpath to the dstpath. NOTE: This causes the item at dstpath to be incref’d
- class synapse.lib.jsonstor.JsonStorCell[source]
Bases:
Cell- cellapi
alias of
JsonStorApi
synapse.lib.layer module
The Layer 2.0 archtecture introduces several optimized node/message serialization formats used by the layers to optimize returning primitives and facilitate efficient node construction:
Note
This interface is subject to change between minor revisions.
Storage Types (<stortype>)
In Layers 2.0, each node property from the model has an associated “storage type”. Each storage type determines how the data is indexed and represented within the Layer. This formalizes the separation of “data model” from “storage model”. Each data model type has a “stortype” property which coresponds to one of the STOR_TYPE_XXX values. The knowledge of the mapping of data model types to storage types is the responsibility of the data model, making the Layer implementation fully decoupled from the data model.
Node Edits / Edits
A node edit consists of a (<buid>, <form>, [edits]) tuple. An edit is Tuple of (<type>, <info>, List[NodeEdits]) where the first element is an int that matches to an EDIT_* constant below, the info is a tuple that varies depending on the first element, and the third element is a list of dependent NodeEdits that will only be applied if the edit actually makes a change.
Storage Node (<sode>)
A storage node is a layer/storage optimized node representation which is similar to a “packed node”. A storage node may be partial ( as it is produced by a given layer ) and are joined by the view/snap into “full” storage nodes which are used to construct Node() instances.
Sode format:
(<buid>, { 'ndef': (<formname>, <formvalu>), 'props': { <propname>: <propvalu>, } 'tags': { <tagname>: <tagvalu>, } 'tagprops: { <tagname>: { <propname>: <propvalu>, }, } # changes that were *just* made. 'edits': [ <edit> ] }),
- class synapse.lib.layer.IndxBy(layr, abrv, db)[source]
Bases:
objectIndxBy sub-classes encapsulate access methods and encoding details for various types of properties within the layer to be lifted/compared by storage types.
- class synapse.lib.layer.Layer[source]
Bases:
PusherThe base class for a cortex layer.
- async getEditIndx()[source]
Returns what will be the next (i.e. 1 past the last) nodeedit log index.
- async getEditOffs()[source]
Return the offset of the last recorded log entry. Returns -1 if nodeedit log is disabled or empty.
- async getNodeValu(buid, prop=None)[source]
Retrieve either the form valu or a prop valu for the given node by buid.
- async getPropArrayCount(formname, propname=None)[source]
Return the number of invidiual value rows in the layer for the given array form/prop.
- async getPropCount(formname, propname=None, maxsize=None)[source]
Return the number of property rows in the layer for the given form/prop.
- async getStorNodes()[source]
Yield (buid, sode) tuples for all the nodes with props/tags/tagprops stored in this layer.
- async getStorNodesByForm(form)[source]
Yield (buid, sode) tuples for nodes of a given form with props/tags/tagprops/edges/nodedata in this layer.
- async getTagCount(tagname, formname=None)[source]
Return the number of tag rows in the layer for the given tag/form.
- async getTagPropCount(form, tag, prop)[source]
Return the number of property rows in the layer for the given form/tag/prop.
- async getUnivPropCount(propname, maxsize=None)[source]
Return the number of universal property rows in the layer for the given prop.
- async iterFormRows(form, stortype=None, startvalu=None)[source]
Yields buid, valu tuples of nodes of a single form, optionally (re)starting at startvalu.
- Parameters:
form (str) – A form name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.
- Returns:
AsyncIterator[Tuple(buid, valu)]
- async iterNodeEditLog(offs=0)[source]
Iterate the node edit log and yield (offs, edits, meta) tuples.
- async iterNodeEditLogBack(offs=0)[source]
Iterate the node edit log and yield (offs, edits, meta) tuples in reverse.
- async iterPropRows(form, prop, stortype=None, startvalu=None)[source]
Yields buid, valu tuples of nodes with a particular secondary property, optionally (re)starting at startvalu.
- Parameters:
form (str) – A form name.
prop (str) – A universal property name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.
- Returns:
AsyncIterator[Tuple(buid, valu)]
- async iterTagPropRows(tag, prop, form=None, stortype=None, startvalu=None)[source]
Yields (buid, valu) that match a tag:prop, optionally (re)starting at startvalu.
- Parameters:
tag (str) – tag name
prop (str) – prop name
form (Optional[str]) – optional form name
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.
- Returns:
AsyncIterator[Tuple(buid, valu)]
- async iterTagRows(tag, form=None, starttupl=None)[source]
Yields (buid, (valu, form)) values that match a tag and optional form, optionally (re)starting at starttupl.
- Parameters:
tag (str) – the tag to match
form (Optional[str]) – if present, only yields buids of nodes that match the form.
starttupl (Optional[Tuple[buid, form]]) – if present, (re)starts the stream of values there.
- Returns:
AsyncIterator[Tuple(buid, (valu, form))]
Note
This yields (buid, (tagvalu, form)) instead of just buid, valu in order to allow resuming an interrupted call by feeding the last value retrieved into starttupl
- async iterUnivRows(prop, stortype=None, startvalu=None)[source]
Yields buid, valu tuples of nodes with a particular universal property, optionally (re)starting at startvalu.
- Parameters:
prop (str) – A universal property name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.
- Returns:
AsyncIterator[Tuple(buid, valu)]
- async saveNodeEdits(edits, meta)[source]
Save node edits to the layer and return a tuple of (nexsoffs, changes).
Note: nexsoffs will be None if there are no changes.
- async storNodeEditsNoLift(nodeedits, meta)[source]
Execute a series of node edit operations.
Does not return the updated nodes.
- async syncIndexEvents(offs, matchdef, wait=True)[source]
Yield (offs, (buid, form, ETYPE, VALS, META)) tuples from the nodeedit log starting from the given offset. Only edits that match the filter in matchdef will be yielded.
Notes
ETYPE is an constant EDIT_* above. VALS is a tuple whose format depends on ETYPE, outlined in the comment next to the constant. META is a dict that may contain keys ‘user’ and ‘time’ to represent the iden of the user that initiated the change, and the time that it took place, respectively.
Additionally, every 1000 entries, an entry (offs, (None, None, EDIT_PROGRESS, (), ())) message is emitted.
The matchdef dict may contain the following keys: forms, props, tags, tagprops. The value must be a sequence of strings. Each key/val combination is treated as an “or”, so each key and value yields more events. forms: EDIT_NODE_ADD and EDIT_NODE_DEL events. Matches events for nodes with forms in the value list. props: EDIT_PROP_SET and EDIT_PROP_DEL events. Values must be in form:prop or .universal form tags: EDIT_TAG_SET and EDIT_TAG_DEL events. Values must be the raw tag with no #. tagprops: EDIT_TAGPROP_SET and EDIT_TAGPROP_DEL events. Values must be just the prop or tag:prop.
Will not yield any values if this layer was not created with logedits enabled
- Parameters:
offs (int) – starting nexus/editlog offset
matchdef (Dict[str, Sequence[str]]) – a dict describing which events are yielded
wait (bool) – whether to pend and stream value until this layer is fini’d
- async syncNodeEdits(offs, wait=True, reverse=False)[source]
Identical to syncNodeEdits2, but doesn’t yield meta
- async syncNodeEdits2(offs, wait=True, reverse=False)[source]
Once caught up with storage, yield them in realtime.
- Returns:
Tuple of offset(int), nodeedits, meta(dict)
- async waitEditOffs(offs, timeout=None)[source]
Wait for the node edit log to write an entry at/past the given offset.
- class synapse.lib.layer.LayerApi[source]
Bases:
CellApi- saveNodeEdits(edits, meta)[source]
Save node edits to the layer and return a tuple of (nexsoffs, changes).
Note: nexsoffs will be None if there are no changes.
- class synapse.lib.layer.StorTypeFloat(layr, stortype, size=8)[source]
Bases:
StorType- FloatPackNegMax = b'\x80\x00\x00\x00\x00\x00\x00\x00'
- FloatPackNegMin = b'\xff\xf0\x00\x00\x00\x00\x00\x00'
- FloatPackPosMax = b'\x7f\xf0\x00\x00\x00\x00\x00\x00'
- FloatPackPosMin = b'\x00\x00\x00\x00\x00\x00\x00\x00'
- FloatPacker = <_struct.Struct object>
- fpack()
S.pack(v1, v2, …) -> bytes
Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.
- class synapse.lib.layer.StorTypeFqdn(layr)[source]
Bases:
StorTypeUtf8
- class synapse.lib.layer.StorTypeLoc(layr)[source]
Bases:
StorTypeHier
- class synapse.lib.layer.StorTypeTag(layr)[source]
Bases:
StorTypeHier
- class synapse.lib.layer.StorTypeTime(layr)[source]
Bases:
StorTypeInt
synapse.lib.link module
- class synapse.lib.link.Link[source]
Bases:
BaseA Link() is created to wrap a socket reader/writer.
- async synapse.lib.link.connect(host, port, ssl=None, hostname=None, linkinfo=None)[source]
Async connect and return a Link().
- async synapse.lib.link.linkfile(mode='wb')[source]
Connect a socketpair to a file-object and return (link, file).
- async synapse.lib.link.listen(host, port, onlink, ssl=None)[source]
Listen on the given host/port and fire onlink(Link).
Returns a server object that contains the listening sockets
synapse.lib.lmdbslab module
- class synapse.lib.lmdbslab.Hist(slab, name)[source]
Bases:
objectA class for storing items in a slab by time.
Each added item is inserted into the specified db within the slab using the current epoch-millis time stamp as the key.
- class synapse.lib.lmdbslab.HotCount[source]
Bases:
HotKeyValLike HotKeyVal, but optimized for integer/count vals
- static DecFunc(b)
Decode a signed 64-bit int from 8 byte big-endian
- static EncFunc(i)
Encode a signed 64-bit int into 8 byte big-endian bytes
- class synapse.lib.lmdbslab.HotKeyVal[source]
Bases:
BaseA hot-loop capable keyval that only syncs on commit.
- static DecFunc(byts, use_list=False)
Use msgpack to de-serialize a python object.
- Parameters:
byts (bytes) – The bytes to de-serialize
Notes
String objects are decoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow decoding bad input strings.- Returns:
The de-serialized object
- Return type:
obj
- static EncFunc(item)
Use msgpack to serialize a compatible python object.
- Parameters:
item (obj) – The object to serialize
Notes
String objects are encoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow encoding bad input strings.- Returns:
The serialized bytes in msgpack format.
- Return type:
bytes
- class synapse.lib.lmdbslab.MultiQueue[source]
Bases:
BaseAllows creation/consumption of multiple durable queues in a slab.
- async dele(name, minoffs, maxoffs)[source]
Remove queue entries from minoffs, up-to (and including) the queue entry at maxoffs.
- async get(name, offs, wait=False, cull=True)[source]
Return (nextoffs, item) tuple or (-1, None) for the given offset.
- async gets(name, offs, size=None, cull=False, wait=False)[source]
Yield (offs, item) tuples from the message queue.
- class synapse.lib.lmdbslab.SafeKeyVal(slab, name, prefix='')[source]
Bases:
objectKey/value storage that does not store items in memory and ensures keys are < 512 characters in length.
- Note:
The key size limitation includes the length of any prefixes added by using getSubKeyVal().
- class synapse.lib.lmdbslab.Scan(slab, db)[source]
Bases:
objectA state-object used by Slab. Not to be instantiated directly.
- Parameters:
slab (Slab) – which slab the scan is over
db (str) – name of open database on the slab
- class synapse.lib.lmdbslab.ScanBack(slab, db)[source]
Bases:
ScanA state-object used by Slab. Not to be instantiated directly.
Scans backwards.
- class synapse.lib.lmdbslab.ScanKeys(slab, db, nodup=False)[source]
Bases:
ScanAn iterator over the keys of the database. If the database is dupsort, a key with multiple values with be yielded once for each value.
- class synapse.lib.lmdbslab.Slab[source]
Bases:
BaseA “monolithic” LMDB instance for use in a asyncio loop thread.
- COMMIT_PERIOD = 0.2
- DEFAULT_GROWSIZE = None
- DEFAULT_MAPSIZE = 1073741824
- WARN_COMMIT_TIME_MS = 1000
- allslabs = {}
- async copydb(sourcedbname, destslab, destdbname=None, progresscb=None)[source]
Copy an entire database in this slab to a new database in potentially another slab.
- Parameters:
sourcedbname (str) – name of the db in the source environment
destslab (LmdbSlab) – which slab to copy rows to
destdbname (str) – the name of the database to copy rows to in destslab
progresscb (Callable[int]) – if not None, this function will be periodically called with the number of rows completed
- Returns:
the number of rows copied
- Return type:
(int)
Note
If any rows already exist in the target database, this method returns an error. This means that one cannot use destdbname=None unless there are no explicit databases in the destination slab.
- async countByPref(byts, db=None, maxsize=None)[source]
Return the number of rows in the given db with the matching prefix bytes.
- dbexists(name)[source]
The DB exists already if there’s a key in the default DB with the name of the database
- async fini()[source]
Shut down the object and notify any onfini() coroutines.
- Returns:
Remaining ref count
- rangeexists(lmin, lmax=None, db=None)[source]
Returns True if at least one key exists in the range.
- scanByPref(byts, startkey=None, startvalu=None, db=None)[source]
- Parameters:
byts (bytes) – prefix to match on
startkey (Optional[bytes]) – if present, will start scanning at key=byts+startkey
startvalu (Optional[bytes]) – if present, will start scanning at (key+startkey, startvalu)
Notes
startvalu only makes sense if byts+startkey matches an entire key. startvalu is only value for dupsort=True dbs
- syncevnt = None
- synctask = None
- class synapse.lib.lmdbslab.SlabAbrv(slab, name)[source]
Bases:
objectA utility for translating arbitrary bytes into fixed with id bytes
- class synapse.lib.lmdbslab.SlabDict(slab, db=None, pref=b'')[source]
Bases:
objectA dictionary-like object which stores its props in a slab via a prefix.
It is assumed that only one SlabDict with a given prefix exists at any given time, but it is up to the caller to cache them.
- get(name, defval=None)[source]
Get a name from the SlabDict.
- Parameters:
name (str) – The key name.
defval (obj) – The default value to return.
- Returns:
The return value, or None.
- Return type:
(obj)
- items()[source]
Return a tuple of (prop, valu) tuples from the SlabDict.
- Returns:
Tuple of (name, valu) tuples.
- Return type:
(((str, object), …))
synapse.lib.modelrev module
- class synapse.lib.modelrev.ModelMigration_0_2_31[source]
Bases:
object
synapse.lib.module module
- class synapse.lib.module.CoreModule(core, conf=None)[source]
Bases:
object- confdefs = ()
- getConfPath()[source]
Get the path to the module specific config file (conf.yaml).
Notes
This creates the parent directory for the conf.yaml file if it does not exist. This API exists to allow a implementor to get the conf path during initCoreModule and drop a example config if needed. One use case of that is for missing configuration values, an example config can be written to the file and a exception raised.
- Returns:
Path to where the conf file is located at.
- Return type:
str
- getModDir()[source]
Get the path to the module specific directory.
Notes
This creates the directory if it did not previously exist.
- Returns:
The filepath to the module specific directory.
- Return type:
str
- getModName()[source]
Return the lowercased name of this module.
Notes
This pulls the
mod_nameattribute on the class. This allows an implementer to set a arbitrary name for the module. If this attribute is not set, it defaults toself.__class__.__name__.lower()and setsmod_nameto that value.- Returns:
The module name.
- Return type:
(str)
- getModPath(*paths)[source]
Construct a path relative to this module’s working directory.
- Parameters:
*paths – A list of path strings
Notes
This creates the module specific directory if it does not exist.
- Returns:
The full path (or None if no cortex dir is configured).
- Return type:
(str)
- getStormCmds()[source]
Module implementers may override this to provide a list of Storm commands which will be loaded into the Cortex.
- Returns:
A list of Storm Command classes (not instances).
- Return type:
list
- async initCoreModule()[source]
Module implementers may override this method to initialize the module after the Cortex has completed and is accessible to perform storage operations.
Notes
This is the preferred function to override for implementing custom code that needs to be executed during Cortex startup.
Any exception raised within this method will remove the module from the list of currently loaded modules.
This is called for modules after getModelDefs() and getStormCmds() has been called, in order to allow for model loading and storm command loading prior to code execution offered by initCoreModule.
A failure during initCoreModule will not unload data model or storm commands registered by the module.
- Returns:
None
- mod_name = None
- async preCoreModule()[source]
Module implementers may override this method to execute code immediately after a module has been loaded.
Notes
The
initCoreModulefunction is preferred for overriding instead ofpreCoreModule().No Cortex layer/storage operations will function in preCoreModule.
Any exception raised within this method will halt additional loading of the module.
- Returns:
None
synapse.lib.modules module
Module which implements the synapse module API/convention.
synapse.lib.msgpack module
- class synapse.lib.msgpack.Unpk[source]
Bases:
objectAn extension of the msgpack streaming Unpacker which reports sizes.
Notes
String objects are decoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow decoding bad input strings.- feed(byts)[source]
Feed bytes to the unpacker and return completed objects.
- Parameters:
byts (bytes) – Bytes to unpack.
Notes
It is intended that this function is called multiple times with bytes from some sort of a stream, as it will unpack and return objects as they are available.
- Returns:
List of tuples containing the item size and the unpacked item.
- Return type:
list
- synapse.lib.msgpack.deepcopy(item, use_list=False)[source]
Copy a msgpack serializable by packing then unpacking it. For complex primitives, this runs in about 1/3 the time of copy.deepcopy()
- synapse.lib.msgpack.dumpfile(item, path)[source]
Dump an object to a file by path.
- Parameters:
item (object) – The object to serialize.
path (str) – The file path to save.
- Returns:
None
- synapse.lib.msgpack.en(item)[source]
Use msgpack to serialize a compatible python object.
- Parameters:
item (obj) – The object to serialize
Notes
String objects are encoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow encoding bad input strings.- Returns:
The serialized bytes in msgpack format.
- Return type:
bytes
- synapse.lib.msgpack.isok(item)[source]
Returns True if the item can be msgpacked (by testing packing).
- synapse.lib.msgpack.iterfd(fd)[source]
Generator which unpacks a file object of msgpacked content.
- Parameters:
fd – File object to consume data from.
Notes
String objects are decoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow decoding bad input strings.- Yields:
Objects from a msgpack stream.
- synapse.lib.msgpack.iterfile(path, since=-1)[source]
Generator which yields msgpack objects from a file path.
- Parameters:
path – File path to open and consume data from.
Notes
String objects are decoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow decoding bad input strings.- Yields:
Objects from a msgpack stream.
- synapse.lib.msgpack.loadfile(path)[source]
Load and upack the msgpack bytes from a file by path.
- Parameters:
path (str) – The file path to a message pack file.
- Raises:
msgpack.exceptions.ExtraData – If the file contains multiple objects.
- Returns:
The decoded python object.
- Return type:
(obj)
- synapse.lib.msgpack.un(byts, use_list=False)[source]
Use msgpack to de-serialize a python object.
- Parameters:
byts (bytes) – The bytes to de-serialize
Notes
String objects are decoded using utf8 encoding. In order to handle potentially malformed input,
unicode_errors='surrogatepass'is set to allow decoding bad input strings.- Returns:
The de-serialized object
- Return type:
obj
synapse.lib.multislabseqn module
- class synapse.lib.multislabseqn.MultiSlabSeqn[source]
Bases:
BaseAn append-optimized sequence of byte blobs stored across multiple slabs for fast rotating/culling
- getOffsetEvent(offs: int) Event[source]
Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.
- async gets(offs, wait=True) AsyncIterator[Tuple[int, Any]][source]
Just like iter, but optionally waits for new entries once the end is reached.
- async iter(offs: int) AsyncIterator[Tuple[int, Any]][source]
Iterate over items in a sequence from a given offset.
- Parameters:
offs (int) – The offset to begin iterating from.
- Yields:
(indx, valu) – The index and valu of the item.
- async rotate() int[source]
Rotate the Nexus log at the current index.
Note
After this executes the tailseqn will be empty. Waiting for this indx to be written will indicate when it is possible to cull 1 minus the return value such that the rotated seqn is deleted.
- Returns:
The starting index of the new seqn
- Return type:
int
synapse.lib.nexus module
- class synapse.lib.nexus.ChangeDist[source]
Bases:
BaseA utility class to distribute new change entries to mirrors/followers
- class synapse.lib.nexus.NexsRoot[source]
Bases:
Base- async eat(nexsiden, event, args, kwargs, meta)[source]
Actually mutate for the given nexsiden instance.
- getChangeDist(offs: int) AsyncIterator[ChangeDist][source]
- async issue(nexsiden, event, args, kwargs, meta=None)[source]
If I’m not a follower, mutate, otherwise, ask the leader to make the change and wait for the follower loop to hand me the result through a future.
- async iter(offs: int, tellready=False, wait=True) AsyncIterator[Any][source]
Returns an iterator of change entries in the log
- async recover() None[source]
Replays the last entry in the nexus log in case we crashed between writing the log and applying it.
Notes
This must be called at cell startup after subsystems are initialized but before any write transactions might happen.
The log can only have recorded 1 entry ahead of what is applied. All log actions are idempotent, so replaying the last action that (might have) already happened is harmless.
- class synapse.lib.nexus.Pusher[source]
Bases:
BaseA mixin-class to manage distributing changes where one might plug in mirroring or consensus protocols
- classmethod onPush(event: str, passitem=False) Callable[source]
Decorator that registers a method to be a handler for a named event
- Parameters:
event – string that distinguishes one handler from another. Must be unique per Pusher subclass
passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”
- classmethod onPushAuto(event: str, passitem=False) Callable[source]
Decorator that does the same as onPush, except automatically creates the top half method
- Parameters:
event – string that distinguishes one handler from another. Must be unique per Pusher subclass
passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”
synapse.lib.node module
- class synapse.lib.node.Node(snap, sode, bylayer=None)[source]
Bases:
objectA Cortex hypergraph node.
NOTE: This object is for local Cortex use during a single Xact.
- async addTag(tag, valu=(None, None))[source]
Add a tag to a node.
- Parameters:
tag (str) – The tag to add to the node.
valu – The optional tag value. If specified, this must be a value that norms as a valid time interval as an ival.
- Returns:
This returns None.
- Return type:
None
- async delete(force=False)[source]
Delete a node from the cortex.
The following tear-down operations occur in order:
validate that you have permissions to delete the node
validate that you have permissions to delete all tags
validate that there are no remaining references to the node.
- delete all the tags (bottom up)
fire onDelTag() handlers
delete tag properties from storage
- delete all secondary properties
fire onDelProp handler
delete secondary property from storage
- delete the primary property
fire onDel handlers for the node
delete primary property from storage
- get(name)[source]
Return a secondary property value from the Node.
- Parameters:
name (str) – The name of a secondary property.
- Returns:
The secondary property value or None.
- Return type:
(obj)
- pack(dorepr=False)[source]
Return the serializable/packed version of the node.
- Parameters:
dorepr (bool) – Include repr information for human readable versions of properties.
- Returns:
An (ndef, info) node tuple.
- Return type:
(tuple)
- reprs()[source]
Return a dictionary of repr values for props whose repr is different than the system mode value.
- async set(name, valu, init=False)[source]
Set a property on the node.
- Parameters:
name (str) – The name of the property.
valu (obj) – The value of the property.
init (bool) – Set to True to disable read-only enforcement
- Returns:
True if the property was changed.
- Return type:
(bool)
- async storm(runt, text, opts=None, path=None)[source]
- Parameters:
path (Path) – If set, then vars from path are copied into the new runtime, and vars are copied back out into path at the end
Note
If opts is not None and opts[‘vars’] is set and path is not None, then values of path vars take precedent
- class synapse.lib.node.Path(vars, nodes, links=None)[source]
Bases:
objectA path context tracked through the storm runtime.
- synapse.lib.node.iden(pode)[source]
Return the iden (buid) of the packed node.
- Parameters:
pode (tuple) – A packed node.
- Returns:
The node iden.
- Return type:
str
- synapse.lib.node.ndef(pode)[source]
Return a node definition (<form>,<valu>) tuple from the node.
- Parameters:
pode (tuple) – A packed node.
- Returns:
The (<form>,<valu>) tuple for the node
- Return type:
((str,obj))
- synapse.lib.node.prop(pode, prop)[source]
Return the valu of a given property on the node.
- Parameters:
pode (tuple) – A packed node.
prop (str) – Property to retrieve.
Notes
The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).
Returns:
- synapse.lib.node.props(pode)[source]
Get the props from the node.
- Parameters:
pode (tuple) – A packed node.
Notes
This will include any universal props present on the node.
- Returns:
A dictionary of properties.
- Return type:
dict
- synapse.lib.node.reprNdef(pode)[source]
Get the ndef of the pode with a human readable value.
- Parameters:
pode (tuple) – A packed node.
Notes
The human readable value is only available if the node came from a storm query execution where the
reprkey was passed into theoptsargument with a True value.- Returns:
A tuple of form and the human readable value.
- Return type:
(str, str)
- synapse.lib.node.reprProp(pode, prop)[source]
Get the human readable value for a secondary property from the pode.
- Parameters:
pode (tuple) – A packed node.
prop
Notes
The human readable value is only available if the node came from a storm query execution where the
reprkey was passed into theoptsargument with a True value.The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).
- Returns:
The human readable property value. If the property is not present, returns None.
- Return type:
str
- synapse.lib.node.reprTag(pode, tag)[source]
Get the human readable value for the tag timestamp from the pode.
- Parameters:
pode (tuple) – A packed node.
tag (str) – The tag to get the value for.
Notes
The human readable value is only available if the node came from a storm query execution where the
reprkey was passed into theoptsargument with a True value.If the tag does not have a timestamp, this returns a empty string.
- Returns:
The human readable value for the tag. If the tag is not present, returns None.
- Return type:
str
- synapse.lib.node.reprTagProps(pode, tag)[source]
Get the human readable values for any tagprops on a tag for a given node.
- Parameters:
pode (tuple) – A packed node.
tag (str) – The tag to get the tagprops reprs for.
Notes
The human readable value is only available if the node came from a storm query execution where the
reprkey was passed into theoptsargument with a True value.If the tag does not have any tagprops associated with it, this returns an empty list.
- Returns:
A list of tuples, containing the name of the tagprop and the repr value.
- Return type:
list
- synapse.lib.node.tagged(pode, tag)[source]
Check if a packed node has a given tag.
- Parameters:
pode (tuple) – A packed node.
tag (str) – The tag to check.
Examples
Check if a node is tagged with “woot” and dostuff if it is.
- if s_node.tagged(node,’woot’):
dostuff()
Notes
If the tag starts with #, this is removed prior to checking.
- Returns:
True if the tag is present. False otherwise.
- Return type:
bool
synapse.lib.oauth module
- class synapse.lib.oauth.OAuthMixin[source]
Bases:
PusherMixin for Cells to organize and execute OAuth token refreshes.
- async clearOAuthAccessToken(provideriden, useriden)[source]
Remove a client access token by clearing the configuration. This will prevent further refreshes (if scheduled), and a new auth code will be required the next time an access token is requested.
synapse.lib.output module
Tools for easily hookable output from cli-like tools.
synapse.lib.parser module
- class synapse.lib.parser.AstConverter(text)[source]
Bases:
TransformerConvert AST from parser into synapse AST, depth first.
If a method with a name that matches the current rule exists, that will be called, otherwise __default__ will be used
- class synapse.lib.parser.AstInfo(text, soff, eoff, sline, eline, scol, ecol, isterm)
Bases:
tuple- ecol
Alias for field number 6
- eline
Alias for field number 4
- eoff
Alias for field number 2
- isterm
Alias for field number 7
- scol
Alias for field number 5
- sline
Alias for field number 3
- soff
Alias for field number 1
- text
Alias for field number 0
- synapse.lib.parser.parseQuery(text, mode='storm')[source]
Parse a storm query and return the Lark AST. Cached here to speed up unit tests
- synapse.lib.parser.parse_cmd_string(text, off)[source]
Parse a command line string which may be quoted.
- synapse.lib.parser.unescape(valu)[source]
Parse a string for backslash-escaped characters and omit them. The full list of escaped characters can be found at https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
synapse.lib.queue module
- class synapse.lib.queue.AQueue[source]
Bases:
BaseAn async queue with chunk optimized sync compatible consumer.
- class synapse.lib.queue.Queue(maxsize=None)[source]
Bases:
objectAn asyncio Queue with batch methods and graceful close.
synapse.lib.ratelimit module
- class synapse.lib.ratelimit.RateLimit(rate, per)[source]
Bases:
objectA RateLimit class may be used to detect/enforce rate limits.
Example
# allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10)
Notes
It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”.
synapse.lib.reflect module
- synapse.lib.reflect.getClsNames(item)[source]
Return a list of “fully qualified” class names for an instance.
Example
- for name in getClsNames(foo):
print(name)
- synapse.lib.reflect.getItemLocals(item)[source]
Iterate the locals of an item and yield (name,valu) pairs.
Example
- for name,valu in getItemLocals(item):
dostuff()
- synapse.lib.reflect.getMethName(meth)[source]
Return a fully qualified string for the <mod>.<class>.<func> name of a given method.
Get a dictionary of special annotations for a Telepath Proxy.
- Parameters:
item – Item to inspect.
Notes
This will set the
_syn_telemethattribute on the item and the items class, so this data is only computed once.- Returns:
A dictionary of methods requiring special handling by the proxy.
- Return type:
dict
synapse.lib.rstorm module
- class synapse.lib.rstorm.OutPutRst[source]
Bases:
OutPutStrRst specific helper for output intended to be indented in RST text as a literal block.
- prefix = ' '
- class synapse.lib.rstorm.StormOutput(core, ctx, stormopts=None, opts=None)[source]
Bases:
StormCmdProduce standard output from a stream of storm runtime messages. Must be instantiated for a single query with a rstorm context.
synapse.lib.schemas module
synapse.lib.scope module
- class synapse.lib.scope.Scope(*frames, **vals)[source]
Bases:
objectThe Scope object assists in creating nested varible scopes.
Example
with Scope() as scope:
scope.set(‘foo’,10)
- with scope:
scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20…
dostuff(scope) # ‘foo’ is 10 again…
- copy()[source]
Create a shallow copy of the current Scope.
- Returns:
A new scope which is a copy of the current scope.
- Return type:
- synapse.lib.scope.clone(task: Task) None[source]
Clone the current task Scope onto the provided task.
- Parameters:
task (asyncio.Task) – The task object to attach the scope too.
Notes
This must be run from an asyncio IO loop.
If the current task does not have a scope, we clone the default global Scope.
This will
enter()the scope, and add a task callback toleave()the scope.- Returns:
None
- synapse.lib.scope.ctor(name, func, *args, **kwargs)[source]
Add a ctor callback to the global scope.
- synapse.lib.scope.get(name, defval=None)[source]
Access this task’s scope with default values from glob.
- synapse.lib.scope.pop(name)[source]
Pop and return a task scope variable. :param name: The task scope variable name. :type name: str
- Returns:
The scope value or None
- Return type:
obj
synapse.lib.scrape module
- synapse.lib.scrape.contextScrape(text, form=None, refang=True, first=False)[source]
Scrape types from a blob of text and yield info dictionaries.
- Parameters:
text (str) – Text to scrape.
form (str) – Optional form to scrape. If present, only scrape items which match the provided form.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.
Notes
The dictionaries yielded by this function contains the following keys:
- match
The raw matching text found in the input text.
- offset
The offset into the text where the match was found.
- valu
The resulting value.
- form
The corresponding form for the valu.
- Returns:
Yield info dicts of results.
- Return type:
(dict)
- async synapse.lib.scrape.contextScrapeAsync(text, form=None, refang=True, first=False)[source]
Scrape types from a blob of text and yield info dictionaries, using the shared forked process pool.
- Parameters:
text (str) – Text to scrape.
form (str) – Optional form to scrape. If present, only scrape items which match the provided form.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.
Notes
The dictionaries yielded by this function contains the following keys:
- match
The raw matching text found in the input text.
- offset
The offset into the text where the match was found.
- valu
The resulting value.
- form
The corresponding form for the valu.
- Returns:
Yield info dicts of results.
- Return type:
(dict)
- synapse.lib.scrape.genMatches(text: str, regx: compile, opts: dict)[source]
Generate regular expression matches for a blob of text.
- Parameters:
text (str) – The text to generate matches for.
regx (regex.Regex) – A compiled regex object. The regex must contained a named match group for
valu.opts (dict) – An options dictionary.
Notes
The dictionaries yielded by this function contains the following keys:
- raw_valu
The raw matching text found in the input text.
- offset
The offset into the text where the match was found.
- valu
The resulting value - this may be altered by callbacks.
The options dictionary can contain a
callbackkey. This function is expected to take a single argument, a regex.Match object, and return a tuple of the new valu and info dictionary. The new valu is used as thevalukey in the returned dictionary, and any other information in the info dictionary is pushed into the return dictionary as well.- Yields:
dict – A dictionary of match results.
- async synapse.lib.scrape.genMatchesAsync(text: str, regx: compile, opts: dict)[source]
Generate regular expression matches for a blob of text, using the shared forked process pool.
- Parameters:
text (str) – The text to generate matches for.
regx (regex.Regex) – A compiled regex object. The regex must contained a named match group for
valu.opts (dict) – An options dictionary.
Notes
The dictionaries yielded by this function contains the following keys:
- raw_valu
The raw matching text found in the input text.
- offset
The offset into the text where the match was found.
- valu
The resulting value - this may be altered by callbacks.
The options dictionary can contain a
callbackkey. This function is expected to take a single argument, a regex.Match object, and return a tuple of the new valu and info dictionary. The new valu is used as thevalukey in the returned dictionary, and any other information in the info dictionary is pushed into the return dictionary as well.- Yields:
dict – A dictionary of match results.
- synapse.lib.scrape.getForms()[source]
Get a list of forms recognized by the scrape APIs.
- Returns:
A list of form values.
- Return type:
list
- synapse.lib.scrape.refang_text(txt)[source]
Remove address de-fanging in text blobs, .e.g. example[.]com to example.com
Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM
- Returns:
Re-fanged text blob
- Return type:
(str)
- synapse.lib.scrape.refang_text2(txt: str, re: compile = regex.Regex('fxp:|fxps:|hxxp:|hxxps:|fxp\\[s\\]:|hxxp\\[s\\]:|ftp\\[:\\]|fxp\\[:\\]|ftps\\[:\\]|fxps\\[:\\]|http\\[:\\]|hxxp\\[:\\]|https\\[:\\]|hxxps\\[:\\]|ftp\\[://\\]|fxp\\[://\\]|ftps\\[://\\]|fxps\\[://\\]|http\\[://\\]|hxxp\\[://\\]|https\\[://\\]|hxxps\\[://\\]|ftp\\[:|fxp\\[:|ftps\\[:|fxps\\[:|http\\[:|hxxp\\[:|https\\[:|hxxps\\[:|ftp\\(:\\)|fxp\\(:\\)|ftps\\(:\\)|fxps\\(:\\)|http\\(:\\)|hxxp\\(:\\)|https\\(:\\)|hxxps\\(:\\)|\\[\\.\\]|\\.\\]|\\[\\.|\\[．\\]|\\[。\\]|\\[｡\\]|\\(\\.\\)|\\(．\\)|\\(。\\)|\\(｡\\)|\\[dot\\]|\\[:\\]|\\[at\\]|\\[@\\]|\\\\\\.', flags=regex.I | regex.V0), fangs: dict = {'(.)': '.', '(。)': '。', '(．)': '．', '(｡)': '｡', '.]': '.', '[.': '.', '[.]': '.', '[:]': ':', '[@]': '@', '[at]': '@', '[dot]': '.', '[。]': '。', '[．]': '．', '[｡]': '｡', '\\.': '.', 'ftp(:)': 'ftp:', 'ftp[:': 'ftp:', 'ftp[://]': 'ftp://', 'ftp[:]': 'ftp:', 'ftps(:)': 'ftps:', 'ftps[:': 'ftps:', 'ftps[://]': 'ftps://', 'ftps[:]': 'ftps:', 'fxp(:)': 'ftp:', 'fxp:': 'ftp:', 'fxp[:': 'ftp:', 'fxp[://]': 'ftp://', 'fxp[:]': 'ftp:', 'fxp[s]:': 'ftps:', 'fxps(:)': 'ftps:', 'fxps:': 'ftps:', 'fxps[:': 'ftps:', 'fxps[://]': 'ftps://', 'fxps[:]': 'ftps:', 'http(:)': 'http:', 'http[:': 'http:', 'http[://]': 'http://', 'http[:]': 'http:', 'https(:)': 'https:', 'https[:': 'https:', 'https[://]': 'https://', 'https[:]': 'https:', 'hxxp(:)': 'http:', 'hxxp:': 'http:', 'hxxp[:': 'http:', 'hxxp[://]': 'http://', 'hxxp[:]': 'http:', 'hxxp[s]:': 'https:', 'hxxps(:)': 'https:', 'hxxps:': 'https:', 'hxxps[:': 'https:', 'hxxps[://]': 'https://', 'hxxps[:]': 'https:'})[source]
Remove address de-fanging in text blobs, .e.g. example[.]com to example.com
Notes
Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example,
HXXP://FOO.COMwill be returned ashttp://FOO.COM- Parameters:
txt (str) – The text to re-fang.
- Returns:
A tuple containing the new text, and a dictionary containing offset information where the new text was altered with respect to the original text.
- Return type:
tuple(str, dict)
- synapse.lib.scrape.scrape(text, ptype=None, refang=True, first=False)[source]
Scrape types from a blob of text and return node tuples.
- Parameters:
text (str) – Text to scrape.
ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.
- Returns:
Yield tuples of node ndef values.
- Return type:
(str, object)
- async synapse.lib.scrape.scrapeAsync(text, ptype=None, refang=True, first=False)[source]
Scrape types from a blob of text and return node tuples, using the shared forked process pool.
- Parameters:
text (str) – Text to scrape.
ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.
- Returns:
Yield tuples of node ndef values.
- Return type:
(str, object)
synapse.lib.slaboffs module
synapse.lib.slabseqn module
- class synapse.lib.slabseqn.SlabSeqn(slab, name: str)[source]
Bases:
objectAn append optimized sequence of byte blobs.
- Parameters:
lenv (lmdb.Environment) – The LMDB Environment.
name (str) – The name of the sequence.
- async aiter(offs, wait=False, timeout=None)[source]
Iterate over items in a sequence from a given offset.
- Parameters:
offs (int) – The offset to begin iterating from.
wait (boolean) – Once caught up, yield new results in realtime.
timeout (int) – Max time to wait for a new item.
- Yields:
(indx, valu) – The index and valu of the item.
- getOffsetEvent(offs)[source]
Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.
- async gets(offs, wait=True)[source]
Returns an async generator of indx/valu tuples, optionally waiting and continuing to yield them as new entries are added
- Parameters:
offs (int) – The offset to begin iterating from.
wait (bool) – Whether to continue yielding tupls when it hits the end of the sequence.
- Yields:
(indx, valu) – The index and valu of the item.
- iter(offs, reverse=False)[source]
Iterate over items in a sequence from a given offset.
- Parameters:
offs (int) – The offset to begin iterating from.
- Yields:
(indx, valu) – The index and valu of the item.
- iterBack(offs)[source]
Iterate backwards over items in a sequence from a given offset.
- Parameters:
offs (int) – The offset to begin iterating from.
- Yields:
(indx, valu) – The index and valu of the item.
- nextindx()[source]
Determine the next insert offset according to storage.
- Returns:
The next insert offset.
- Return type:
int
synapse.lib.snap module
- class synapse.lib.snap.ProtoNode(ctx, buid, form, valu, node)[source]
Bases:
objectA prototype node used for staging node adds using a SnapEditor.
- TODO: This could eventually fully mirror the synapse.lib.node.Node API and be used
to slipstream into sections of the pipeline to facilitate a bulk edit / transaction
- class synapse.lib.snap.Snap[source]
Bases:
BaseA “snapshot” is a transaction across multiple Cortex layers.
The Snap object contains the bulk of the Cortex API to facilitate performance through careful use of transaction boundaries.
Transactions produce the following EventBus events:
(‘print’, {}),
- async addFeedNodes(name, items)[source]
Call a feed function and return what it returns (typically yields Node()s).
- Parameters:
name (str) – The name of the feed record type.
items (list) – A list of records of the given feed type.
- Returns:
The return value from the feed function. Typically Node() generator.
- Return type:
(object)
- async addNode(name, valu, props=None, norminfo=None)[source]
Add a node by form name and value with optional props.
- Parameters:
name (str) – The form of node to add.
valu (obj) – The value for the node.
props (dict) – Optional secondary properties for the node.
Notes
If a props dictionary is provided, it may be mutated during node construction.
- Returns:
A Node object. It may return None if the snap is unable to add or lift the node.
- Return type:
s_node.Node
- async addNodes(nodedefs)[source]
Add/merge nodes in bulk.
The addNodes API is designed for bulk adds which will also set properties, add tags, add edges, and set nodedata to existing nodes. Nodes are specified as a list of the following tuples:
( (form, valu), {‘props’:{}, ‘tags’:{}})
- Parameters:
nodedefs (list) – A list of nodedef tuples.
- Returns:
A list of xact messages.
- Return type:
(list)
- async applyNodeEdits(edits, nodecache=None, meta=None)[source]
Sends edits to the write layer and evaluates the consequences (triggers, node object updates)
- buidcachesize = 100000
- async getNodeByBuid(buid)[source]
Retrieve a node tuple by binary id.
- Parameters:
buid (bytes) – The binary ID for the node.
- Returns:
The node object or None.
- Return type:
Optional[s_node.Node]
- async getNodeByNdef(ndef)[source]
Return a single Node by (form,valu) tuple.
- Parameters:
ndef ((str,obj)) – A (form,valu) ndef tuple. valu must be
normalized.
- Returns:
The Node or None.
- Return type:
- async getNodeData(buid, name, defv=None)[source]
Get nodedata from closest to write layer, no merging involved
- async hasNodeData(buid, name)[source]
Return True if the buid has nodedata set on it under the given name False otherwise
- async iterStormPodes(text, opts, user=None)[source]
Yield packed node tuples for the given storm query text.
- async storm(text, opts=None, user=None)[source]
Execute a storm query and yield (Node(), Path()) tuples.
- tagcachesize = 1000
synapse.lib.spooled module
- class synapse.lib.spooled.Set[source]
Bases:
SpooledA minimal set-like implementation that will spool to a slab on large growth.
- class synapse.lib.spooled.Spooled[source]
Bases:
BaseA Base class that can be used to implement objects which fallback to lmdb.
These objects are intended to fallback from Python to lmbd slabs, which aligns them together. Under memory pressure, these objects have a better shot of getting paged out.
synapse.lib.storm module
- class synapse.lib.storm.BackgroundCmd(runt, runtsafe)[source]
Bases:
CmdExecute a query pipeline as a background task. NOTE: Variables are passed through but nodes are not
- name = 'background'
- class synapse.lib.storm.BatchCmd(runt, runtsafe)[source]
Bases:
CmdRun a query with batched sets of nodes.
The batched query will have the set of inbound nodes available in the variable $nodes.
This command also takes a conditional as an argument. If the conditional evaluates to true, the nodes returned by the batched query will be yielded, if it evaluates to false, the inbound nodes will be yielded after executing the batched query.
- NOTE: This command is intended to facilitate use cases such as queries to external
APIs with aggregate node values to reduce quota consumption. As this command interrupts the node stream, it should be used carefully to avoid unintended slowdowns in the pipeline.
Example
// Execute a query with batches of 5 nodes, then yield the inbound nodes batch $lib.false –size 5 { $lib.print($nodes) }
- name = 'batch'
- class synapse.lib.storm.Cmd(runt, runtsafe)[source]
Bases:
objectA one line description of the command.
Command usage details and long form description.
Example
cmd –help
Notes
Python Cmd implementers may override the
formsattribute with a dictionary to provide information about Synapse forms which are possible input and output nodes that a Cmd may recognize. A list of (key, form) tuples may also be added to provide information about forms which may have additional nodedata added to them by the Cmd.Example:
{ 'input': ( 'inet:ipv4', 'tel:mob:telem', ), 'output': ( 'geo:place', ), 'nodedata': ( ('foodata', 'inet:http:request'), ('bardata', 'inet:ipv4'), ), }
- asroot = False
- forms = {}
- name = 'cmd'
- pkgname = ''
- readonly = False
- svciden = ''
- class synapse.lib.storm.CopyToCmd(runt, runtsafe)[source]
Bases:
CmdCopy nodes from the current view into another view.
Examples
// Copy all nodes tagged with #cno.mal.redtree to the target view.
#cno.mal.redtree | copyto 33c971ac77943da91392dadd0eec0571
- name = 'copyto'
- class synapse.lib.storm.CountCmd(runt, runtsafe)[source]
Bases:
CmdIterate through query results, and print the resulting number of nodes which were lifted. This does not yield the nodes counted, unless the –yield switch is provided.
Example
# Count the number of IPV4 nodes with a given ASN. inet:ipv4:asn=20 | count
# Count the number of IPV4 nodes with a given ASN and yield them. inet:ipv4:asn=20 | count –yield
- name = 'count'
- readonly = True
- class synapse.lib.storm.DelNodeCmd(runt, runtsafe)[source]
Bases:
CmdDelete nodes produced by the previous query logic.
(no nodes are returned)
Example
inet:fqdn=vertex.link | delnode
- name = 'delnode'
- class synapse.lib.storm.DiffCmd(runt, runtsafe)[source]
Bases:
CmdGenerate a list of nodes with changes in the top layer of the current view.
Examples
// Lift all nodes with any changes
diff
// Lift ou:org nodes that were added in the top layer.
diff –prop ou:org
// Lift inet:ipv4 nodes with the :asn property modified in the top layer.
diff –prop inet:ipv4:asn
// Lift the nodes with the tag #cno.mal.redtree added in the top layer.
diff –tag cno.mal.redtree
// Lift nodes by multiple tags (results are uniqued)
diff –tag cno.mal.redtree rep.vt
- name = 'diff'
- readonly = True
- class synapse.lib.storm.DivertCmd(runt, runtsafe)[source]
Bases:
CmdEither consume a generator or yield it’s results based on a conditional.
- NOTE: This command is purpose built to facilitate the –yield convention
common to storm commands.
- NOTE: The genr argument must not be a function that returns, else it will
be invoked for each inbound node.
Example
divert $cmdopts.yield $fooBarBaz()
- name = 'divert'
- class synapse.lib.storm.EdgesDelCmd(runt, runtsafe)[source]
Bases:
CmdBulk delete light edges from input nodes.
Examples
# Delete all “foo” light edges from an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo
# Delete light edges with any verb from a node inet:ipv4=1.2.3.4 | edges.del *
# Delete all “foo” light edges to an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo –n2
- name = 'edges.del'
- class synapse.lib.storm.GraphCmd(runt, runtsafe)[source]
Bases:
CmdGenerate a subgraph from the given input nodes and command line options.
Example
Using the graph command:
inet:fqdn | graph --degrees 2 --filter { -#nope } --pivot { <(seen)- meta:source } --form-pivot inet:fqdn {<- * | limit 20} --form-pivot inet:fqdn {-> * | limit 20} --form-filter inet:fqdn {-inet:fqdn:issuffix=1} --form-pivot syn:tag {-> *} --form-pivot * {-> #}
- name = 'graph'
- class synapse.lib.storm.HelpCmd(runt, runtsafe)[source]
Bases:
CmdList available information about Storm and brief descriptions of different items.
Notes
If an item is provided, this can be a string or a function.
Examples
// Get all available commands, libraries, types, and their brief descriptions.
help
// Only get commands which have “model” in the name.
help model
// Get help about the base Storm library
help $lib
// Get detailed help about a specific library or library function
help –verbose $lib.print
// Get detailed help about a named Storm type
help –verbose str
// Get help about a method from a $node object
<inbound $node> help $node.tags
- name = 'help'
- class synapse.lib.storm.IdenCmd(runt, runtsafe)[source]
Bases:
CmdLift nodes by iden.
Example
iden b25bc9eec7e159dce879f9ec85fb791f83b505ac55b346fcb64c3c51e98d1175 | count
- name = 'iden'
- readonly = True
- class synapse.lib.storm.IntersectCmd(runt, runtsafe)[source]
Bases:
CmdYield an intersection of the results of running inbound nodes through a pivot.
Note
This command must consume the entire inbound stream to produce the intersection. This type of stream consuming before yielding results can cause the query to appear laggy in comparison with normal incremental stream operations.
Examples
// Show the it:mitre:attack:technique nodes common to several groups
it:mitre:attack:group*in=(G0006, G0007) | intersect { -> it:mitre:attack:technique }
- name = 'intersect'
- class synapse.lib.storm.LiftByVerb(runt, runtsafe)[source]
Bases:
CmdLift nodes from the current view by an light edge verb.
Examples
# Lift all the n1 nodes for the light edge “foo” lift.byverb “foo”
# Lift all the n2 nodes for the light edge “foo” lift.byverb –n2 “foo”
Notes
Only a single instance of a node will be yielded from this command when that node is lifted via the light edge membership.
- name = 'lift.byverb'
- class synapse.lib.storm.LimitCmd(runt, runtsafe)[source]
Bases:
CmdLimit the number of nodes generated by the query in the given position.
Example
inet:ipv4 | limit 10
- name = 'limit'
- readonly = True
- class synapse.lib.storm.MaxCmd(runt, runtsafe)[source]
Bases:
CmdConsume nodes and yield only the one node with the highest value for an expression.
Examples
// Yield the file:bytes node with the highest :size property file:bytes#foo.bar | max :size
// Yield the file:bytes node with the highest value for $tick file:bytes#foo.bar +.seen ($tick, $tock) = .seen | max $tick
// Yield the it:dev:str node with the longest length it:dev:str | max $lib.len($node.value())
- name = 'max'
- readonly = True
- class synapse.lib.storm.MergeCmd(runt, runtsafe)[source]
Bases:
CmdMerge edits from the incoming nodes down to the next layer.
NOTE: This command requires the current view to be a fork.
- NOTE: The arguments for including/excluding tags can accept tag glob
expressions for specifying tags. For more information on tag glob expressions, check the Synapse documentation for $node.globtags().
- NOTE: If –wipe is specified, and there are nodes that cannot be merged,
they will be skipped (with a warning printed) and removed when the top layer is replaced. This should occur infrequently, for example, when a form is locked due to deprecation, a form no longer exists, or the data at rest fails normalization.
Examples
// Having tagged a new #cno.mal.redtree subgraph in a forked view…
#cno.mal.redtree | merge –apply
// Print out what the merge command would do but dont.
#cno.mal.redtree | merge
// Merge any org nodes with changes in the top layer.
diff | +ou:org | merge –apply
// Merge all tags other than cno.* from ou:org nodes with edits in the // top layer.
diff | +ou:org | merge –only-tags –exclude-tags cno.** –apply
// Merge only tags rep.vt.* and rep.whoxy.* from ou:org nodes with edits // in the top layer.
diff | +ou:org | merge –include-tags rep.vt.* rep.whoxy.* –apply
// Lift only inet:ipv4 nodes with a changed :asn property in top layer // and merge all changes.
diff –prop inet:ipv4:asn | merge –apply
// Lift only nodes with an added #cno.mal.redtree tag in the top layer and merge them.
diff –tag cno.mal.redtree | merge –apply
- name = 'merge'
- class synapse.lib.storm.MinCmd(runt, runtsafe)[source]
Bases:
CmdConsume nodes and yield only the one node with the lowest value for an expression.
Examples
// Yield the file:bytes node with the lowest :size property file:bytes#foo.bar | min :size
// Yield the file:bytes node with the lowest value for $tick file:bytes#foo.bar +.seen ($tick, $tock) = .seen | min $tick
// Yield the it:dev:str node with the shortest length it:dev:str | min $lib.len($node.value())
- name = 'min'
- readonly = True
- class synapse.lib.storm.MoveNodesCmd(runt, runtsafe)[source]
Bases:
CmdMove storage nodes between layers.
Storage nodes will be removed from the source layers and the resulting storage node in the destination layer will contain the merged values (merged in bottom up layer order by default).
Examples
// Move storage nodes for ou:org nodes to the top layer
ou:org | movenodes –apply
// Print out what the movenodes command would do but dont.
ou:org | movenodes
// In a view with many layers, only move storage nodes from the bottom layer // to the top layer.
$layers = $lib.view.get().layers $top = $layers.0.iden $bot = $layers.”-1”.iden
ou:org | movenodes –srclayers $bot –destlayer $top
// In a view with many layers, move storage nodes to the top layer and // prioritize values from the bottom layer over the other layers.
$layers = $lib.view.get().layers $top = $layers.0.iden $mid = $layers.1.iden $bot = $layers.2.iden
ou:org | movenodes –precedence $bot $top $mid
- name = 'movenodes'
- class synapse.lib.storm.MoveTagCmd(runt, runtsafe)[source]
Bases:
CmdRename an entire tag tree and preserve time intervals.
Example
movetag foo.bar baz.faz.bar
- name = 'movetag'
- class synapse.lib.storm.OnceCmd(runt, runtsafe)[source]
Bases:
CmdThe once command is used to filter out nodes which have already been processed via the use of a named key. It includes an optional parameter to allow the node to pass the filter again after a given amount of time.
For example, to run an enrichment command on a set of nodes just once:
file:bytes#my.files | once enrich:foo | enrich.foo
The once command filters out any nodes which have previously been through any other use of the “once” command using the same <name> (in this case “enrich:foo”).
You may also specify the –asof option to allow nodes to pass the filter after a given amount of time. For example, the following command will allow any given node through every 2 days:
file:bytes#my.files | once enrich:foo –asof “-2 days” | enrich.foo
Use of “–asof now” or any future date or positive relative time offset will always allow the node to pass the filter.
State tracking data for the once command is stored as nodedata which is stored in your view’s write layer, making it view-specific. So if you have two views, A and B, and they do not share any layers between them, and you execute this query in view A:
inet:ipv4=8.8.8.8 | once enrich:address | enrich.baz
And then you run it in view B, the node will still pass through the once command to the enrich.baz portion of the query because the tracking data for the once command does not yet exist in view B.
- name = 'once'
- class synapse.lib.storm.ParallelCmd(runt, runtsafe)[source]
Bases:
CmdExecute part of a query pipeline in parallel. This can be useful to minimize round-trip delay during enrichments.
Examples
inet:ipv4#foo | parallel { $place = $lib.import(foobar).lookup(:latlong) [ :place=$place ] }
NOTE: Storm variables set within the parallel query pipelines do not interact.
- NOTE: If there are inbound nodes to the parallel command, parallel pipelines will be created as each node
is processed, up to the number specified by –size. If the number of nodes in the pipeline is less than the value specified by –size, additional pipelines with no inbound node will not be created. If there are no inbound nodes to the parallel command, the number of pipelines specified by –size will always be created.
- name = 'parallel'
- readonly = True
- class synapse.lib.storm.Parser(prog=None, descr=None, root=None, model=None, cdef=None)[source]
Bases:
object
- class synapse.lib.storm.ReIndexCmd(runt, runtsafe)[source]
Bases:
CmdUse admin privileges to re index/normalize node properties.
NOTE: Currently does nothing but is reserved for future use.
- name = 'reindex'
- class synapse.lib.storm.RunAsCmd(runt, runtsafe)[source]
Bases:
CmdExecute a storm query as a specified user.
NOTE: This command requires admin privileges.
- NOTE: Heavy objects (for example a View or Layer) are bound to the context which they
are instantiated in and methods on them will be run using the user in that context. This means that executing a method on a variable containing a heavy object which was instantiated outside of the runas command and then used within the runas command will check the permissions of the outer user, not the one specified by the runas command.
Examples
// Create a node as another user. runas someuser { [ inet:fqdn=foo.com ] }
- name = 'runas'
- class synapse.lib.storm.Runtime[source]
Bases:
BaseA Runtime represents the instance of a running query.
The runtime should maintain a firm API boundary using the snap. Parallel query execution requires that the snap be treated as an opaque object which is called through, but not dereferenced.
- confirm(perms, gateiden=None, default=None)[source]
Raise AuthDeny if the user doesn’t have the permission.
Notes
An elevated runtime with asroot=True will always return True.
- Parameters:
perms (tuple) – The permission tuple.
gateiden (str) – The gateiden.
default (bool) – The default value.
- Returns:
If the permission is allowed.
- Return type:
True
- Raises:
AuthDeny – If the user does not have the permission.
- getCmdRuntime(query, opts=None)[source]
Yield a runtime with proper scoping for use in executing a pure storm command.
- async getModRuntime(query, opts=None)[source]
Construct a non-context managed runtime for use in module imports.
- async getOneNode(propname, valu, filt=None, cmpr='=')[source]
Return exactly 1 node by <prop> <cmpr> <valu>
- getSubRuntime(query, opts=None)[source]
Yield a runtime with shared scope that will populate changes upward.
- async initSubRuntime(query, opts=None)[source]
Construct and return sub-runtime with a shared scope. ( caller must fini )
- class synapse.lib.storm.ScrapeCmd(runt, runtsafe)[source]
Bases:
CmdUse textual properties of existing nodes to find other easily recognizable nodes.
Examples
# Scrape properties from inbound nodes and create standalone nodes. inet:search:query | scrape
# Scrape properties from inbound nodes and make refs light edges to the scraped nodes. inet:search:query | scrape –refs
# Scrape only the :engine and :text props from the inbound nodes. inet:search:query | scrape :text :engine
# Scrape the primary property from the inbound nodes. it:dev:str | scrape $node.repr()
# Scrape properties inbound nodes and yield newly scraped nodes. inet:search:query | scrape –yield
# Skip re-fanging text before scraping. inet:search:query | scrape –skiprefang
# Limit scrape to specific forms. inet:search:query | scrape –forms (inet:fqdn, inet:ipv4)
- name = 'scrape'
- class synapse.lib.storm.SleepCmd(runt, runtsafe)[source]
Bases:
CmdIntroduce a delay between returning each result for the storm query.
NOTE: This is mostly used for testing / debugging.
Example
#foo.bar | sleep 0.5
- name = 'sleep'
- readonly = True
- class synapse.lib.storm.SpinCmd(runt, runtsafe)[source]
Bases:
CmdIterate through all query results, but do not yield any. This can be used to operate on many nodes without returning any.
Example
foo:bar:size=20 [ +#hehe ] | spin
- name = 'spin'
- readonly = True
- class synapse.lib.storm.StormDmon[source]
Bases:
BaseA background storm runtime which is restarted by the cortex.
- class synapse.lib.storm.TagPruneCmd(runt, runtsafe)[source]
Bases:
CmdPrune a tag (or tags) from nodes.
This command will delete the tags specified as parameters from incoming nodes, as well as all of their parent tags that don’t have other tags as children.
For example, given a node with the tags:
#parent #parent.child #parent.child.grandchild
Pruning the parent.child.grandchild tag would remove all tags. If the node had the tags:
#parent #parent.child #parent.child.step #parent.child.grandchild
Pruning the parent.child.grandchild tag will only remove the parent.child.grandchild tag as the parent tags still have other children.
Examples
# Prune the parent.child.grandchild tag inet:ipv4=1.2.3.4 | tag.prune parent.child.grandchild
- name = 'tag.prune'
- class synapse.lib.storm.TeeCmd(runt, runtsafe)[source]
Bases:
CmdExecute multiple Storm queries on each node in the input stream, joining output streams together.
Commands are executed in order they are given; unless the
--parallelswitch is provided.Examples
# Perform a pivot out and pivot in on a inet:ivp4 node inet:ipv4=1.2.3.4 | tee { -> * } { <- * }
# Also emit the inbound node inet:ipv4=1.2.3.4 | tee –join { -> * } { <- * }
# Execute multiple enrichment queries in parallel. inet:ipv4=1.2.3.4 | tee -p { enrich.foo } { enrich.bar } { enrich.baz }
- name = 'tee'
- readonly = True
- class synapse.lib.storm.TreeCmd(runt, runtsafe)[source]
Bases:
CmdWalk elements of a tree using a recursive pivot.
Examples
# pivot upward yielding each FQDN inet:fqdn=www.vertex.link | tree { :domain -> inet:fqdn }
- name = 'tree'
- readonly = True
- class synapse.lib.storm.UniqCmd(runt, runtsafe)[source]
Bases:
CmdFilter nodes by their uniq iden values. When this is used a Storm pipeline, only the first instance of a given node is allowed through the pipeline.
A relative property or variable may also be specified, which will cause this command to only allow through the first node with a given value for that property or value rather than checking the node iden.
Examples
# Filter duplicate nodes after pivoting from inet:ipv4 nodes tagged with #badstuff #badstuff +inet:ipv4 ->* | uniq
# Unique inet:ipv4 nodes by their :asn property #badstuff +inet:ipv4 | uniq :asn
- name = 'uniq'
- readonly = True
- class synapse.lib.storm.ViewExecCmd(runt, runtsafe)[source]
Bases:
CmdExecute a storm query in a different view.
NOTE: Variables are passed through but nodes are not. The behavior of this command may be non-intuitive in relation to the way storm normally operates. For further information on behavior and limitations when using view.exec, reference the view.exec section of the Synapse User Guide: https://v.vtx.lk/view-exec.
Examples
// Move some tagged nodes to another view inet:fqdn#foo.bar $fqdn=$node.value() | view.exec 95d5f31f0fb414d2b00069d3b1ee64c6 { [ inet:fqdn=$fqdn ] }
- name = 'view.exec'
- readonly = True
synapse.lib.storm_format module
- class synapse.lib.storm_format.StormLexer(parser)[source]
Bases:
Lexer- get_tokens_unprocessed(text)[source]
This method should process the text and return an iterable of
(index, tokentype, value)tuples whereindexis the starting position of the token within the input text.It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness.
synapse.lib.stormctrl module
- exception synapse.lib.stormctrl.StormBreak(*args, **info)[source]
Bases:
StormLoopCtrl,StormCtrlFlow- statement = 'break'
- exception synapse.lib.stormctrl.StormContinue(*args, **info)[source]
Bases:
StormLoopCtrl,StormCtrlFlow- statement = 'continue'
- exception synapse.lib.stormctrl.StormCtrlFlow[source]
Bases:
ExceptionBase class all StormCtrlFlow exceptions derive from.
- exception synapse.lib.stormctrl.StormExit(*args, **info)[source]
Bases:
_SynErrMixin,StormCtrlFlow
- exception synapse.lib.stormctrl.StormGenrCtrl(*args, **info)[source]
Bases:
_SynErrMixin- statement = ''
- exception synapse.lib.stormctrl.StormLoopCtrl(*args, **info)[source]
Bases:
_SynErrMixin- statement = ''
- exception synapse.lib.stormctrl.StormReturn(item=None)[source]
Bases:
StormCtrlFlow
- exception synapse.lib.stormctrl.StormStop(*args, **info)[source]
Bases:
StormGenrCtrl,StormCtrlFlow- statement = 'stop'
synapse.lib.stormhttp module
- class synapse.lib.stormhttp.HttpResp(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a HTTP response.
- class synapse.lib.stormhttp.LibHttp(runt, name=())[source]
Bases:
LibA Storm Library exposing an HTTP client API.
For APIs that accept an ssl_opts argument, the dictionary may contain the following values:
({ 'verify': <bool> - Perform SSL/TLS verification. Is overridden by the ssl_verify argument. 'client_cert': <str> - PEM encoded full chain certificate for use in mTLS. 'client_key': <str> - PEM encoded key for use in mTLS. Alternatively, can be included in client_cert. 'ca_cert': <str> - A PEM encoded full chain CA certificate for use when verifying the request. })
For APIs that accept a proxy argument, the following values are supported:
(null): Deprecated - Use the proxy defined by the http:proxy configuration option if set. (true): Use the proxy defined by the http:proxy configuration option if set. (false): Do not use the proxy defined by the http:proxy configuration option if set. <str>: A proxy URL string.
- getObjLocals()[source]
Get the default list of key-value pairs which may be added to the object
.loclsdictionary.- Returns:
A key/value pairs.
- Return type:
dict
synapse.lib.stormsvc module
synapse.lib.stormtypes module
- class synapse.lib.stormtypes.Bool(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a boolean instance.
- class synapse.lib.stormtypes.Bytes(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a Bytes object.
- class synapse.lib.stormtypes.CmdOpts(valu, path=None)[source]
Bases:
DictA dictionary like object that holds a reference to a command options namespace. ( This allows late-evaluation of command arguments rather than forcing capture )
- class synapse.lib.stormtypes.CronJob(runt, cdef, path=None)[source]
Bases:
PrimImplements the Storm api for a cronjob instance.
- class synapse.lib.stormtypes.Dict(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a Dictionary object.
- class synapse.lib.stormtypes.Layer(runt, ldef, path=None)[source]
Bases:
PrimImplements the Storm api for a layer instance.
- class synapse.lib.stormtypes.Lib(runt, name=())[source]
Bases:
StormTypeA collection of storm methods under a name
- class synapse.lib.stormtypes.LibAxon(runt, name=())[source]
Bases:
LibA Storm library for interacting with the Cortex’s Axon.
For APIs that accept an ssl_opts argument, the dictionary may contain the following values:
({ 'verify': <bool> - Perform SSL/TLS verification. Is overridden by the ssl argument. 'client_cert': <str> - PEM encoded full chain certificate for use in mTLS. 'client_key': <str> - PEM encoded key for use in mTLS. Alternatively, can be included in client_cert. })
For APIs that accept a proxy argument, the following values are supported:
``(null)``: Deprecated - Use the proxy defined by the http:proxy configuration option if set. ``(true)``: Use the proxy defined by the http:proxy configuration option if set. ``(false)``: Do not use the proxy defined by the http:proxy configuration option if set. <str>: A proxy URL string.
- getObjLocals()[source]
Get the default list of key-value pairs which may be added to the object
.loclsdictionary.- Returns:
A key/value pairs.
- Return type:
dict
- class synapse.lib.stormtypes.LibBase(runt, name=())[source]
Bases:
LibThe Base Storm Library. This mainly contains utility functionality.
- class synapse.lib.stormtypes.LibBase64(runt, name=())[source]
Bases:
LibA Storm Library for encoding and decoding base64 data.
- class synapse.lib.stormtypes.LibBytes(runt, name=())[source]
Bases:
LibA Storm Library for interacting with bytes storage. This Library is deprecated; use
$lib.axon.*instead.
- class synapse.lib.stormtypes.LibCron(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Cron Jobs in the Cortex.
- class synapse.lib.stormtypes.LibCsv(runt, name=())[source]
Bases:
LibA Storm Library for interacting with csvtool.
- class synapse.lib.stormtypes.LibDict(runt, name=())[source]
Bases:
LibA Storm Library for interacting with dictionaries.
- class synapse.lib.stormtypes.LibDmon(runt, name=())[source]
Bases:
LibA Storm Library for interacting with StormDmons.
- class synapse.lib.stormtypes.LibExport(runt, name=())[source]
Bases:
LibA Storm Library for exporting data.
- class synapse.lib.stormtypes.LibFeed(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Cortex feed functions.
- class synapse.lib.stormtypes.LibGlobals(runt, name)[source]
Bases:
LibA Storm Library for interacting with global variables which are persistent across the Cortex.
- class synapse.lib.stormtypes.LibJsonStor(runt, name=())[source]
Bases:
LibImplements cortex JSON storage.
- class synapse.lib.stormtypes.LibLayer(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Layers in the Cortex.
- class synapse.lib.stormtypes.LibLift(runt, name=())[source]
Bases:
LibA Storm Library for interacting with lift helpers.
- class synapse.lib.stormtypes.LibPipe(runt, name=())[source]
Bases:
LibA Storm library for interacting with non-persistent queues.
- class synapse.lib.stormtypes.LibPkg(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Storm Packages.
- class synapse.lib.stormtypes.LibPs(runt, name=())[source]
Bases:
LibA Storm Library for interacting with running tasks on the Cortex.
- class synapse.lib.stormtypes.LibQueue(runt, name=())[source]
Bases:
LibA Storm Library for interacting with persistent Queues in the Cortex.
- class synapse.lib.stormtypes.LibRegx(runt, name=())[source]
Bases:
LibA Storm library for searching/matching with regular expressions.
- class synapse.lib.stormtypes.LibService(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Storm Services.
- class synapse.lib.stormtypes.LibStr(runt, name=())[source]
Bases:
LibA Storm Library for interacting with strings.
- class synapse.lib.stormtypes.LibTags(runt, name=())[source]
Bases:
LibStorm utility functions for tags.
- class synapse.lib.stormtypes.LibTelepath(runt, name=())[source]
Bases:
LibA Storm Library for making Telepath connections to remote services.
- class synapse.lib.stormtypes.LibTime(runt, name=())[source]
Bases:
LibA Storm Library for interacting with timestamps.
- class synapse.lib.stormtypes.LibTrigger(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Triggers in the Cortex.
- class synapse.lib.stormtypes.LibVars(runt, name=())[source]
Bases:
LibA Storm Library for interacting with runtime variables.
- class synapse.lib.stormtypes.LibView(runt, name=())[source]
Bases:
LibA Storm Library for interacting with Views in the Cortex.
- class synapse.lib.stormtypes.List(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a List instance.
- class synapse.lib.stormtypes.Node(node, path=None)[source]
Bases:
PrimImplements the Storm api for a node instance.
- class synapse.lib.stormtypes.NodeData(node, path=None)[source]
Bases:
PrimA Storm Primitive representing the NodeData stored for a Node.
- class synapse.lib.stormtypes.NodeProps(node, path=None)[source]
Bases:
PrimA Storm Primitive representing the properties on a Node.
- getObjLocals()[source]
Get the default list of key-value pairs which may be added to the object
.loclsdictionary.- Returns:
A key/value pairs.
- Return type:
dict
- class synapse.lib.stormtypes.Number(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a Number instance.
Storm Numbers are high precision fixed point decimals corresponding to the the hugenum storage type.
- class synapse.lib.stormtypes.Path(node, path=None)[source]
Bases:
PrimImplements the Storm API for the Path object.
- class synapse.lib.stormtypes.PathMeta(path)[source]
Bases:
PrimPut the storm deref/setitem/iter convention on top of path meta information.
- class synapse.lib.stormtypes.PathVars(path)[source]
Bases:
PrimPut the storm deref/setitem/iter convention on top of path variables.
- class synapse.lib.stormtypes.Pipe(runt, size)[source]
Bases:
StormTypeA Storm Pipe provides fast ephemeral queues.
- class synapse.lib.stormtypes.Prim(valu, path=None)[source]
Bases:
StormTypeThe base type for all Storm primitive values.
- class synapse.lib.stormtypes.Proxy(runt, proxy, path=None)[source]
Bases:
StormTypeImplements the Storm API for a Telepath proxy.
These can be created via
$lib.telepath.open(). Storm Service objects are also Telepath proxy objects.Methods called off of these objects are executed like regular Telepath RMI calls.
An example of calling a method which returns data:
$prox = $lib.telepath.open($url) $result = $prox.doWork($data) return ( $result )
An example of calling a method which is a generator:
$prox = $lib.telepath.open($url) for $item in $prox.genrStuff($data) { $doStuff($item) }
- class synapse.lib.stormtypes.ProxyGenrMethod(meth, path=None)[source]
Bases:
StormTypeImplements the generator methods for the telepath:proxy.
An example of calling a method which is a generator:
$prox = $lib.telepath.open($url) for $item in $prox.genrStuff($data) { $doStuff($item) }
- class synapse.lib.stormtypes.ProxyMethod(runt, meth, path=None)[source]
Bases:
StormTypeImplements the call methods for the telepath:proxy.
An example of calling a method which returns data:
$prox = $lib.telepath.open($url) $result = $prox.doWork($data) $doStuff($result)
- class synapse.lib.stormtypes.Query(text, varz, runt, path=None)[source]
Bases:
PrimA storm primitive representing an embedded query.
- class synapse.lib.stormtypes.Queue(runt, name, info)[source]
Bases:
StormTypeA StormLib API instance of a named channel in the Cortex multiqueue.
- class synapse.lib.stormtypes.Set(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a Set object.
- class synapse.lib.stormtypes.StormHiveDict(runt, info)[source]
Bases:
PrimA Storm Primitive representing a HiveDict.
- class synapse.lib.stormtypes.StormType(path=None)[source]
Bases:
objectThe base type for storm runtime value objects.
- class synapse.lib.stormtypes.StormTypesRegistry[source]
Bases:
object- base_undefined_types = ('any', 'int', 'lib', 'null', 'time', 'prim', 'undef', 'float', 'generator')
- known_types = {'aha:pool', 'auth:gate', 'auth:role', 'auth:user', 'auth:user:json', 'auth:user:profile', 'auth:user:vars', 'boolean', 'bytes', 'cache:fixed', 'cmdopts', 'cronjob', 'dict', 'hive:dict', 'http:api', 'http:api:methods', 'http:api:perms', 'http:api:request', 'http:api:request:headers', 'http:api:vars', 'inet:http:oauth:v1:client', 'inet:http:resp', 'inet:http:socket', 'inet:imap:server', 'inet:smtp:message', 'json:schema', 'layer', 'list', 'model:form', 'model:property', 'model:tagprop', 'model:type', 'node', 'node:data', 'node:path', 'node:path:meta', 'node:path:vars', 'node:props', 'number', 'pipe', 'proj:comment', 'proj:comments', 'proj:epic', 'proj:epics', 'proj:project', 'proj:sprint', 'proj:sprints', 'proj:ticket', 'proj:tickets', 'queue', 'random', 'set', 'spooled:set', 'stat:tally', 'stix:bundle', 'storm:query', 'str', 'tabular:printer', 'telepath:proxy', 'telepath:proxy:genrmethod', 'telepath:proxy:method', 'text', 'trigger', 'user:profile:dict', 'user:vars:dict', 'vault', 'vault:data', 'view', 'xml:element'}
- rtypes = {}
- undefined_types = {'any', 'float', 'generator', 'int', 'lib', 'null', 'prim', 'time', 'undef'}
- class synapse.lib.stormtypes.Str(valu, path=None)[source]
Bases:
PrimImplements the Storm API for a String object.
- class synapse.lib.stormtypes.Text(valu, path=None)[source]
Bases:
PrimA mutable text type for simple text construction.
- class synapse.lib.stormtypes.Trigger(runt, tdef)[source]
Bases:
PrimImplements the Storm API for a Trigger.
- class synapse.lib.stormtypes.View(runt, vdef, path=None)[source]
Bases:
PrimImplements the Storm api for a View instance.
- synapse.lib.stormtypes.getCallSig(func) Signature[source]
Get the callsig of a function, stripping self if present.
- async synapse.lib.stormtypes.kwarg_format(_text, **kwargs)[source]
Replaces instances curly-braced argument names in text with their values
- async synapse.lib.stormtypes.resolveAxonProxyArg(valu)[source]
Resolve a proxy value to the kwarg to set for an Axon HTTP call.
- Parameters:
valu (str|null|bool) – The proxy value.
- Returns:
A retn tuple where the proxy kwarg should not be set if ok=False, otherwise a proxy URL or None.
- Return type:
tuple
- async synapse.lib.stormtypes.resolveCoreProxyUrl(valu)[source]
Resolve a proxy value to a proxy URL.
- Parameters:
valu (str|None|bool) – The proxy value.
- Returns:
A proxy URL string or None.
- Return type:
(str|None)
- async synapse.lib.stormtypes.totype(valu, basetypes=False) str[source]
Convert a value to its Storm type string.
- Parameters:
valu – The object to check.
basetypes (bool) – If True, return the base Python class name as a fallback.
- Returns:
The type name.
- Return type:
str
- Raises:
StormRuntimeError – If the valu does not resolve to a known type and basetypes=False.
synapse.lib.stormwhois module
synapse.lib.structlog module
- class synapse.lib.structlog.JsonFormatter(*args, **kwargs)[source]
Bases:
Formatter- format(record: LogRecord)[source]
Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
synapse.lib.task module
- class synapse.lib.task.Task[source]
Bases:
BaseThe synapse Task object implements concepts similar to process trees for asyncio.Task instances.
- async synapse.lib.task.executor(func, *args, **kwargs)[source]
Execute a function in an executor thread.
- Parameters:
todo ((func,args,kwargs)) – A todo tuple.
- synapse.lib.task.syntask(task)[source]
Return the synapse task associated with the provided asyncio task.
- synapse.lib.task.vardefault(name, func)[source]
Add a default constructor for a particular task-local variable
All future calls to taskVarGet with the same name will return the result of calling func
- synapse.lib.task.varget(name, defval=None, task=None)[source]
Access a task local variable by name
- Precondition:
If task is None, this must be called from task context
synapse.lib.thishost module
synapse.lib.thisplat module
synapse.lib.threads module
synapse.lib.time module
Time related utilities for synapse “epoch millis” time values.
- synapse.lib.time.parse(text, base=None, chop=False)[source]
Parse a time string into an epoch millis value.
- Parameters:
text (str) – Time string to parse
base (int or None) – Milliseconds to offset the time from
chop (bool) – Whether to chop the digit-only string to 17 chars
- Returns:
Epoch milliseconds
- Return type:
int
- synapse.lib.time.parsetz(text)[source]
Parse timezone from time string, with UTC as the default.
- Parameters:
text (str) – Time string
- Returns:
A tuple of text with tz chars removed and base milliseconds to offset time.
- Return type:
tuple
- synapse.lib.time.repr(tick, pack=False)[source]
Return a date string for an epoch-millis timestamp.
- Parameters:
tick (int) – The timestamp in milliseconds since the epoch.
- Returns:
A date time string
- Return type:
(str)
synapse.lib.trigger module
- class synapse.lib.trigger.Triggers(view)[source]
Bases:
objectManages “triggers”, conditions where changes in data result in new storm queries being executed.
Note
These methods should not be called directly under normal circumstances. Use the owning “View” object to ensure that mirrors/clusters members get the same changes.
synapse.lib.types module
- class synapse.lib.types.Bool(modl, name, info, opts)[source]
Bases:
Type- repr(valu)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 2
- class synapse.lib.types.Comp(modl, name, info, opts)[source]
Bases:
Type- getCompOffs(name)[source]
If this type is a compound, return the field offset for the given property name or None.
- repr(valu)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 13
- class synapse.lib.types.Data(modl, name, info, opts)[source]
Bases:
Type- norm(valu)[source]
Normalize the value for a given type.
- Parameters:
valu (obj) – The value to normalize.
- Returns:
The normalized valu, info tuple.
- Return type:
((obj,dict))
Notes
- The info dictionary uses the following key conventions:
subs (dict): The normalized sub-fields as name: valu entries.
- stortype: int = 13
- class synapse.lib.types.Duration(modl, name, info, opts)[source]
Bases:
IntBase- repr(valu)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 5
- class synapse.lib.types.Edge(modl, name, info, opts)[source]
Bases:
Type- getCompOffs(name)[source]
If this type is a compound, return the field offset for the given property name or None.
- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 13
- class synapse.lib.types.FieldHelper(modl, tname, fields)[source]
Bases:
defaultdictHelper for Comp types. Performs Type lookup/creation upon first use.
- class synapse.lib.types.Float(modl, name, info, opts)[source]
Bases:
Type- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 22
- class synapse.lib.types.HugeNum(modl, name, info, opts)[source]
Bases:
Type- norm(valu)[source]
Normalize the value for a given type.
- Parameters:
valu (obj) – The value to normalize.
- Returns:
The normalized valu, info tuple.
- Return type:
((obj,dict))
Notes
- The info dictionary uses the following key conventions:
subs (dict): The normalized sub-fields as name: valu entries.
- stortype: int = 23
- class synapse.lib.types.Int(modl, name, info, opts)[source]
Bases:
IntBase
- class synapse.lib.types.Ival(modl, name, info, opts)[source]
Bases:
TypeAn interval, i.e. a range, of times
- merge(oldv, newv)[source]
Allow types to “merge” data from two sources based on value precedence.
- Parameters:
valu (object) – The current value.
newv (object) – The updated value.
- Returns:
The merged value.
- Return type:
(object)
- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 12
- class synapse.lib.types.Loc(modl, name, info, opts)[source]
Bases:
Type- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 15
- class synapse.lib.types.Ndef(modl, name, info, opts)[source]
Bases:
Type- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 25
- class synapse.lib.types.Range(modl, name, info, opts)[source]
Bases:
Type- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 13
- class synapse.lib.types.Str(modl, name, info, opts)[source]
Bases:
Type- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 1
- class synapse.lib.types.Time(modl, name, info, opts)[source]
Bases:
IntBase- getTickTock(vals)[source]
Get a tick, tock time pair.
- Parameters:
vals (list) – A pair of values to norm.
- Returns:
A ordered pair of integers.
- Return type:
(int, int)
- merge(oldv, newv)[source]
Allow types to “merge” data from two sources based on value precedence.
- Parameters:
valu (object) – The current value.
newv (object) – The updated value.
- Returns:
The merged value.
- Return type:
(object)
- repr(valu)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 11
- class synapse.lib.types.TimeEdge(modl, name, info, opts)[source]
Bases:
Edge- getCompOffs(name)[source]
If this type is a compound, return the field offset for the given property name or None.
- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- stortype: int = 13
- class synapse.lib.types.Type(modl, name, info, opts)[source]
Bases:
object- clone(opts)[source]
Create a new instance of this type with the specified options.
- Parameters:
opts (dict) – The type specific options for the new instance.
- extend(name, opts, info)[source]
Extend this type to construct a sub-type.
- Parameters:
name (str) – The name of the new sub-type.
opts (dict) – The type options for the sub-type.
info (dict) – The type info for the sub-type.
- Returns:
A new sub-type instance.
- Return type:
(synapse.types.Type)
- getCompOffs(name)[source]
If this type is a compound, return the field offset for the given property name or None.
- isarray = False
- merge(oldv, newv)[source]
Allow types to “merge” data from two sources based on value precedence.
- Parameters:
valu (object) – The current value.
newv (object) – The updated value.
- Returns:
The merged value.
- Return type:
(object)
- norm(valu)[source]
Normalize the value for a given type.
- Parameters:
valu (obj) – The value to normalize.
- Returns:
The normalized valu, info tuple.
- Return type:
((obj,dict))
Notes
- The info dictionary uses the following key conventions:
subs (dict): The normalized sub-fields as name: valu entries.
- repr(norm)[source]
Return a printable representation for the value. This may return a string or a tuple of values for display purposes.
- setCmprCtor(name, func)[source]
Set a comparator ctor for a given named comparison operation.
- Parameters:
name (str) – Name of the comparison operation.
func – Function which returns a comparator.
Notes
Comparator ctors should expect to get the right-hand-side of the comparison as their argument, and the returned function should expect to get the left hand side of the comparison and return a boolean from there.
- setNormFunc(typo, func)[source]
Register a normalizer function for a given python type.
- Parameters:
typo (type) – A python type/class to normalize.
func (function) – A callback which normalizes a python value.
- stortype: int = None
synapse.lib.urlhelp module
- synapse.lib.urlhelp.chopurl(url)[source]
A sane “stand alone” url parser.
Example
info = chopurl(url)
- synapse.lib.urlhelp.sanitizeUrl(url)[source]
Returns a URL with the password (if present) replaced with
****RFC 3986 3.2.1 ‘Applications should not render as clear text any data after the first colon (“:”) character found within a userinfo subcomponent unless the data after the colon is the empty string (indicating no password)’
Essentially, replace everything between the 2nd colon (if it exists) and the first succeeding at sign. Return the original string otherwise.
Note: this depends on this being a reasonably-well formatted URI that starts with a scheme (e.g. http) and ‘//:’ Failure of this condition yields the original string.
synapse.lib.version module
Synapse utilites for dealing with Semvar versioning. This includes the Synapse version information.
- synapse.lib.version.fmtVersion(*vsnparts)[source]
Join a string of parts together with a . separator.
- Parameters:
*vsnparts
Returns:
- synapse.lib.version.matches(vers, cmprvers)[source]
Check if a version string matches a version comparison string.
- synapse.lib.version.packVersion(major, minor=0, patch=0)[source]
Pack a set of major/minor/patch integers into a single integer for storage.
- Parameters:
major (int) – Major version level integer.
minor (int) – Minor version level integer.
patch (int) – Patch version level integer.
- Returns:
System normalized integer value to represent a software version.
- Return type:
int
- synapse.lib.version.parseSemver(text)[source]
Parse a Semantic Version string into is component parts.
- Parameters:
text (str) – A text string to parse into semver components. This string has whitespace and leading ‘v’
it. (characters stripped off of)
Examples
Parse a string into it semvar parts:
parts = parseSemver('v1.2.3')
- Returns:
The dictionary will contain the keys ‘major’, ‘minor’ and ‘patch’ pointing to integer values. The dictionary may also contain keys for ‘build’ and ‘pre’ information if that data is parsed out of a semver string. None is returned if the string is not a valid Semver string.
- Return type:
dict
- synapse.lib.version.parseVersionParts(text, seps=('.', '-', '_', '+'))[source]
Extract a list of major/minor/version integer strings from a string.
- Parameters:
text (str) – String to parse
seps (tuple) – A tuple or list of separators to use when parsing the version string.
Examples
Parse a simple version string into a major and minor parts:
parts = parseVersionParts('1.2')
Parse a complex version string into a major and minor parts:
parts = parseVersionParts('wowsoft_1.2')
Parse a simple version string into a major, minor and patch parts. Parts after the “3.” are dropped from the results:
parts = parseVersionParts('1.2.3.4.5')
Notes
This attempts to brute force out integers from the version string by stripping any leading ascii letters and part separators, and then regexing out numeric parts optionally followed by part separators. It will stop at the first mixed-character part encountered. For example, “1.2-3a” would only parse out the “1” and “2” from the string.
- Returns:
Either a empty dictionary or dictionary containing up to three keys, ‘major’, ‘minor’ and ‘patch’.
- Return type:
dict
- synapse.lib.version.reqVersion(valu, reqver, exc=<class 'synapse.exc.BadVersion'>, mesg='Provided version does not match required version.')[source]
Require a given version tuple is valid for a given requirements string.
- Parameters:
Optional[Tuple[int (valu) – Major, minor and patch value to check.
int – Major, minor and patch value to check.
int]] – Major, minor and patch value to check.
reqver (str) – A requirements version string.
exc (s_exc.SynErr) – The synerr class to raise.
mesg (str) – The message to pass in the exception.
- Returns:
If the value is in bounds of minver and maxver.
- Return type:
None
- Raises:
s_exc.BadVersion – If a precondition is incorrect or a version value is out of bounds.
- synapse.lib.version.unpackVersion(ver)[source]
Unpack a system normalized integer representing a softare version into its component parts.
- Parameters:
ver (int) – System normalized integer value to unpack into a tuple.
- Returns:
A tuple containing the major, minor and patch values shifted out of the integer.
- Return type:
(int, int, int)
synapse.lib.view module
- class synapse.lib.view.View[source]
Bases:
PusherA view represents a cortex as seen from a specific set of layers.
The view class is used to implement Copy-On-Write layers as well as interact with a subset of the layers configured in a Cortex.
- async addNodeEdits(edits, meta)[source]
A telepath compatible way to apply node edits to a view.
NOTE: This does cause trigger execution.
- async delete()[source]
Delete the metadata for this view.
Note: this does not delete any layer storage.
- async detach()[source]
Detach the view from its parent but do not change the layers. ( this is not reversible! )
- async fork(ldef=None, vdef=None)[source]
Make a new view inheriting from this view with the same layers and a new write layer on top
- Parameters:
ldef – layer parameter dict
vdef – view parameter dict
cortex.addLayer (Passed through to)
- Returns:
new view object, with an iden the same as the new write layer iden
- init2()[source]
We have a second round of initialization so the views can get a handle to their parents which might not be initialized yet
- async insertParentFork(useriden, name=None)[source]
Insert a new View between a forked View and its parent.
- Returns:
New view definition with the same perms as the current fork.
- async merge(useriden=None, force=False)[source]
Merge this view into its parent. All changes made to this view will be applied to the parent. Parent’s triggers will be run.
- async mergeAllowed(user=None, force=False)[source]
Check whether a user can merge a view into its parent.
NOTE: This API may not be used to check for merges based on quorum votes.
- async mergeStormIface(name, todo)[source]
Allow an interface which specifies a generator use case to yield (priority, value) tuples and merge results from multiple generators yielded in ascending priority order.
- async setLayers(layers)[source]
Set the view layers from a list of idens. NOTE: view layers are stored “top down” (the write layer is self.layers[0])
- async classmethod snapctor(*args, **kwargs)
- async storm(text, opts=None)[source]
Evaluate a storm query and yield result messages. :Yields: ((str,dict)) – Storm messages.