synapse.lib package

Subpackages

Submodules

synapse.lib.agenda module

class synapse.lib.agenda.Agenda[source]

Bases: Base

Organize and execute all the scheduled storm queries in a cortex.

async add(cdef)[source]

Persistently adds an appointment

Parameters:: cdef (dict) – Dictionary containing the Cron definition.

Notes

The cron definition may contain the following keys:

creator (str)
Iden of the creating user.

iden (str)
Iden of the appointment.

storm (str)
The Storm query to run.

reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…])
One or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.

incunit (Union[None, TimeUnit])
The unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.

incvals (Union[None, int, Iterable[int])
Count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)

If the values for req and incvals are both lists, all combinations of all values (the product) are used.

Returns:: Packed appointment definition

async clearRunningStatus()[source]: Used for clearing the running state at startup or change of leadership.

async delete(iden)[source]: Delete an appointment

async disable(iden)[source]

async enable(iden)[source]

async get(iden)[source]

list()[source]

async mod(iden, query)[source]: Change the query of an appointment

async move(croniden, viewiden)[source]: Move a cronjob from one view to another

async runloop()[source]: Task loop to issue query tasks at the right times.

class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)[source]

Bases: object

Represents a single element of a single combination of an appointment

nexttime(lastts)[source]: Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

pack()[source]: Make ApptRec json/msgpack-friendly

classmethod unpack(val)[source]: Convert from json/msgpack-friendly

class synapse.lib.agenda.TimeUnit(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: IntEnum

Unit of time that recurring and required parts of appointments are made of

DAY = 5

DAYOFMONTH = 3

DAYOFWEEK = 4

HOUR = 6

MINUTE = 7

MONTH = 2

NOW = 8

YEAR = 1

classmethod fromString(s)[source]

synapse.lib.aha module

class synapse.lib.aha.AhaApi[source]

Bases: CellApi

addAhaPool(name, info)[source]

addAhaPoolSvc(poolname, svcname, info)[source]

async addAhaSvc(name, info, network=None)[source]

Register a service with the AHA discovery server.

NOTE: In order for the service to remain marked “up” a caller: must maintain the telepath link.

addAhaSvcProv(name, provinfo=None)[source]: Provision the given relative service name within the configured network name.

addAhaUserEnroll(name, userinfo=None, again=False)[source]: Create and return a one-time user enroll key.

clearAhaSvcProvs()[source]: Remove all unused service provisioning values.

clearAhaUserEnrolls()[source]: Remove all unused user enrollment provisioning values.

delAhaPool(name)[source]

delAhaPoolSvc(poolname, svcname)[source]

async delAhaSvc(name, network=None)[source]: Remove an AHA service entry.

delAhaSvcProv(iden)[source]: Remove a previously added provisioning entry by iden.

delAhaUserEnroll(iden)[source]: Remove a previously added enrollment entry by iden.

async genCaCert(network)[source]

async getAhaPool(name)[source]

async getAhaPools()[source]

async getAhaSvc(name, filters=None)[source]: Return an AHA service description dictionary for a service name.

async getAhaSvcMirrors(name)[source]: Return list of AHA svcinfo dictionaries for mirrors of a service.

async getAhaSvcs(network=None)[source]

Yield AHA svcinfo dictionaries.

Parameters:: network (str) – Optionally specify a network to filter on.

async getAhaUrls()[source]

async getCaCert(network)[source]

async iterPoolTopo(name)[source]

async modAhaSvcInfo(name, svcinfo)[source]

async signHostCsr(csrtext, signas=None, sans=None)[source]

async signUserCsr(csrtext, signas=None)[source]

class synapse.lib.aha.AhaCell[source]

Bases: Cell

async addAhaPool(name, info)[source]

async addAhaPoolSvc(poolname, svcname, info)[source]

async addAhaSvc(name, info, network=None)[source]

async addAhaSvcProv(name, provinfo=None)[source]

async addAhaUserEnroll(name, userinfo=None, again=False)[source]

cellapi: alias of AhaApi

async clearAhaSvcProvs()[source]

async clearAhaUserEnrolls()[source]

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:parse:proxy:remoteip': {'default': False, 'description': 'Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.', 'type': 'boolean'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'max:users': {'default': 0, 'description': 'Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).', 'minimum': 0, 'type': 'integer'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': False, 'hidedocs': False, 'type': ['string', 'null']}, 'nexslog:async': {'default': True, 'description': 'Set to false to disable async memory mapping of the nexus change log.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.', 'type': 'boolean'}}

confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'provision:listen': {'description': 'A telepath URL for the AHA provisioning listener.', 'type': ['string', 'null']}}

async delAhaPool(name)[source]

async delAhaPoolSvc(poolname, svcname)[source]

async delAhaSvc(name, network=None)[source]

async delAhaSvcProv(iden)[source]

async delAhaUserEnroll(iden)[source]

async genCaCert(network)[source]

async getAhaPool(name)[source]

async getAhaPools()[source]

async getAhaSvc(name, filters=None)[source]

async getAhaSvcMirrors(iden, network=None)[source]

async getAhaSvcProv(iden)[source]

async getAhaSvcs(network=None)[source]

async getAhaUserEnroll(iden)[source]

async getCaCert(network)[source]

classmethod getEnvPrefix()[source]: Get a list of envar prefixes for config resolution.

async initServiceNetwork()[source]

async initServiceRuntime()[source]

async initServiceStorage()[source]

async iterPoolTopo(name)[source]

async modAhaSvcInfo(name, svcinfo)[source]

async saveCaCert(name, cakey, cacert)[source]

async saveHostCert(name, hostkey, hostcert)[source]

async saveUserCert(name, userkey, usercert)[source]

async setAhaSvcDown(name, linkiden, network=None)[source]

async signHostCsr(csrtext, signas=None, sans=None)[source]

async signUserCsr(csrtext, signas=None)[source]

class synapse.lib.aha.AhaProvisionServiceV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.aha.AhaServicesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.aha.EnrollApi(aha, userinfo)[source]

Bases: object

async getCaCert()[source]

async getUserInfo()[source]

async signUserCsr(byts)[source]

class synapse.lib.aha.ProvApi(aha, provinfo)[source]

Bases: object

async getCaCert()[source]

async getProvInfo()[source]

async signHostCsr(byts)[source]

async signUserCsr(byts)[source]

class synapse.lib.aha.ProvDmon[source]: Bases: Daemon

synapse.lib.ast module

class synapse.lib.ast.AbsPropCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AndCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]

class synapse.lib.ast.ArgvQuery(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

isRuntSafe(runt)[source]

runtopaque = True

validate(runt)[source]

class synapse.lib.ast.ArrayCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AstNode(astinfo, kids=())[source]

Bases: object

Base class for all nodes in the Storm abstract syntax tree.

addExcInfo(exc)[source]

addKid(astn)[source]

format(depth=0)[source]

getAstText()[source]

getPosInfo()[source]

getRuntVars(runt)[source]

hasAstClass(clss)[source]

hasVarName(name)[source]

init(core)[source]

isRuntSafe(runt)[source]

isRuntSafeAtom(runt)[source]

iterright()[source]: Yield “rightward” siblings until None.

optimize()[source]

prepare()[source]

repr()[source]

reqRuntSafe(runt, mesg)[source]

runtopaque = False

sibling(offs=1)[source]: Return sibling node by relative offset from self.

validate(runt)[source]

class synapse.lib.ast.Bool(astinfo, valu, kids=())[source]: Bases: Const

class synapse.lib.ast.BreakOper(astinfo, kids=())[source]

Bases: AstNode

async run(runt, genr)[source]

class synapse.lib.ast.CallArgs(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.CallKwarg(astinfo, kids=())[source]: Bases: CallArgs

class synapse.lib.ast.CallKwargs(astinfo, kids=())[source]: Bases: CallArgs

class synapse.lib.ast.CaseEntry(astinfo, kids=())[source]: Bases: AstNode

class synapse.lib.ast.CatchBlock(astinfo, kids=())[source]

Bases: AstNode

async catches(name, runt, path=None)[source]

errvar()[source]

getRuntVars(runt)[source]

async run(runt, genr)[source]

class synapse.lib.ast.CmdOper(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

class synapse.lib.ast.Cmpr(astinfo, valu, kids=())[source]: Bases: Const

class synapse.lib.ast.Cond(astinfo, kids=())[source]

Bases: Value

A condition that is evaluated to filter nodes.

class synapse.lib.ast.Const(astinfo, valu, kids=())[source]

Bases: Value

async compute(runt, path)[source]

isRuntSafe(runt)[source]

repr()[source]

value()[source]

class synapse.lib.ast.ContinueOper(astinfo, kids=())[source]

Bases: AstNode

async run(runt, genr)[source]

class synapse.lib.ast.DollarExpr(astinfo, kids=())[source]

Bases: Value

Top level node for $(…) expressions

async compute(runt, path)[source]

class synapse.lib.ast.Edit(astinfo, kids=())[source]: Bases: Oper

class synapse.lib.ast.EditEdgeAdd(astinfo, kids=(), n2=False)[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditEdgeDel(astinfo, kids=(), n2=False)[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditNodeAdd(astinfo, kids=())[source]

Bases: Edit

async addFromPath(form, runt, path)[source]

Add a node using the context from path.

NOTE: CALLER MUST CHECK PERMS

prepare()[source]

async run(runt, genr)[source]

class synapse.lib.ast.EditParens(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditPropDel(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditPropSet(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditTagAdd(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditTagDel(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EditTagPropDel(astinfo, kids=())[source]

Bases: Edit

[ -#foo.bar:baz ]

async run(runt, genr)[source]

class synapse.lib.ast.EditTagPropSet(astinfo, kids=())[source]

Bases: Edit

[ #foo.bar:baz=10 ]

async run(runt, genr)[source]

class synapse.lib.ast.EditUnivDel(astinfo, kids=())[source]

Bases: Edit

async run(runt, genr)[source]

class synapse.lib.ast.EmbedQuery(astinfo, valu, kids=())[source]

Bases: Const

async compute(runt, path)[source]

getRuntVars(runt)[source]

hasVarName(name)[source]

runtopaque = True

validate(runt)[source]

class synapse.lib.ast.Emit(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

class synapse.lib.ast.EmptyBlock(astinfo, kids=())[source]

Bases: AstNode

An AST node that only runs if there are not inbound nodes in the pipeline. It is capable of yielding nodes into the pipeline.

Example

Using an empty block:

empty {
    // the pipeline is empty so this block will execute
}

[foo:bar=*]
empty {
    // there is a node in the pipeline so this block will not run
}

async run(runt, genr)[source]

class synapse.lib.ast.ExprAndNode(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.ExprDict(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.ExprList(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.ExprNode(astinfo, kids=())[source]

Bases: Value

A binary (i.e. two argument) expression node

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.ExprOrNode(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.FiltByArray(astinfo, kids=())[source]

Bases: FiltOper

+:foo*[^=visi]

class synapse.lib.ast.FiltOper(astinfo, kids=())[source]

Bases: Oper

async getLiftHints(runt, path)[source]

async run(runt, genr)[source]

class synapse.lib.ast.FiniBlock(astinfo, kids=())[source]

Bases: AstNode

An AST node that runs only once after all nodes have been consumed.

Example

Using a fini block:

fini {
   // stuff here runs *once* after the last node yield (even if there are no nodes)
}

Notes

A fini block must be runtsafe.

async run(runt, genr)[source]

class synapse.lib.ast.ForLoop(astinfo, kids=())[source]

Bases: Oper

getRuntVars(runt)[source]

async run(runt, genr)[source]

class synapse.lib.ast.FormName(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.FormPivot(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

-> foo:bar

buildgenr(runt, name)[source]

pivogenr(runt, prop)[source]

async run(runt, genr)[source]

class synapse.lib.ast.FormTagProp(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.FormatString(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.FuncArgs(astinfo, kids=())[source]

Bases: AstNode

Represents the function arguments in a function definition

async compute(runt, path)[source]

class synapse.lib.ast.FuncCall(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.Function(astinfo, kids=())[source]

Bases: AstNode

( name, args, body )

// use args/kwargs syntax function bar(x, v=$(30)) { }

# we auto-detect the behavior of the target function

# return a value function bar(x, y) { return ($(x + y)) }

# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }

$foo = $bar(10, v=20)

async callfunc(runt, argdefs, args, kwargs)[source]

Execute a function call using the given runtime.

This function may return a value / generator / async generator

getRuntVars(runt)[source]

isRuntSafe(runt)[source]

prepare()[source]

async run(runt, genr)[source]

runtopaque = True

validate(runt)[source]

class synapse.lib.ast.HasAbsPropCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.HasRelPropCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]

async hasProp(node, runt, name)[source]

class synapse.lib.ast.HasTagPropCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.IfClause(astinfo, kids=())[source]: Bases: AstNode

class synapse.lib.ast.IfStmt(astinfo, kids=())[source]

Bases: Oper

prepare()[source]

async run(runt, genr)[source]

class synapse.lib.ast.InitBlock(astinfo, kids=())[source]

Bases: AstNode

An AST node that runs only once before yielding nodes.

Example

Using a init block:

init {
    // stuff here runs *once* before the first node yield (even if there are no nodes)
}

async run(runt, genr)[source]

class synapse.lib.ast.LiftByArray(astinfo, kids=())[source]

Bases: LiftOper

:prop*[range=(200, 400)]

async lift(runt, path)[source]

class synapse.lib.ast.LiftFormTag(astinfo, kids=())[source]

Bases: LiftOper

async lift(runt, path)[source]

class synapse.lib.ast.LiftFormTagProp(astinfo, kids=())[source]

Bases: LiftOper

hehe:haha#foo.bar:baz [ = x ]

async lift(runt, path)[source]

class synapse.lib.ast.LiftOper(astinfo, kids=())[source]

Bases: Oper

async lift(runt, path)[source]

reverseLift(astinfo)[source]

async run(runt, genr)[source]

class synapse.lib.ast.LiftProp(astinfo, kids=())[source]

Bases: LiftOper

async getRightHints(runt, path)[source]

async lift(runt, path)[source]

async proplift(prop, runt, path)[source]

class synapse.lib.ast.LiftPropBy(astinfo, kids=())[source]

Bases: LiftOper

async lift(runt, path)[source]

class synapse.lib.ast.LiftTag(astinfo, kids=())[source]

Bases: LiftOper

async lift(runt, path)[source]

class synapse.lib.ast.LiftTagProp(astinfo, kids=())[source]

Bases: LiftOper

#foo.bar:baz [ = x ]

async lift(runt, path)[source]

class synapse.lib.ast.LiftTagTag(astinfo, kids=())[source]

Bases: LiftOper

##foo.bar

async lift(runt, path)[source]

class synapse.lib.ast.List(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

prepare()[source]

repr()[source]

class synapse.lib.ast.LookList(astinfo, kids=())[source]: Bases: AstNode

class synapse.lib.ast.Lookup(astinfo, kids, autoadd=False)[source]

Bases: Query

When storm input mode is “lookup”

async run(runt, genr)[source]

class synapse.lib.ast.N1Walk(astinfo, kids=(), isjoin=False)[source]

Bases: Oper

buildfilter(runt, destforms, cmpr)[source]

repr()[source]

async run(runt, genr)[source]

async walkNodeEdges(runt, node, verb=None)[source]

class synapse.lib.ast.N1WalkNPivo(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOut

async run(runt, genr)[source]

class synapse.lib.ast.N2Walk(astinfo, kids=(), isjoin=False)[source]

Bases: N1Walk

async walkNodeEdges(runt, node, verb=None)[source]

class synapse.lib.ast.N2WalkNPivo(astinfo, kids=(), isjoin=False)[source]

Bases: PivotIn

async run(runt, genr)[source]

class synapse.lib.ast.NotCond(astinfo, kids=())[source]

Bases: Cond

not <cond>

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.Oper(astinfo, kids=())[source]: Bases: AstNode

class synapse.lib.ast.OrCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.PivotIn(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

<- *

async getPivsIn(runt, node, path)[source]

async run(runt, genr)[source]

class synapse.lib.ast.PivotInFrom(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

<- foo:edge

async run(runt, genr)[source]

class synapse.lib.ast.PivotOper(astinfo, kids=(), isjoin=False)[source]

Bases: Oper

repr()[source]

class synapse.lib.ast.PivotOut(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

-> *

async getPivsOut(runt, node, path)[source]

async run(runt, genr)[source]

class synapse.lib.ast.PivotToTags(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present

async run(runt, genr)[source]

class synapse.lib.ast.PropName(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.PropPivot(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

:foo -> bar:foo

buildgenr(runt, name)[source]

pivogenr(runt, prop)[source]

async run(runt, genr)[source]

class synapse.lib.ast.PropPivotOut(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

:prop -> *

async run(runt, genr)[source]

class synapse.lib.ast.PropValue(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

async getPropAndValu(runt, path)[source]

isRuntSafe(runt)[source]

isRuntSafeAtom(runt)[source]

prepare()[source]

class synapse.lib.ast.Query(astinfo, kids=())[source]

Bases: AstNode

async iterNodePaths(runt, genr=None)[source]

async run(runt, genr)[source]

class synapse.lib.ast.RawPivot(astinfo, kids=(), isjoin=False)[source]

Bases: PivotOper

-> { <varsfrompath> }

async run(runt, genr)[source]

class synapse.lib.ast.RelProp(astinfo, kids=())[source]: Bases: PropName

class synapse.lib.ast.RelPropCond(astinfo, kids=())[source]

Bases: Cond

(:foo:bar or .univ) <cmpr> <value>

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]

class synapse.lib.ast.RelPropValue(astinfo, kids=())[source]: Bases: PropValue

class synapse.lib.ast.Return(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

class synapse.lib.ast.Search(astinfo, kids=())[source]

Bases: Query

async run(runt, genr)[source]

class synapse.lib.ast.SetItemOper(astinfo, kids=())[source]

Bases: Oper

$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz

async run(runt, genr)[source]

class synapse.lib.ast.SetVarOper(astinfo, kids=())[source]

Bases: Oper

getRuntVars(runt)[source]

async run(runt, genr)[source]

class synapse.lib.ast.Stop(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

class synapse.lib.ast.SubGraph(rules)[source]

Bases: object

An Oper like object which generates a subgraph.

Notes

The rules format for the subgraph is shaped like the following:

rules = {

    'degrees': 1,

    'edges': True,
    'edgelimit': 3000,
    'filterinput': True,
    'yieldfiltered': False,

    'filters': [
        '-(#foo or #bar)',
        '-(foo:bar or baz:faz)',
    ],

    'pivots': [
        '-> * | limit 100',
        '<- * | limit 100',
    ]

    'forms': {

        'inet:fqdn':{
            'filters': [],
            'pivots': [],
        }

        '*': {
            'filters': [],
            'pivots': [],
        },
    },
}

Nodes which were original seeds have path.meta(‘graph:seed’).

All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.

async omit(runt, node)[source]

async pivots(runt, node, path, existing)[source]

async run(runt, genr)[source]

class synapse.lib.ast.SubQuery(astinfo, kids=())[source]

Bases: Oper

async compute(runt, path)[source]

Use subquery as a value. It is error if the subquery used in this way doesn’t yield exactly one node or has a return statement.

Its value is the primary property of the node yielded, or the returned value.

async compute_array(runt, path)[source]: Use subquery as an array.

async inline(runt, genr)[source]: Operate subquery as if it were inlined

isRuntSafe(runt)[source]

async run(runt, genr)[source]

class synapse.lib.ast.SubqCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.SwitchCase(astinfo, kids=())[source]

Bases: Oper

prepare()[source]

async run(runt, genr)[source]

class synapse.lib.ast.TagCond(astinfo, kids=())[source]

Bases: Cond

#foo.bar

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]

class synapse.lib.ast.TagMatch(astinfo, kids=())[source]

Bases: TagName

Like TagName, but can have asterisks

async compute(runt, path)[source]

hasglob()[source]

class synapse.lib.ast.TagName(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

async computeTagArray(runt, path, excignore=())[source]

prepare()[source]

class synapse.lib.ast.TagProp(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.TagPropCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagPropValue(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.TagValuCond(astinfo, kids=())[source]

Bases: Cond

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagValue(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

isRuntSafe(runt)[source]

isRuntSafeAtom(runt)[source]

class synapse.lib.ast.TryCatch(astinfo, kids=())[source]

Bases: AstNode

async getCatchBlock(name, runt, path=None)[source]

async getErrValu(e)[source]

async run(runt, genr)[source]

class synapse.lib.ast.UnaryExprNode(astinfo, kids=())[source]

Bases: Value

A unary (i.e. single-argument) expression node

async compute(runt, path)[source]

prepare()[source]

class synapse.lib.ast.UnivProp(astinfo, kids=())[source]

Bases: RelProp

async compute(runt, path)[source]

class synapse.lib.ast.UnivPropValue(astinfo, kids=())[source]: Bases: PropValue

class synapse.lib.ast.Value(astinfo, kids=())[source]

Bases: AstNode

The base class for all values and value expressions.

async compute(runt, path)[source]

async getCondEval(runt)[source]: Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]

isRuntSafe(runt)[source]

class synapse.lib.ast.VarDeref(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

class synapse.lib.ast.VarEvalOper(astinfo, kids=())[source]

Bases: Oper

Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)

async run(runt, genr)[source]

class synapse.lib.ast.VarList(astinfo, valu, kids=())[source]: Bases: Const

class synapse.lib.ast.VarListSetOper(astinfo, kids=())[source]

Bases: Oper

getRuntVars(runt)[source]

async run(runt, genr)[source]

class synapse.lib.ast.VarValue(astinfo, kids=())[source]

Bases: Value

async compute(runt, path)[source]

hasVarName(name)[source]

isRuntSafe(runt)[source]

isRuntSafeAtom(runt)[source]

prepare()[source]

validate(runt)[source]

class synapse.lib.ast.WhileLoop(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

class synapse.lib.ast.YieldValu(astinfo, kids=())[source]

Bases: Oper

async run(runt, genr)[source]

async yieldFromValu(runt, valu)[source]

async synapse.lib.ast.expr_add(x, y)[source]

async synapse.lib.ast.expr_div(x, y)[source]

async synapse.lib.ast.expr_eq(x, y)[source]

async synapse.lib.ast.expr_ge(x, y)[source]

async synapse.lib.ast.expr_gt(x, y)[source]

async synapse.lib.ast.expr_le(x, y)[source]

async synapse.lib.ast.expr_lt(x, y)[source]

async synapse.lib.ast.expr_mod(x, y)[source]

async synapse.lib.ast.expr_mul(x, y)[source]

async synapse.lib.ast.expr_ne(x, y)[source]

async synapse.lib.ast.expr_neg(x)[source]

async synapse.lib.ast.expr_not(x)[source]

async synapse.lib.ast.expr_pow(x, y)[source]

async synapse.lib.ast.expr_prefix(x, y)[source]

async synapse.lib.ast.expr_re(x, y)[source]

async synapse.lib.ast.expr_sub(x, y)[source]

synapse.lib.ast.parseNumber(x)[source]

async synapse.lib.ast.pullone(genr)[source]

synapse.lib.autodoc module

class synapse.lib.autodoc.RstHelp[source]

Bases: object

addHead(name, lvl=0, link=None, addprefixline=True, addsuffixline=True)[source]

addLines(*lines)[source]

getRstText()[source]

synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)[source]

Process a list of StormTypes doc information to add them to an RstHelp object.

Notes: This will create internal hyperlink link targets for each header item. The link prefix string must be given with the linkprefix argument.

Parameters:

page (RstHelp) – The RST page to add .
docinfo (dict) – A Stormtypes Doc.
linkprefix (str) – The RST link prefix string to use.
islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.
lvl (int) – The base header level to use when adding headers to the page.

Returns:

None

synapse.lib.autodoc.genCallsig(rtype)[source]

synapse.lib.autodoc.getArgLines(rtype)[source]

synapse.lib.autodoc.getLink(sname, linkprefix, ref=False, suffix=None)[source]

synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None, isstor=False)[source]

synapse.lib.autodoc.getRtypeStr(rtype, known_types, types_prefix, suffix)[source]

synapse.lib.autodoc.ljuster(ilines)[source]: Helper to lstrip lines of whitespace an appropriate amount.

synapse.lib.autodoc.prepareRstLines(doc)[source]: Prepare a desc string for RST lines.

synapse.lib.autodoc.runtimeDocStormTypes(page, docinfo, islib=False, lvl=1, oneline=False, addheader=True, preamble=None)[source]

Process a list of StormTypes doc information to add them to a RstHelp object.

Used for Storm runtime help generation.

Parameters:

page (RstHelp) – The RST page to add .
docinfo (dict) – A Stormtypes Doc.
linkprefix (str) – The RST link prefix string to use.
islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.
lvl (int) – The base header level to use when adding headers to the page.
oneline (bool) – Only display the first line of description. Omits local headers.
preamble (list) – Lines added after the header; and before locls.

Returns:

None

synapse.lib.autodoc.runtimeGetArgLines(rtype)[source]

synapse.lib.autodoc.runtimeGetReturnLines(rtype, isstor=False)[source]

synapse.lib.autodoc.scrubLines(lines)[source]: Remove any empty lines until we encounter non-empty linee

synapse.lib.base module

class synapse.lib.base.Base[source]

Bases: object

Base class for Synapse objects.

Acts as an observable, enables async init and fini.

Example

class Foo(Base):

async def __anit__(self, x, y):

await Base.__anit__(self)

await stuff(x, y)

foo = await Foo.anit(10)

Note

One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

async addSignalHandlers()[source]: Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

async classmethod anit(*args, **kwargs)[source]

async dist(mesg)[source]

Distribute an existing event tuple.

Parameters:: mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) )

async enter_context(item)[source]

Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.

Returns:: The result of item’s own __aenter__ or __enter__() method.

async fini()[source]

Shut down the object and notify any onfini() coroutines.

Returns:: Remaining ref count

async fire(evtname, **info)[source]

Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’):: print(‘got: %r’ % (ret,))

incref()[source]: Increment the reference count for this base. This API may be optionally used to control fini().

link(func)[source]

Add a callback function to receive all events.

Example

base1 = Base() base2 = Base()

base1.link( base2.dist )

# all events on base1 are also propagated on base2

async main()[source]: Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )

Note

This API may only be used when the ioloop is also the main thread.

off(evnt, func)[source]

Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc )

on(evnt, func, base=None)[source]

Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.

Parameters:

evnt (str) – An event name
func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it:

async def baz(event):
x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y

d.on(‘foo’, baz)

# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)

Return type:: None

onWith(evnt, func)[source]

A context manager which can be used to add a callback and remove it when using a with statement.

Parameters:

evnt (str) – An event name
func (function) – A callback function to receive event tufo

onfini(func)[source]: Add a function/coroutine/Base to be called on fini().

async postAnit()[source]: Method called after self.__anit__() has completed, but before anit() returns the object to the caller.

schedCallSafe(func, *args, **kwargs)[source]

Schedule a function to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on the function completion.

Parameters:

func –
*args –
**kwargs –

Notes

This method may be called from outside of the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.

Returns:: A Future representing the eventual function execution.
Return type:: concurrent.futures.Future

schedCoro(coro)[source]

Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.

Parameters:: coro – The coroutine to schedule.

Notes

This function is not threadsafe and must be run on the Base’s event loop. Tasks created by this function do inherit the synapse.lib.scope Scope from the current task.

Returns:: An asyncio.Task object.
Return type:: asyncio.Task

schedCoroSafe(coro)[source]

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on coroutine completion.

Notes

This method may be run outside the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.

Returns:: A Future representing the eventual coroutine execution.
Return type:: concurrent.futures.Future

schedCoroSafePend(coro)[source]: Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note

This method may not be run inside an event loop

unlink(func)[source]

Remove a callback function previously added with link()

Example

base.unlink( callback )

waiter(count, *names)[source]

Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events…

waiter = base.waiter(10,’foo:bar’)

# .. fire task that will cause foo:bar events

events = await waiter.wait(timeout=3)

if events == None:: # handle the timeout case…
for event in events:: # parse the events if you need…

Note

Use this with caution. It’s easy to accidentally construct race conditions with this mechanism ;)

async waitfini(timeout=None)[source]

Wait for the base to fini()

Returns:: None if timed out, True if fini happened

Example

base.waitfini(timeout=30)

class synapse.lib.base.BaseRef[source]

Bases: Base

An object for managing multiple Base instances by name.

async gen(name)[source]

Atomically get/gen a Base and incref. (requires ctor during BaseRef init)

Parameters:: name (str) – The name/iden of the Base instance.

get(name)[source]

Retrieve a Base instance by name.

Parameters:: name (str) – The name/iden of the Base
Returns:: The Base instance (or None)
Return type:: (Base)

items()[source]

pop(name)[source]

Remove and return a Base from the BaseRef.

Parameters:: name (str) – The name/iden of the Base instance
Returns:: The named base ( or None )
Return type:: (Base)

put(name, base)[source]

Add a Base (or sub-class) to the BaseRef by name.

Parameters:

name (str) – The name/iden of the Base
base (Base) – The Base instance

Returns:

(None)

vals()[source]

class synapse.lib.base.Waiter(base, count, *names)[source]

Bases: object

A helper to wait for a given number of events on a Base.

fini()[source]

async wait(timeout=None)[source]

Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30)

if evnts == None:: handleTimedOut() return
for evnt in evnts:: doStuff(evnt)

async synapse.lib.base.main(coro)[source]

async synapse.lib.base.schedGenr(genr, maxsize=100)[source]: Schedule a generator to run on a separate task and yield results to this task (pipelined generator).

synapse.lib.boss module

class synapse.lib.boss.Boss[source]

Bases: Base

An object to track “promoted” async tasks.

async execute(coro, name, user, info=None, iden=None)[source]: Create a synapse task from the given coroutine.

get(iden)[source]

async promote(name, user, info=None, taskiden=None)[source]

Promote the currently running task.

Parameters:

name (str) – The name of the task.
user – The User who owns the task.
taskiden – An optional GUID for the task.
info – An optional information dictionary containing information about the task.

Returns:

The Synapse Task object.

Return type:

s_task.Task

async promotetask(task, name, user, info=None, taskiden=None)[source]

ps()[source]

synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully.

class synapse.lib.cache.EdgeGlobs[source]: Bases: TagGlobs

class synapse.lib.cache.FixedCache(callback, size=10000)[source]

Bases: object

async aget(key)[source]

clear()[source]

get(key)[source]

pop(key)[source]

put(key, val)[source]

class synapse.lib.cache.LruDict(size=10000)[source]

Bases: MutableMapping

Maintains the last n accessed keys

get(key, default=None)[source]: Note: we override default impl from parent to avoid costly KeyError

items() → a set-like object providing a view on D's items[source]

values() → an object providing a view on D's values[source]

class synapse.lib.cache.TagGlobs[source]

Bases: object

An object that manages multiple tag globs and values for caching.

add(name, valu, base=None)[source]

get(name)[source]

rem(name, valu)[source]

synapse.lib.cache.getEdgeGlobRegx(name)[source]

synapse.lib.cache.getTagGlobRegx(name)[source]

synapse.lib.cache.memoize(size=16384)[source]

synapse.lib.cache.memoizemethod(size=16384)[source]: A version of memoize that doesn’t cause GC cycles when applied to a method.

synapse.lib.cache.regexizeEdgeGlob(edge)[source]

synapse.lib.cache.regexizeTagGlob(tag)[source]

Returns:: a regular expression string with ** and * interpreted as tag globs

Precondition:: tag is a valid tagmatch

Notes

A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.

The returned string does not contain a starting ‘^’ or trailing ‘$’.

synapse.lib.cell module

class synapse.lib.cell.Cell[source]

Bases: Pusher, Aware

A Cell() implements a synapse micro-service.

A Cell has 5 phases of startup:

Universal cell data structures
Service specific storage/data (pre-nexs)
Nexus subsystem initialization
Service specific startup (with nexus)
Networking and mirror services

BACKUP_SPAWN_TIMEOUT = 60.0

COMMIT = ''

FREE_SPACE_CHECK_FREQ = 60.0

VERSION = (2, 167, 0)

VERSTRING = '2.167.0'

addActiveCoro(func, iden=None, base=None)[source]

Add a function callback to be run as a coroutine when the Cell is active.

Parameters:

func (coroutine function) – The function run as a coroutine.
iden (str) – The iden to use for the coroutine.
base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d

Returns:

A GUID string that identifies the coroutine for delActiveCoro()

Return type:

str

Note

This will re-fire the coroutine if it exits and the Cell is still active.

addHealthFunc(func)[source]: Register a callback function to get a HealthCheck object.

addHttpApi(path, ctor, info)[source]

async addHttpSess(iden, info)[source]

async addHttpsPort(port, host='0.0.0.0', sslctx=None)[source]

Add a HTTPS listener to the Cell.

Parameters:

port (int) – The listening port to bind.
host (str) – The listening host.
sslctx (ssl.SSLContext) – An externally managed SSL Context.

Note

If the SSL context is not provided, the Cell will assume it manages the SSL context it creates for a given listener and will add a reload handler named https:certs to enabled reloading the SSL certificates from disk.

addReloadableSystem(name: str, func: callable)[source]

Add a reloadable system. This may be dynamically called at at time.

Parameters:

name (str) – Name of the system.
func – The callable for reloading a given system.

Note

Reload functions take no arguments when they are executed. Values returned by the reload function must be msgpack friendly.

Returns:: None

async addRole(name)[source]

async addRoleRule(iden, rule, indx=None, gateiden=None)[source]

async addSignalHandlers()[source]: Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

async addUser(name, passwd=None, email=None, iden=None)[source]

async addUserApiKey(useriden, name, duration=None)[source]

Add an API key for a user.

Notes

The secret API key is only available once.

Parameters:

useriden (str) – User iden value.
name (str) – Name of the API key.
duration (int or None) – Duration of time for the API key to be valid ( in milliseconds ).

Returns:

A tuple of the secret API key value and the API key metadata information.

Return type:

tuple

async addUserRole(useriden, roleiden, indx=None)[source]

async addUserRule(iden, rule, indx=None, gateiden=None)[source]

async behold()[source]

beholder()[source]

cellapi: alias of CellApi

checkFreeSpace()[source]

async checkUserApiKey(key)[source]

Check if a user API key is valid.

Notes

If the key is not valid, the dictionary will contain a mesg key. If the key is valid, the dictionary will contain the user def in a udef key, and the key metadata in a kdef key.

Parameters:: key (str) – The API key to check.
Returns:: Tuple of two items, a boolean if the key is valid and a dictionary.
Return type:: tuple

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:parse:proxy:remoteip': {'default': False, 'description': 'Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.', 'type': 'boolean'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'max:users': {'default': 0, 'description': 'Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).', 'minimum': 0, 'type': 'integer'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': True, 'hidedocs': True, 'type': ['string', 'null']}, 'nexslog:async': {'default': True, 'description': 'Set to false to disable async memory mapping of the nexus change log.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.', 'type': 'boolean'}}

confdefs = {}

async cullNexsLog(offs)[source]

async delActiveCoro(iden)[source]

Remove an Active coroutine previously added with addActiveCoro().

Parameters:: iden (str) – The iden returned by addActiveCoro()

async delBackup(name)[source]

async delHttpSess(iden)[source]

async delRole(iden)[source]

async delRoleRule(iden, rule, gateiden=None)[source]

async delUser(iden)[source]

async delUserApiKey(iden)[source]

Delete an existing API key.

Parameters:: iden (str) – The iden of the API key to delete.
Returns:: True indicating the key was deleted.
Return type:: bool

async delUserRole(useriden, roleiden)[source]

async delUserRule(iden, rule, gateiden=None)[source]

async dyncall(iden, todo, gatekeys=())[source]

async dyniter(iden, todo, gatekeys=())[source]

async classmethod execmain(argv, outp=None)[source]

The main entry point for running the Cell as an application.

Parameters:

argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.

Returns:: None.

async feedBeholder(name, info, gates=None, perms=None)[source]

Feed a named event onto the cell:beholder message bus that will sent to any listeners.

Parameters:

info (dict) – An information dictionary to be sent to any consumers.
gates (list) – List of gate idens, whose details will be added to the outbound message(s).
perms (list) – List of permission names, whose details will be added to the outbound message(s).

Returns:

None

async fini()[source]: Fini override that ensures locking teardown order.

async genHttpSess(iden)[source]

async genUserOnepass(iden, duration=600000)[source]

async getAhaInfo()[source]

async getApiKeys()[source]

Get all API keys in the cell.

Yields:: tuple – kdef values

classmethod getArgParser(conf=None)[source]

Get an argparse.ArgumentParser for the Cell.

Parameters:: conf (s_config.Config) – Optional, a Config object which

Notes

Boot time configuration data is placed in the argument group called config. This adds default dirn, --telepath, --https and --name arguements to the argparser instance. Configuration values which have the hideconf or hidecmdl value set to True are not added to the argparser instance.

Returns:: A ArgumentParser for the Cell.
Return type:: argparse.ArgumentParser

async getAuthGate(iden)[source]

async getAuthGates()[source]

async getAuthRoles()[source]

async getAuthUsers(archived=False)[source]

async getBackupInfo()[source]: Gets information about recent backup activity

async getBackups()[source]

getCachedSslCtx(opts=None, verify=None)[source]

async getCellApi(link, user, path)[source]

Get an instance of the telepath Client object for a given user, link and path.

Parameters:

link (s_link.Link) – The link object.
user (s_hive.HiveUser) – The heavy user object.
path (str) – The path requested.

Notes

This defaults to the self.cellapi class. Implementors may override the default class attribute for cellapi to share a different interface.

Returns:: The shared object for this cell.
Return type:: object

getCellIden()[source]

async getCellInfo()[source]

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns:: A Dictionary of metadata.
Return type:: Dict

getCellNexsRoot()[source]

async getCellRunId()[source]

classmethod getCellType()[source]

async getConfOpt(name)[source]

async getDmonSessions()[source]

getDmonUser()[source]: Return the user IDEN of a telepath caller who invoked the current task. ( defaults to returning current root user )

classmethod getEnvPrefix()[source]: Get a list of envar prefixes for config resolution.

async getHealthCheck()[source]

async getHiveKey(path)[source]: Get the value of a key in the cell default hive

async getHiveKeys(path)[source]: Return a list of (name, value) tuples for nodes under the path.

async getHttpSessDict(iden)[source]

getLocalProxy(share='*', user='root')[source]

getLocalUrl(share='*', user='root')[source]

async getLogExtra(**kwargs)[source]

Get an extra dictionary for structured logging which can be used as a extra argument for loggers.

Parameters:: **kwargs – Additional key/value items to add to the log.
Returns:: A dictionary
Return type:: Dict

async getMirrorUrls()[source]

async getNexsIndx()[source]

async getNexusChanges(offs, tellready=False)[source]

getPermDef(perm)[source]

getPermDefs()[source]

getReloadableSystems()[source]

async getRoleDef(iden)[source]

async getRoleDefByName(name)[source]

async getRoleDefs()[source]

getSpooledSet()[source]

async getSystemInfo()[source]

Get info about the system in which the cell is running

Returns:

volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
cellstarttime - Cell start time in epoch milliseconds
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system
cpucount - Number of CPUs on system
tmpdir - The temporary directory interpreted by the Python runtime.

Return type:

A dictionary with the following keys. All size values are in bytes

async getTeleApi(link, mesg, path)[source]: Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

getTempDir()[source]

async getUserApiKey(iden)[source]

Get a user API key via iden.

Notes

This contains the raw value. Callers are responsible for removing the shadow key.

Parameters:: iden (str) – The key iden.
Returns:: The key dictionary; or none.
Return type:: dict

async getUserDef(iden, packroles=True)[source]

async getUserDefByName(name)[source]

async getUserDefs()[source]

getUserName(iden, defv='<unknown>')[source]: Translate the user iden to a user name.

async getUserProfInfo(iden, name)[source]

async getUserProfile(iden)[source]

async getUserVarValu(iden, name)[source]

async handoff(turl, timeout=30)[source]: Hand off leadership to a mirror in a transactional fashion.

async hasHttpSess(iden)[source]

classmethod initCellConf(conf=None)[source]

Create a Config object for the Cell.

Parameters:: conf (s_config.Config) – An optional config structure. This has _opts_data taken from it.

Notes

The Config object has a envar_prefix set according to the results of cls.getEnvPrefix().

Returns:: A Config helper object.
Return type:: s_config.Config

async classmethod initFromArgv(argv, outp=None)[source]

Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.

Parameters:

argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This does the following items:

Create a Config object from the Cell class.
Creates an Argument Parser from the Cell class and Config object.
Parses the provided arguments.
Loads configuration data from the parsed options and environment variables.
Sets logging for the process.
Creates the Cell from the Cell Ctor.
Adds a Telepath listener, HTTPs port listeners and Telepath share names.
Returns the Cell.

Returns:: This returns an instance of the Cell.
Return type:: Cell

async initNexusSubsystem()[source]

async initServiceActive()[source]

async initServiceNetwork()[source]

async initServicePassive()[source]

async initServiceRuntime()[source]

async initServiceStorage()[source]

initSslCtx(certpath, keypath)[source]

isActiveCoro(iden)[source]

async isCellActive()[source]

async isRoleAllowed(iden, perm, gateiden=None)[source]

async isUserAllowed(iden, perm, gateiden=None, default=False)[source]

async iterBackupArchive(name, user)[source]

async iterNewBackupArchive(user, name=None, remove=False)[source]

async iterUserVars(iden)[source]

async kill(user, iden)[source]

async listHiveKey(path=None)[source]

async listUserApiKeys(useriden)[source]

Get all the API keys for a user.

Parameters:: useriden (str) – The user iden.
Returns:: A list of kdef values.
Return type:: list

async loadHiveTree(tree, path=(), trim=False)[source]: Note: this is for expert emergency use only.

modCellConf(conf)[source]

Modify the Cell’s ondisk configuration overrides file and runtime configuration.

Parameters:: conf (dict) – A dictionary of items to set.

Notes

This does require the data being set to be schema valid.

Returns:: None.

async modUserApiKey(iden, key, valu)[source]

Update a value in the user API key metadata.

Parameters:

iden (str) – Iden of the key to update.
key (str) – Name of the valu to update.
valu – The new value.

Returns:

An updated key metadata dictionary.

Return type:

dict

popCellConf(name)[source]

Remove a key from the Cell’s ondisk configuration overrides file and runtime configuration.

Parameters:: name (str) – Name of the value to remove.

Notes

This does not modify the cell.yaml file. This does re-validate the configuration after removing the value, so if the value removed had a default populated by schema, that default would be reset.

Returns:: None

async popHiveKey(path)[source]

Remove and return the value of a key in the cell default hive.

Note: this is for expert emergency use only.

async popUserProfInfo(iden, name, default=None)[source]

async popUserVarValu(iden, name, default=None)[source]

async promote(graceful=False)[source]: Transform this cell from a passive follower to an active cell that writes changes locally.

async ps(user)[source]

async readyToMirror()[source]

async reload(subsystem=None)[source]

async reqAhaProxy(timeout=None)[source]

async reqGateKeys(gatekeys)[source]

async rotateNexsLog()[source]

runActiveTask(coro)[source]

async runBackup(name=None, wait=True)[source]

async saveHiveTree(path=())[source]

async setCellActive(active)[source]

async setHiveKey(path, valu)[source]: Set or change the value of a key in the cell default hive

async setHttpSessInfo(iden, name, valu)[source]

async setNexsIndx(indx)[source]

async setRoleName(iden, name)[source]

async setRoleRules(iden, rules, gateiden=None)[source]

async setUserAdmin(iden, admin, gateiden=None)[source]

async setUserArchived(iden, archived)[source]

async setUserEmail(useriden, email)[source]

async setUserLocked(iden, locked)[source]

async setUserName(useriden, name)[source]

async setUserPasswd(iden, passwd)[source]

async setUserProfInfo(iden, name, valu)[source]

async setUserRoles(useriden, roleidens)[source]

async setUserRules(iden, rules, gateiden=None)[source]

async setUserVarValu(iden, name, valu)[source]

async sync()[source]: no-op mutable for testing purposes. If I am follower, when this returns, I have received and applied all the writes that occurred on the leader before this call.

async trimNexsLog(consumers=None, timeout=30)[source]

async tryUserPasswd(name, passwd)[source]

async updateHttpSessInfo(iden, vals: dict)[source]

async waitNexsOffs(offs, timeout=None)[source]

class synapse.lib.cell.CellApi[source]

Bases: Base

addAuthRole(name)[source]

addAuthRule(name, rule, indx=None, gateiden=None)[source]: This API is deprecated.

addRole(name)[source]

addRoleRule(iden, rule, indx=None, gateiden=None)[source]

addUser(name, passwd=None, email=None, iden=None)[source]

addUserRole(useriden, roleiden, indx=None)[source]

addUserRule(iden, rule, indx=None, gateiden=None)[source]

async allowed(perm, default=None)[source]

Check if the user has the requested permission.

Parameters:

perm – permission path components to check
default – Value returned if no value stored

Examples

Form a path and check the permission from a remote proxy:

perm = ('node', 'add', 'inet:ipv4')
allowed = await prox.allowed(perm)
if allowed:
    dostuff()

Returns:: True if the user has permission, False if explicitly denied, None if no entry
Return type:: Optional[bool]

behold()[source]: Yield Cell system messages

checkUserApiKey(key)[source]

cullNexsLog(offs)[source]

Remove Nexus log entries up to (and including) the given offset.

Note

If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.

Only rotated logs where the last index is less than the provided offset will be removed from disk.

Parameters:: offs (int) – The offset to remove entries up to.
Returns:: Whether the cull was executed
Return type:: bool

delAuthRole(name)[source]

delAuthRule(name, rule, gateiden=None)[source]: This API is deprecated.

delAuthUser(name)[source]

delBackup(name)[source]

Delete a backup by name.

Parameters:: name (str) – The name of the backup to delete.

delRole(iden)[source]

delRoleRule(iden, rule, gateiden=None)[source]

delUser(iden)[source]

delUserRole(useriden, roleiden)[source]

delUserRule(iden, rule, gateiden=None)[source]

dyncall(iden, todo, gatekeys=())[source]

dyniter(iden, todo, gatekeys=())[source]

genUserOnepass(iden, duration=60000)[source]

getAuthGate(iden)[source]

getAuthGates()[source]

getAuthInfo(name)[source]: This API is deprecated.

getAuthRoles()[source]

getAuthUsers(archived=False)[source]

Parameters:: archived (bool) – If true, list all users, else list non-archived users

getBackupInfo()[source]

Get information about recent backup activity.

Returns:

currduration - If backup currently running, time in ms since backup started, otherwise None
laststart - Last time (in epoch milliseconds) a backup started
lastend - Last time (in epoch milliseconds) a backup ended
lastduration - How long last backup took in ms
lastsize - Disk usage of last backup completed
lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive
lastexception - Tuple of exception information if last backup failed, otherwise None

Return type:

(dict) It has the following keys

Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.

getBackups()[source]

Retrieve a list of backups.

Returns:: A list of backup names.
Return type:: list[str]

getCellIden()[source]

async getCellInfo()[source]

async getCellRunId()[source]

getCellType()[source]

getCellUser()[source]

getDiagInfo()[source]

getDmonSessions()[source]

getGcInfo()[source]

For diagnostic purposes only!

NOTE: This API is not supported and can be removed at any time!

async getHealthCheck()[source]

getHiveKey(path)[source]

getHiveKeys(path)[source]

getMirrorUrls()[source]

getNexsIndx()[source]

getNexusChanges(offs, tellready=False)[source]

getPermDef(perm)[source]: Return a specific permission definition.

getPermDefs()[source]: Return a non-comprehensive list of perm definitions.

getReloadableSystems()[source]

getRoleDef(iden)[source]

getRoleDefByName(name)[source]

getRoleDefs()[source]

async getRoleInfo(name)[source]

getSystemInfo()[source]

Get info about the system in which the cell is running

Returns:

volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system

Return type:

A dictionary with the following keys. All size values are in bytes

getUserDef(iden, packroles=True)[source]

getUserDefByName(name)[source]

getUserDefs()[source]

async getUserInfo(name)[source]

getUserProfInfo(iden, name)[source]

getUserProfile(iden)[source]

handoff(turl, timeout=30)[source]

async initCellApi()[source]

async isCellActive()[source]: Returns True if the cell is an active/leader cell.

isRoleAllowed(iden, perm, gateiden=None)[source]

isUserAllowed(iden, perm, gateiden=None, default=False)[source]

issue(nexsiden: str, event: str, args, kwargs, meta=None)[source]

iterBackupArchive(name)[source]

Retrieve a backup by name as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters:: name (str) – The name of the backup to retrieve.

iterNewBackupArchive(name=None, remove=False)[source]

Run a new backup and return it as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters:

name (str) – The name of the backup to retrieve.
remove (bool) – Delete the backup after streaming.

async kill(iden)[source]

listHiveKey(path=None)[source]

popHiveKey(path)[source]

popUserProfInfo(iden, name, default=None)[source]

promote(graceful=False)[source]

async ps()[source]

readyToMirror()[source]

reload(subsystem=None)[source]

rotateNexsLog()[source]

Rotate the Nexus log at the current offset.

Returns:: The starting index of the active Nexus log
Return type:: int

runBackup(name=None, wait=True)[source]

Run a new backup.

Parameters:

name (str) – The optional name of the backup.
wait (bool) – On True, wait for backup to complete before returning.

Returns:

The name of the newly created backup.

Return type:

str

runGcCollect(generation=2)[source]

For diagnostic purposes only!

NOTE: This API is not supported and can be removed at any time!

saveHiveTree(path=())[source]

setAuthAdmin(name, isadmin)[source]: This API is deprecated.

setCellUser(iden)[source]

Switch to another user (admin only).

This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.

setHiveKey(path, valu)[source]

setRoleRules(iden, rules, gateiden=None)[source]

setUserAdmin(iden, admin, gateiden=None)[source]

setUserArchived(useriden, archived)[source]

setUserEmail(useriden, email)[source]

setUserLocked(useriden, locked)[source]

async setUserPasswd(iden, passwd)[source]

setUserProfInfo(iden, name, valu)[source]

setUserRoles(useriden, roleidens)[source]

setUserRules(iden, rules, gateiden=None)[source]

trimNexsLog(consumers=None, timeout=60)[source]

Rotate and cull the Nexus log (and those of any consumers) at the current offset.

Note

If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.

Parameters:

consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.
timeout (int) – Time in seconds to wait for downstream consumers to be caught up.

Returns:

The offset that the Nexus log was culled up to and including.

Return type:

int

tryUserPasswd(name, passwd)[source]

waitNexsOffs(offs, timeout=None)[source]

Wait for the Nexus log to write an offset.

Parameters:

offs (int) – The offset to wait for.
timeout (int or None) – An optional timeout in seconds.

Returns:

True if the offset was written, False if it timed out.

Return type:

bool

synapse.lib.cell.SSLCTX_CACHE_SIZE = 64: Base classes for the synapse “cell” microservice architecture.

synapse.lib.cell.adminapi(log=False)[source]

Decorator for CellApi (and subclasses) for requiring a method to be called only by an admin user.

Parameters:: log (bool) – If set to True, log the user, function and arguments.

synapse.lib.certdir module

class synapse.lib.certdir.CertDir(path: str | None = None)[source]

Bases: object

Certificate loading/generation/signing utilities.

Features:

Locates and load certificates, keys, and certificate signing requests (CSRs).
Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.
Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).
Signs certificate signing requests (CSRs).
Generates PKCS#12 archives for use in browser.

Parameters:: path (str) – Optional path which can override the default path directory.

Notes

All certificates will be loaded from and written to ~/.syn/certs by default. Set the environment variable SYN_CERT_DIR to override.
All certificate generation methods create 4096 bit RSA keypairs.
All certificate signing methods use sha256 as the signature algorithm.
CertDir does not currently support signing CA CSRs.

addCertPath(*path: str)[source]

delCertPath(*path: str)[source]

genCaCert(name: str, signas: str | None = None, outp: OutPut | None = None, save: bool = True) → Tuple[RSAPrivateKey, Certificate][source]

Generates a CA keypair.

Parameters:

name – The name of the CA keypair.
signas – The CA keypair to sign the new CA with.
outp – The output buffer.
save – Save the certificate and key to disk.

Examples

Make a CA named “myca”:

mycakey, mycacert = cdir.genCaCert('myca')

Returns:: Tuple containing the private key and certificate objects.

genCaCrl(name: str) → Crl[source]

Get the CRL for a given CA.

Parameters:: name – The CA name.
Returns:: The CRL object.

genClientCert(name: str, outp: OutPut | None = None) → None[source]

Generates a user PKCS #12 archive.

Please note that the resulting file will contain private key material.

Parameters:

name (str) – The name of the user keypair.
outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”:

myuserpkcs12 = cdir.genClientCert('myuser')

Returns:: None

genCodeCert(name: str, signas: str | None = None, outp: OutPut | None = None, save: bool = True) → Tuple[RSAPrivateKey, Certificate][source]

Generates a code signing keypair.

Parameters:

name – The name of the code signing cert.
signas – The CA keypair to sign the new code keypair with.
outp – The output buffer.

Examples

Generate a code signing cert for the name “The Vertex Project”:

myuserkey, myusercert = cdir.genCodeCert('The Vertex Project')

Returns:: Tuple containing the key and certificate objects.

genCrlPath(name: str) → str[source]

Generates a host keypair.

Parameters:

name – The name of the host keypair.
signas – The CA keypair to sign the new host keypair with.
outp – The output buffer.
csr – The CSR public key when generating the keypair from a CSR.
sans – String of comma separated alternative names.

Examples

Make a host keypair named “myhost”:

myhostkey, myhostcert = cdir.genHostCert('myhost')

Returns:: Tuple containing the private key and certificate objects. Private key may be None when signing a CSR.

genHostCsr(name: str, outp: OutPut | None = None) → bytes[source]

Generates a host certificate signing request.

Parameters:

name – The name of the host CSR.
outp – The output buffer.

Examples

Generate a CSR for the host key named “myhost”:

cdir.genHostCsr('myhost')

Returns:: The bytes of the CSR.

Generates a user keypair.

Parameters:

name – The name of the user keypair.
signas – The CA keypair to sign the new user keypair with.
outp – The output buffer.
csr – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”:

myuserkey, myusercert = cdir.genUserCert('myuser')

Returns:: Tuple containing the key and certificate objects.

genUserCsr(name: str, outp: OutPut | None = None) → bytes[source]

Generates a user certificate signing request.

Parameters:

name – The name of the user CSR.
outp – The output buffer.

Examples

Generate a CSR for the user “myuser”:

cdir.genUserCsr('myuser')

Returns:: The bytes of the CSR.

getCaCert(name: str) → Certificate | None[source]

Loads the X509 object for a given CA.

Parameters:: name – The name of the CA keypair.

Examples

Get the certificate for the CA “myca”:

mycacert = cdir.getCaCert('myca')

Returns:: The certificate, if exists.

getCaCertBytes(name: str) → bytes[source]

getCaCertPath(name: str) → str | None[source]

Gets the path to a CA certificate.

Parameters:: name – The name of the CA keypair.

Examples

Get the path to the CA certificate for the CA “myca”:

mypath = cdir.getCACertPath('myca')

Returns:: The path, if exists.

getCaCerts() → List[Certificate][source]

Return a list of CA certs from the CertDir.

Returns:: List of CA certificates.

getCaKey(name) → RSAPrivateKey | DSAPrivateKey | None[source]

Loads the PKey object for a given CA keypair.

Parameters:: name – The name of the CA keypair.

Examples

Get the private key for the CA “myca”:

mycakey = cdir.getCaKey('myca')

Returns:: The private key, if exists.

getCaKeyPath(name: str) → str | None[source]

Gets the path to a CA key.

Parameters:: name – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”:

mypath = cdir.getCAKeyPath('myca')

Returns:: The path, if exists.

getClientCert(name: str) → PKCS12KeyAndCertificates | None[source]

Loads the PKCS12 archive object for a given user keypair.

Parameters:: name – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”:

mypkcs12 = cdir.getClientCert('myuser')

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool

Returns:: The PKCS12 archive, if exists.

getClientCertPath(name: str) → str | None[source]

Gets the path to a client certificate.

Parameters:: name – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”:

mypath = cdir.getClientCertPath('myuser')

Returns:: The path, if exists.

getClientSSLContext(certname: str | None = None) → SSLContext[source]

Returns an ssl.SSLContext appropriate for initiating a TLS session

Parameters:: certname – If specified, use the user certificate with the matching name to authenticate to the remote service.
Returns:: A SSLContext object.

getCodeCert(name: str) → Certificate | None[source]

getCodeCertPath(name: str) → str | None[source]

getCodeKey(name: str) → PriKey | None[source]

getCodeKeyPath(name: str) → str | None[source]

getCrlPath(name: str) → str | None[source]

getHostCaPath(name: str) → str | None[source]

Gets the path to the CA certificate that issued a given host keypair.

Parameters:: name – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”:

mypath = cdir.getHostCaPath('myhost')

Returns:: The path, if exists.

getHostCert(name: str) → Certificate | None[source]

Loads the X509 object for a given host keypair.

Parameters:: name – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”:

myhostcert = cdir.getHostCert('myhost')

Returns:: The certificate, if exists.

getHostCertHash(name: str) → str | None[source]

getHostCertPath(name: str) → str | None[source]

Gets the path to a host certificate.

Parameters:: name – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”:

mypath = cdir.getHostCertPath('myhost')

Returns:: The path, if exists.

getHostCsrPath(name: str) → str | None[source]

getHostKey(name: str) → RSAPrivateKey | DSAPrivateKey | None[source]

Loads the PKey object for a given host keypair.

Parameters:: name – The name of the host keypair.

Examples

Get the private key object for the host “myhost”:

myhostkey = cdir.getHostKey('myhost')

Returns:: The private key, if exists.

getHostKeyPath(name: str) → str | None[source]

Gets the path to a host key.

Parameters:: name – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”:

mypath = cdir.getHostKeyPath('myhost')

Returns:: The path if exists.
Return type:: str

getServerSSLContext(hostname: str | None = None, caname: str | None = None) → SSLContext[source]

Returns an ssl.SSLContext appropriate to listen on a socket

Parameters:

hostname – If None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory.
caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.

Returns:

A SSLContext object.

getUserCaPath(name: str) → str | None[source]

Gets the path to the CA certificate that issued a given user keypair.

Parameters:: name – The name of the user keypair.

Examples

Get the path to the CA cert which issue the cert for “myuser”:

mypath = cdir.getUserCaPath('myuser')

Returns:: The path, if exists.

getUserCert(name: str) → Certificate | None[source]

Loads the X509 object for a given user keypair.

Parameters:: name – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”:

myusercert = cdir.getUserCert('myuser')

Returns:: The certificate, if exists.

getUserCertPath(name: str) → str | None[source]

Gets the path to a user certificate.

Parameters:: name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”:

mypath = cdir.getUserCertPath('myuser')

Returns:: The path, if exists.

getUserCsrPath(name: str) → str | None[source]

getUserForHost(user: str, host: str) → str | None[source]

Gets the name of the first existing user cert for a given user and host.

Parameters:

user – The name of the user.
host – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”:

usercertname = cdir.getUserForHost('myuser', 'cool.vertex.link')

Returns:: The cert name, if exists.
Return type:: str

getUserKey(name: str) → RSAPrivateKey | DSAPrivateKey | None[source]

Loads the PKey object for a given user keypair.

Parameters:: name – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”:

myuserkey = cdir.getUserKey('myuser')

Returns:: The private key, if exists.

getUserKeyPath(name: str) → str | None[source]

Gets the path to a user key.

Parameters:: name – The name of the user keypair.

Examples

Get the path to the user key for “myuser”:

mypath = cdir.getUserKeyPath('myuser')

Returns:: The path, if exists.

importFile(path: str, mode: str, outp: OutPut | None = None) → None[source]

Imports certs and keys into the Synapse cert directory

Parameters:

path – The path of the file to be imported.
mode – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory:

certdir.importFile('mycoolca.crt', 'cas')

Notes

importFile does not perform any validation on the files it imports.

Returns:: None

isCaCert(name: str) → bool[source]

Checks if a CA certificate exists.

Parameters:: name – The name of the CA keypair.

Examples

Check if the CA certificate for “myca” exists:

exists = cdir.isCaCert('myca')

Returns:: True if the certificate is present, False otherwise.

isClientCert(name: str) → bool[source]

Checks if a user client certificate (PKCS12) exists.

Parameters:: name – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists:

exists = cdir.isClientCert('myuser')

Returns:: True if the certificate is present, False otherwise.

isCodeCert(name: str) → bool[source]

Checks if a code certificate exists.

Parameters:: name – The name of the code keypair.

Examples

Check if the code cert “mypipeline” exists:

exists = cdir.isCodeCert('mypipeline')

Returns:: True if the certificate is present, False otherwise.

isHostCert(name: str) → bool[source]

Checks if a host certificate exists.

Parameters:: name – The name of the host keypair.

Examples

Check if the host cert “myhost” exists:

exists = cdir.isUserCert('myhost')

Returns:: True if the certificate is present, False otherwise.

isUserCert(name: str) → bool[source]

Checks if a user certificate exists.

Parameters:: name – The name of the user keypair.

Examples

Check if the user cert “myuser” exists:

exists = cdir.isUserCert('myuser')

Returns:: True if the certificate is present, False otherwise.

loadCertByts(byts: bytes) → Certificate[source]

Load a X509 certificate from its PEM encoded bytes.

Parameters:: byts – The PEM encoded bytes of the certificate.
Returns:: The X509 certificate.
Raises:: BadCertBytes – If the certificate bytes are invalid.

saveCaCertByts(byts: bytes) → str[source]

saveCertPem(cert: Certificate, path: str) → None[source]: Save a certificate in PEM format to a file outside the certdir.

saveCodeCertBytes(byts: bytes) → str[source]

saveHostCertByts(byts: bytes) → str[source]

savePkeyPem(pkey: DHPrivateKey | Ed25519PrivateKey | Ed448PrivateKey | RSAPrivateKey | DSAPrivateKey | EllipticCurvePrivateKey | X25519PrivateKey | X448PrivateKey, path: str) → None[source]: Save a private key in PEM format to a file outside the certdir.

saveUserCertByts(byts: bytes) → str[source]

selfSignCert(builder: CertificateBuilder, pkey: RSAPrivateKey | DSAPrivateKey) → Certificate[source]

Self-sign a certificate.

Parameters:

cert – The certificate to sign.
pkey – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key:

cdir.selfSignCert(mycert, myotherprivatekey)

Returns:: None

signCertAs(builder: CertificateBuilder, signas: str) → Certificate[source]

Signs a certificate with a CA keypair.

Parameters:

cert – The certificate to sign.
signas – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”:

cdir.signCertAs(mycert, 'myca')

Returns:: None

signHostCsr(xcsr: CertificateSigningRequest, signas: str, outp: OutPut | None = None, sans: str | None = None, save: bool = True) → Tuple[RSAPrivateKey | None, Certificate][source]

Signs a host CSR with a CA keypair.

Parameters:

xcsr – The certificate signing request.
signas – The CA keypair name to sign the CSR with.
outp – The output buffer.
sans – List of subject alternative names.

Examples

Sign a host key with the CA “myca”:

cdir.signHostCsr(mycsr, 'myca')

Returns:: Tuple containing the public key and certificate objects.

signUserCsr(xcsr: CertificateSigningRequest, signas: str, outp: OutPut | None = None, save: bool = True) → Tuple[RSAPrivateKey | None, Certificate][source]

Signs a user CSR with a CA keypair.

Parameters:

xcsr – The certificate signing request.
signas – The CA keypair name to sign the CSR with.
outp – The output buffer.

Examples

Sign a user CSR with “myca”:

cdir.signUserCsr(mycsr, 'myca')

Returns:: Tuple containing the public key and certificate objects.
Return type:: ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

valCodeCert(byts: bytes) → Certificate[source]

Verify a code cert is valid according to certdir’s available CAs and CRLs.

Parameters:: byts – The certificate bytes.
Raises:: s_exc.BadCertVerify if we are unable to verify the certificate. –
Returns:: The certificate.

valUserCert(byts: bytes, cacerts: List[Certificate] | None = None) → Certificate[source]

Validate the PEM encoded x509 user certificate bytes and return it.

Parameters:

byts – The bytes for the User Certificate.
cacerts – A tuple of CA Certificates to use for validating the user cert.

Raises:

BadCertVerify – If the certificate is not valid.

Returns:

The certificate, if it is valid.

class synapse.lib.certdir.Crl(cdir, name)[source]

Bases: object

revoke(cert: Certificate) → None[source]

Revoke a certificate with the CRL.

Parameters:: cert – The certificate to revoke.
Returns:: None

synapse.lib.certdir.addCertPath(path)[source]

synapse.lib.certdir.delCertPath(path)[source]

synapse.lib.certdir.getCertDir() → CertDir[source]

Get the singleton CertDir instance.

Returns:: A certdir object.
Return type:: CertDir

synapse.lib.certdir.getCertDirn() → str[source]

Get the expanded default path used by the singleton CertDir instance.

Returns:: The path string.
Return type:: str

synapse.lib.certdir.getServerSSLContext() → SSLContext[source]

Get a server SSLContext object.

This object has a minimum TLS version of 1.2, a subset of ciphers in use, and disabled client renegotiation.

This object has no certificates loaded in it.

Returns:: The context object.
Return type:: ssl.SSLContext

synapse.lib.certdir.iterFqdnUp(fqdn)[source]

synapse.lib.chop module

synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0): Shared primitive routines for chopping up strings and values.

synapse.lib.chop.cvss2_normalize(vect)[source]: Helper function to normalize CVSS2 vectors

synapse.lib.chop.cvss3x_normalize(vect)[source]: Helper function to normalize CVSS3.X vectors

synapse.lib.chop.cvss_normalize(vdict, vers)[source]: Normalize CVSS vectors

synapse.lib.chop.cvss_validate(vect, vers)[source]

Validate (as best as possible) the CVSS vector string. Look for issues such as:

No duplicated metrics
Invalid metrics
Invalid metric values
Missing mandatory metrics

Returns a dictionary with the parsed metric:value pairs.

synapse.lib.chop.digits(text)[source]

synapse.lib.chop.hexstr(text)[source]

Ensure a string is valid hex.

Parameters:: text (str) – String to normalize.

Examples

Norm a few strings:

hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.

Returns:: Normalized hex string.
Return type:: str

synapse.lib.chop.intstr(text)[source]

synapse.lib.chop.onespace(text)[source]

synapse.lib.chop.printables(text)[source]

synapse.lib.chop.replaceUnicodeDashes(valu)[source]

Replace unicode dashes in a string with regular dashes.

Parameters:: valu (str) – A string.
Returns:: A new string with replaced dashes.
Return type:: str

synapse.lib.chop.stormstring(s)[source]

Make a string storm safe by escaping backslashes and double quotes.

Parameters:: s (str) – String to make storm safe.

Notes

This does not encapsulate a string in double quotes.

Returns:: A string which can be embedded directly into a storm query.
Return type:: str

synapse.lib.chop.tag(text)[source]

synapse.lib.chop.tagpath(text)[source]

synapse.lib.chop.tags(norm)[source]: Divide a normalized tag string into hierarchical layers.

synapse.lib.chop.uncnorm(valu)[source]

Validate and normalize the UNC path passed in valu into a URI.

This function will accept @SSL and @<port> as part of the host name to indicate SSL (https) or a specific port number. It can also accept IPv6 addresses in the host name even though those are non-standard according to the spec.

synapse.lib.chop.validateTagMatch(tag)[source]: Raises an exception if tag is not a valid tagmatch (i.e. a tag that might have globs)

synapse.lib.cli module

class synapse.lib.cli.Cli[source]

Bases: Base

A modular / event-driven CLI base object.

addCmdClass(ctor, **opts)[source]: Add a Cmd subclass to this cli.

async addSignalHandlers()[source]: Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.

get(name, defval=None)[source]

getCmdByName(name)[source]: Return a Cmd instance by name.

getCmdNames()[source]: Return a list of all the known command names for the CLI.

getCmdPrompt()[source]

Get the command prompt.

Returns:: Configured command prompt
Return type:: str

histfile = 'cmdr_history'

initCmdClasses()[source]

printf(mesg, addnl=True, color=None)[source]

async prompt(text=None)[source]: Prompt for user input from stdin.

async runCmdLine(line)[source]

Run a single command line.

Parameters:: line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch:

await cli.runCmdLine(‘woot –help’)

Returns:: Arbitrary data from the cmd class.
Return type:: object

async runCmdLoop()[source]: Run commands from a user in an interactive fashion until fini() or EOFError is raised.

set(name, valu)[source]

class synapse.lib.cli.Cmd(cli, **opts)[source]

Bases: object

Base class for modular commands in the synapse CLI.

getCmdBrief()[source]: Return the single-line description for this command.

getCmdDoc()[source]: Return the help/doc output for this command.

getCmdItem()[source]: Get a reference to the object we are commanding.

getCmdName()[source]

getCmdOpts(text)[source]

Use the _cmd_syntax def to split/parse/normalize the cmd line.

Parameters:: text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type

Returns:: An opts dictionary.
Return type:: dict

printf(mesg, addnl=True, color=None)[source]

async runCmdLine(line)[source]

Run a line of command input for this command.

Parameters:: line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:: opts (dict) – Options dictionary.

class synapse.lib.cli.CmdHelp(cli, **opts)[source]

Bases: Cmd

List commands and display help output.

Example

help foocmd

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:: opts (dict) – Options dictionary.

class synapse.lib.cli.CmdLocals(cli, **opts)[source]

Bases: Cmd

List the current locals for a given CLI object.

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:: opts (dict) – Options dictionary.

class synapse.lib.cli.CmdQuit(cli, **opts)[source]

Bases: Cmd

Quit the current command line interpreter.

Example

quit

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:: opts (dict) – Options dictionary.

synapse.lib.cmd module

class synapse.lib.cmd.Parser(prog=None, outp=<synapse.lib.output.OutPut object>, **kwargs)[source]

Bases: ArgumentParser

exit(status=0, message=None)[source]: Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead.

synapse.lib.cmdr module

async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)[source]

Construct and return a cmdr for the given remote cell.

Parameters:

cell – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.

Examples

Get the cmdr for a proxy:

cmdr = await getItemCmdr(foo)

Returns:: A Cli instance with Cmds loaeded into it.
Return type:: s_cli.Cli

async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)[source]

Create a cmdr for the given item and run the cmd loop.

Parameters:

item – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.

Notes

This function does not return while the command loop is run.

Examples

Run the Cmdr for a proxy:

await runItemCmdr(foo)

Returns:: This function returns None.
Return type:: None

synapse.lib.config module

class synapse.lib.config.Config(schema, conf=None, envar_prefixes=None)[source]

Bases: MutableMapping

Synapse configuration helper based on JSON Schema.

Parameters:

schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.
conf (dict) – Optional, a set of configuration data to preload.
envar_prefixes (list) – Optional, a list of prefix strings used when collecting configuration data from environment variables.

Notes

This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.

The default values provided in the schema must be able to be recreated from the repr() of their Python value.

Default values are not loaded into the configuration data until the reqConfValid() method is called.

asDict()[source]

Get a copy of configuration data.

Returns:: A copy of the configuration data.
Return type:: dict

getArgParseArgs()[source]

getCmdlineMapping()[source]

classmethod getConfFromCell(cell, conf=None, envar_prefixes=None)[source]

Get a Config object from a Cell directly (either the ctor or the instance thereof).

Returns:: A Config object.
Return type:: Config

getEnvarMapping(prefix=None)[source]

Get a mapping of config values to envars.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

reqConfValid()[source]

Validate that the loaded configuration data is valid according to the schema.

Notes

The validation set does set any default values which are not currently set for configuration options.

Returns:: This returns nothing.
Return type:: None

reqConfValu(key)[source]

Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.

Parameters:: key (str) – The key to require.
Returns:: The requested value.

reqKeyValid(key, value)[source]

Test if a key is valid for the provided schema it is associated with.

Parameters:

key (str) – Key to check.
value – Value to check.

Raises:

BadArg – If the key has no associated schema.
BadConfValu – If the data is not schema valid.

Returns:

None when valid.

setConfFromEnvs()[source]

Set configuration options from environment variables.

Notes

Environment variables are resolved from configuration options after doing the following transform:

Replace : characters with _.
Add a config provided prefix, if set.
Uppercase the string.
Resolve the environment variable
If the environment variable is set, set the config value to the results of yaml.yaml_safeload() on the value.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

Examples

For the configuration value auth:passwd, the environment variable is resolved as AUTH_PASSWD. With the prefix cortex, the the environment variable is resolved as CORTEX_AUTH_PASSWD.

Returns:: Returns a dictionary of values which were set from enviroment variables.
Return type:: dict

setConfFromFile(path, force=False)[source]

Set the opts for a conf object from YAML file path.

Parameters:

path (str) – Path to the yaml load. If it exists, it must represent a dictionary.
force (bool) – Force the update instead of using setdefault() behavior.

Returns:

None

setConfFromOpts(opts=None)[source]

Set the opts for a conf object from a namespace object.

Parameters:

opts (argparse.Namespace) – A Namespace object made from parsing args with an ArgumentParser
getArgumentParser. (made with) –

Returns:

Returns None.

Return type:

None

synapse.lib.config.getJsSchema(confbase, confdefs)[source]

Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.

Parameters:

confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.
confdefs (dict) – A JSON Schema dictionary of properties for the object.

Notes

This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify

Returns:: A complete JSON schema.
Return type:: dict

synapse.lib.config.getJsValidator(schema, use_default=True, handlers={'http': <function localSchemaRefHandler>, 'https': <function localSchemaRefHandler>})[source]

Get a fastjsonschema callable.

Parameters:

schema (dict) – A JSON Schema object.
use_default (bool) – Whether to insert “default” key arguments into the validated data structure.

Returns:

A callable function that can be used to validate data against the json schema.

Return type:

callable

synapse.lib.config.localSchemaRefHandler(uri)[source]: This function parses the given URI to get the path component and then tries to resolve the referenced schema from the ‘jsonschemas’ directory of synapse.data.

synapse.lib.config.make_envar_name(key, prefix=None)[source]

Convert a colon delimited string into an uppercase, underscore delimited string.

Parameters:

key (str) – Config key to convert.
prefix (str) – Optional string prefix to prepend the the config key.

Returns:

The string to lookup against a envar.

Return type:

str

synapse.lib.const module

synapse.lib.coro module

Async/Coroutine related utilities.

class synapse.lib.coro.Event[source]

Bases: Event

async timewait(timeout=None)[source]

class synapse.lib.coro.GenrHelp(genr)[source]

Bases: object

async list()[source]

async spin()[source]

async synapse.lib.coro.agen(item)[source]

Wrap an async_generator or generator in an async_generator.

Notes

Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.

async synapse.lib.coro.event_wait(event: Event, timeout=None)[source]

Wait on an an asyncio event with an optional timeout

Returns:: true if the event got set, False if timed out

synapse.lib.coro.executor(func, *args, **kwargs)[source]

Execute a non-coroutine function in the ioloop executor pool.

Parameters:

func – Function to execute.
*args – Args for the function.
**kwargs – Kwargs for the function.

Examples

Execute a blocking API call in the executor pool:

import requests

def block(url, params=None):
    return requests.get(url, params=params).json()

fut = s_coro.executor(block, 'http://some.tld/thign')
resp = await fut

Returns:: An asyncio future.
Return type:: asyncio.Future

async synapse.lib.coro.forked(func, *args, **kwargs)[source]

Execute a target function in the shared forked process pool and fallback to running in a spawned process if the pool is unavailable.

Parameters:

func – The target function.
*args – Function positional arguments.
**kwargs – Function keyword arguments.

Returns:

The target function return.

synapse.lib.coro.genrhelp(f)[source]

synapse.lib.coro.iscoro(item)[source]

async synapse.lib.coro.ornot(func, *args, **kwargs)[source]

Calls func and awaits it if a returns a coroutine.

Note

This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.

This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage:: ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)

async synapse.lib.coro.semafork(func, *args, **kwargs)[source]

Execute a target function in the shared forked process pool gated by a semaphore to ensure there are workers reserved for the Storm parser.

Parameters:

func – The target function.
*args – Function positional arguments.
**kwargs – Function keyword arguments.

Returns:

The target function return.

synapse.lib.coro.set_pool_logging(logger_, logconf)[source]

async synapse.lib.coro.spawn(todo, timeout=None, ctx=None, log_conf=None)[source]

Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.

Parameters:

todo (tuple) – A tuple of function, *args, and **kwargs.
timeout (int) – The timeout to wait for the todo function to finish.
ctx (multiprocess.Context) – A optional multiprocessing context object.
log_conf (dict) – An optional logging configuration for the spawned process.

Notes

The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.

Returns:: The return value of executing the todo function.

async synapse.lib.coro.waittask(task, timeout=None)[source]

Await a task without cancelling it when you time out.

Returns:: True if the task completed before the timeout.
Return type:: boolean

synapse.lib.datfile module

Utilities for handling data files embedded within python packages.

synapse.lib.datfile.openDatFile(datpath)[source]

Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.dyndeps module

synapse.lib.dyndeps.getDynLocal(name)[source]

Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()

synapse.lib.dyndeps.getDynMeth(name)[source]: Retrieve and return an unbound method by python path.

synapse.lib.dyndeps.getDynMod(name)[source]

Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’)

synapse.lib.dyndeps.runDynTask(task)[source]

Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)[source]: Dynamically import a module and call a function or raise an exception.

synapse.lib.dyndeps.tryDynLocal(name)[source]: Dynamically import a module and return a module local or raise an exception.

synapse.lib.dyndeps.tryDynMod(name)[source]: Dynamically import a python module or exception.

synapse.lib.encoding module

synapse.lib.encoding.addFormat(name, fn, opts)[source]: Add an additional ingest file format

synapse.lib.encoding.decode(name, byts, **opts)[source]

Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with +: to encode for that name/layer.

synapse.lib.encoding.encode(name, item, **opts)[source]

synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)[source]

Iterate through the data provided by a file like object.

Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd:
    for row in iterdata(fd, format='csv', encoding='utf8'):
        dostuff(row)

Parameters:

fd (file) – File like object to iterate over.
close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.
**opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.

Yields:

An item to process. The type of the item is dependent on the format parameters.

synapse.lib.gis module

synapse.lib.gis.bbox(lat, lon, dist)[source]

Calculate a min/max bounding box for the circle defined by lat/lon/dist.

Parameters:

lat (float) – The latitude in degrees
lon (float) – The longitude in degrees
dist (int) – A distance in geo:dist base units (mm)

Returns:

(latmin, latmax, lonmin, lonmax)

Return type:

(float,float,float,float)

synapse.lib.gis.dms2dec(degs, mins, secs)[source]

Convert degrees, minutes, seconds lat/long form to degrees float.

Parameters:

degs (int) – Degrees
mins (int) – Minutes
secs (int) – Seconds

Returns:

Degrees

Return type:

(float)

synapse.lib.gis.haversine(px, py, r=6371008800.0)[source]

Calculate the haversine distance between two points defined by (lat,lon) tuples.

Parameters:

px ((float,float)) – lat/long position 1
py ((float,float)) – lat/long position 2
r (float) – Radius of sphere

Returns:

Distance in mm.

Return type:

(int)

synapse.lib.gis.latlong(text)[source]

Chop a latlong string and return (float,float). Does not perform validation on the coordinates.

Parameters:: text (str) – A longitude,latitude string.
Returns:: A longitude, latitude float tuple.
Return type:: (float,float)

synapse.lib.gis.near(point, dist, points)[source]

Determine if the given point is within dist of any of points.

Parameters:

point ((float,float)) – A latitude, longitude float tuple.
dist (int) – A distance in mm ( base units )
points (list) – A list of latitude, longitude float tuples to compare against.

synapse.lib.grammar module

synapse.lib.grammar.chop_float(text, off)[source]

synapse.lib.grammar.isBasePropNoPivprop(name)[source]

synapse.lib.grammar.isCmdName(name)[source]

synapse.lib.grammar.isFormName(name)[source]

synapse.lib.grammar.isPropName(name)[source]

synapse.lib.grammar.isUnivName(name)[source]

synapse.lib.grammar.meh(txt, off, cset)[source]

synapse.lib.grammar.nom(txt, off, cset, trim=True)[source]

Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’)

name,off = nom(text,0,chars)

Note:

This really shouldn’t be used for new code

synapse.lib.grammar.parse_float(text, off)[source]

synapse.lib.hashitem module

synapse.lib.hashitem.hashitem(item)[source]: Generate a uniq hash for the JSON compatible primitive data structure.

synapse.lib.hashitem.normdict(item)[source]

synapse.lib.hashitem.normitem(item)[source]

synapse.lib.hashitem.normiter(item)[source]

synapse.lib.hashset module

class synapse.lib.hashset.HashSet[source]

Bases: object

digests()[source]: Get a list of (name, bytes) tuples for the hashes in the hashset.

eatfd(fd)[source]

Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd)

guid()[source]: Use elements from this hash set to create a unique (re)identifier.

update(byts)[source]: Update all the hashes in the set with the given bytes.

synapse.lib.health module

class synapse.lib.health.HealthCheck(iden)[source]

Bases: object

getStatus()[source]

pack()[source]

setStatus(valu)[source]

update(name, status, mesg='', data=None)[source]

Append a new component to the Healcheck object.

Parameters:

name (str) – Name of the reported component.
status (str) – nomdinal/degraded/failed status code.
mesg (str) – Optional message about the component status.
data (dict) – Optional arbitrary dictionary of additional metadata about the component.

Returns:

None

synapse.lib.hive module

class synapse.lib.hive.Hive[source]

Bases: Pusher, Aware

An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.

async add(full, valu)[source]: Atomically increments a node’s value.

async dict(full, nexs=False)[source]

Open a HiveDict at the given full path.

Parameters:: full (tuple) – A full path tuple.
Returns:: A HiveDict for the full path.
Return type:: HiveDict

dir(full)[source]

List subnodes of the given Hive path.

Parameters:: full (tuple) – A full path tuple.

Notes

This returns None if there is not a node at the path.

Returns:: A list of tuples. Each tuple contains the name, node value, and the number of children nodes.
Return type:: list

async exists(full)[source]: Returns whether the Hive path has already been created.

async get(full, defv=None)[source]

Get the value of a node at a given path.

Parameters:: full (tuple) – A full path tuple.
Returns:: Arbitrary node value.

async getHiveAuth()[source]: Retrieve a HiveAuth for hive standalone or non-cell uses.

Note

This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell

async getTeleApi(link, mesg, path)[source]: Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async loadHiveTree(tree, path=(), trim=False)[source]

async open(full)[source]

Open and return a hive Node().

Parameters:: full (tuple) – A full path tuple.
Returns:: A Hive node.
Return type:: Node

async pop(full, nexs=False)[source]: Remove and return the value for the given node.

async rename(oldpath, newpath)[source]: Moves a node at oldpath and all its descendant nodes to newpath. newpath must not exist

async saveHiveTree(path=())[source]

async set(full, valu, nexs=False)[source]: A set operation at the hive level (full path).

async storNodeDele(path)[source]

async storNodeValu(full, valu)[source]

class synapse.lib.hive.HiveApi[source]

Bases: Base

async addAndSync(path, valu, iden)[source]

async edits()[source]

async get(full)[source]

async loadHiveTree(tree, path=(), trim=False)[source]

async popAndSync(path, iden, nexs=False)[source]

async saveHiveTree(path=())[source]

async setAndSync(path, valu, iden, nexs=False)[source]

async treeAndSync(path, iden)[source]

class synapse.lib.hive.HiveDict[source]

Bases: Base

get(name, default=None)[source]

items()[source]

pack()[source]

async pop(name, default=None)[source]

async set(name, valu, nexs=None)[source]

setdefault(name, valu)[source]

values()[source]

class synapse.lib.hive.Node[source]

Bases: Base

A single node within the Hive tree.

async add(valu)[source]: Increments existing node valu

async dict(nexs=False)[source]

Get a HiveDict for this Node.

Returns:: A HiveDict for this Node.
Return type:: HiveDict

dir()[source]

get(name)[source]

name()[source]

async open(path)[source]

Open a child Node of the this Node.

Parameters:: path (tuple) – A child path of the current node.
Returns:: A Node at the child path.
Return type:: Node

parent()[source]

async pop(path=())[source]

async set(valu)[source]

class synapse.lib.hive.SlabHive[source]

Bases: Hive

async storNodeDele(full)[source]

async storNodeValu(full, valu)[source]

class synapse.lib.hive.TeleHive[source]

Bases: Hive

A Hive that acts as a consistent read cache for a telepath proxy Hive

async add(path, valu)[source]: Atomically increments a node’s value.

async get(path)[source]

Get the value of a node at a given path.

Parameters:: full (tuple) – A full path tuple.
Returns:: Arbitrary node value.

async open(path)[source]

Open and return a hive Node().

Parameters:: full (tuple) – A full path tuple.
Returns:: A Hive node.
Return type:: Node

async pop(path, nexs=False)[source]: Remove and return the value for the given node.

async set(path, valu, nexs=False)[source]: A set operation at the hive level (full path).

synapse.lib.hive.iterpath(path)[source]

async synapse.lib.hive.opendir(dirn, conf=None)[source]

async synapse.lib.hive.openurl(url, **opts)[source]

synapse.lib.hiveauth module

class synapse.lib.hiveauth.Auth[source]

Bases: Pusher

Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.

Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).

Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.

Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.

Node layout:

Auth root (passed into constructor)
├ roles
│   ├ <role iden 1>
│   ├ ...
│   └ last role
├ users
│   ├ <user iden 1>
│   ├ ...
│   └ last user
└ authgates
    ├ <iden 1>
    │   ├ roles
    │   │   ├ <role iden 1>
    │   │   ├ ...
    │   │   └ last role
    │   └ users
    │       ├ <user iden 1>
    │       ├ ...
    │       └ last user
    ├ <iden 2>
    │   ├ ...
    └ ... last authgate

async addAuthGate(iden, authgatetype)[source]

Retrieve AuthGate by iden. Create if not present.

Note

Not change distributed

Returns:: (HiveAuthGate)

async addRole(name, iden=None)[source]

async addUser(name, passwd=None, email=None, iden=None)[source]

Add a User to the Hive.

Parameters:

name (str) – The name of the User.
passwd (str) – A optional password for the user.
email (str) – A optional email for the user.
iden (str) – A optional iden to use as the user iden.

Returns:

A Hive User.

Return type:

HiveUser

checkUserLimit()[source]

Check if we’re at the specified user limit.

This should be called right before adding/unlocking/unarchiving a user.

Raises: s_exc.HitLimit if the number of active users is at the maximum.

async delAuthGate(iden)[source]: Delete AuthGate by iden.

Note

Not change distributed

async delRole(iden)[source]

async delUser(iden)[source]

async feedBeholder(evnt, info, gateiden=None, logged=True)[source]

getAuthGate(iden)[source]

getAuthGates()[source]

async getRoleByName(name)[source]

async getUserByName(name)[source]

Get a user by their username.

Parameters:: name (str) – Name of the user to get.
Returns:: A Hive User. May return None if there is no user by the requested name.
Return type:: HiveUser

async getUserIdenByName(name)[source]

reqAuthGate(iden)[source]

async reqRole(iden)[source]

async reqRoleByName(name)[source]

async reqUser(iden)[source]

async reqUserByName(name)[source]

async reqUserByNameOrIden(name)[source]

role(iden)[source]

roles()[source]

async setRoleInfo(iden, name, valu, gateiden=None, logged=True, mesg=None)[source]

async setRoleName(iden, name)[source]

async setUserInfo(iden, name, valu, gateiden=None, logged=True, mesg=None)[source]

async setUserName(iden, name)[source]

user(iden)[source]

users()[source]

class synapse.lib.hiveauth.AuthGate[source]

Bases: Base

The storage object for object specific rules for users/roles.

async delete()[source]

async genRoleInfo(iden)[source]

async genUserInfo(iden)[source]

pack()[source]

class synapse.lib.hiveauth.HiveRole[source]

Bases: HiveRuler

A role within the Hive authorization subsystem.

A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.

allowed(perm, default=None, gateiden=None)[source]

clearAuthCache()[source]

async genGateInfo(gateiden)[source]

pack()[source]

async setName(name)[source]

class synapse.lib.hiveauth.HiveRuler[source]

Bases: Base

A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those

async addRule(rule, indx=None, gateiden=None, nexs=True)[source]

async delRule(rule, gateiden=None)[source]

getRules(gateiden=None)[source]

async setRules(rules, gateiden=None, nexs=True, mesg=None)[source]

class synapse.lib.hiveauth.HiveUser[source]

Bases: HiveRuler

A user (could be human or computer) of the system within HiveAuth.

Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.

async allow(perm)[source]

allowed(perm, default=None, gateiden=None)[source]

clearAuthCache()[source]

confirm(perm, default=None, gateiden=None)[source]

async genGateInfo(gateiden)[source]

getAllowedReason(perm, default=None, gateiden=None)[source]: A routine which will return a tuple of (allowed, info).

getRoles()[source]

async grant(roleiden, indx=None)[source]

hasRole(iden)[source]

isAdmin(gateiden=None)[source]

isArchived()[source]

isLocked()[source]

pack(packroles=False)[source]

raisePermDeny(perm, gateiden=None)[source]

reqAdmin(gateiden=None, mesg=None)[source]

async revoke(iden, nexs=True)[source]

async setAdmin(admin, gateiden=None, logged=True)[source]

async setArchived(archived)[source]

async setLocked(locked, logged=True)[source]

async setName(name)[source]

async setPasswd(passwd, nexs=True)[source]

async setRoles(roleidens)[source]

Replace all the roles for a given user with a new list of roles.

Parameters:: roleidens (list) – A list of roleidens.

Notes

The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.

Returns:: None

async tryPasswd(passwd, nexs=True)[source]

synapse.lib.hiveauth.getShadow(passwd)[source]: This API is deprecated.

synapse.lib.hiveauth.textFromRule(rule)[source]

synapse.lib.httpapi module

class synapse.lib.httpapi.ActiveV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.AuthAddRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.AuthAddUserV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.AuthDelRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.AuthGrantV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

/api/v1/auth/grant?user=iden&role=iden

async get()[source]

async post()[source]

class synapse.lib.httpapi.AuthRevokeV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

/api/v1/auth/grant?user=iden&role=iden

async get()[source]

async post()[source]

class synapse.lib.httpapi.AuthRoleV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get(iden)[source]

async post(iden)[source]

class synapse.lib.httpapi.AuthRolesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.AuthUserPasswdV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post(iden)[source]

class synapse.lib.httpapi.AuthUserV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get(iden)[source]

async post(iden)[source]

class synapse.lib.httpapi.AuthUsersV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.BeholdSockV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: WebSocket

async onInitMessage(byts)[source]

async on_message(byts)[source]

Handle incoming messages on the WebSocket

This method must be overridden.

Changed in version 4.5: on_message can be a coroutine.

class synapse.lib.httpapi.CoreInfoV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

/api/v1/core/info

async get()[source]

class synapse.lib.httpapi.ExtApiHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

/api/ext/.*

compute_etag()[source]

Computes the etag header to be used for this request.

By default uses a hash of the content written so far.

May be overridden to provide custom etag implementations, or may return None to disable tornado’s default etag support.

async delete(path)[source]

async get(path)[source]

async head(path)[source]

async options(path)[source]

async patch(path)[source]

async post(path)[source]

async put(path)[source]

set_default_headers()[source]

Override this to set HTTP headers at the beginning of the request.

For example, this is the place to set a custom Server header. Note that setting such headers in the normal flow of request processing may not do what you want, since headers may be reset during error handling.

storm_prefix = 'init { $request = $lib.cortex.httpapi.response($_http_request_info) }'

class synapse.lib.httpapi.FeedV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

/api/v1/feed

Examples

Example data:

{
    'name': 'syn.nodes',
    'view': null,
    'items': [...],
}

async post()[source]

class synapse.lib.httpapi.Handler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: HandlerBase, RequestHandler

on_connection_close()[source]

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

prepare()[source]

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

class synapse.lib.httpapi.HandlerBase[source]

Bases: object

async allowed(perm, default=False, gateiden=None)[source]

Check if the authenticated user has the given permission.

Parameters:

perm (tuple) – The permission tuple to check.
default (boolean) – The default value for the permission.
gateiden (str) – The gateiden to check the permission against.

Notes

This API sets up HTTP response values if it returns False.

Returns:: True if the user has the requested permission.
Return type:: bool

async authenticated()[source]

Check if the request has an authenticated user or not.

Returns:: True if the request has an authenticated user, false otherwise.
Return type:: bool

check_origin(origin)[source]

getAuthCell()[source]: Return a reference to the cell used for auth operations.

getCustomHeaders()[source]

getJsonBody(validator=None)[source]

async getUseridenBody(validator=None)[source]

Helper function to confirm that there is an auth user and a valid JSON body in the request.

Parameters:: validator – Validator function run on the deserialized JSON body.
Returns:: The user definition and body of the request as deserialized JSON, or a tuple of s_common.novalu objects if there was no user or json body.
Return type:: (str, object)

async handleApiKeyAuth()[source]

async handleBasicAuth()[source]

Handle basic authentication in the handler.

Notes

Implementors may override this to disable or implement their own basic auth schemes. This is expected to set web_useriden and web_username upon successful authentication.

Returns:: The user iden of the logged in user.
Return type:: str

initialize(cell)[source]

isOrigHost(origin)[source]

async isUserAdmin()[source]

Check if the current authenticated user is an admin or not.

Returns:: True if the user is an admin, false otherwise.
Return type:: bool

loadJsonMesg(byts, validator=None)[source]

logAuthIssue(mesg=None, user=None, username=None, level=30)[source]

Helper to log issues related to request authentication.

Parameters:

mesg (str) – Additional message to log.
user (str) – User iden, if available.
username (str) – Username, if available.
level (int) – Logging level to log the message at. Defaults to logging.WARNING.

Returns:

None

options()[source]

async reqAuthAdmin()[source]

Require the current authenticated user to be an admin.

Notes

If this returns False, an error message has already been sent and no additional processing for the request should be done.

Returns:: True if the user is an admin, false otherwise.
Return type:: bool

async reqAuthUser()[source]

sendAuthRequired()[source]

sendRestErr(code, mesg)[source]

sendRestExc(e)[source]

sendRestRetn(valu)[source]

async sess(gen=True)[source]

Get the heavy Session object for the request.

Parameters:: gen (bool) – If set to True, generate a new session if there is no sess cookie.

Notes

This stores the identifier in the sess cookie for with a 14 day expiration, stored in the Cell. Valid requests with that sess cookie will resolve to the same Session object.

Returns:: A heavy session object. If the sess cookie is invalid or gen is false, this returns None.
Return type:: Sess

set_default_headers()[source]

async useriden()[source]

Get the user iden of the current session user.

Note

This function will pull the iden from the current session, or attempt to resolve the useriden with basic authentication.

Returns:: The iden of the current session user.
Return type:: str

class synapse.lib.httpapi.HealthCheckV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.LoginV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.LogoutV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.ModelNormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

async post()[source]

class synapse.lib.httpapi.ModelV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.OnePassIssueV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

/api/v1/auth/onepass/issue

async post()[source]

class synapse.lib.httpapi.ReqValidStormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

async get()[source]

async post()[source]

class synapse.lib.httpapi.RobotHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: HandlerBase, RequestHandler

async get()[source]

class synapse.lib.httpapi.Sess[source]

Bases: Base

addWebSock(sock)[source]

delWebSock(sock)[source]

async login(user)[source]

async logout()[source]

async set(name, valu)[source]

async update(vals: dict)[source]

class synapse.lib.httpapi.StormCallV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

async get()[source]

async post()[source]

class synapse.lib.httpapi.StormExportV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

async get()[source]

async post()[source]

class synapse.lib.httpapi.StormHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

getCore()[source]

class synapse.lib.httpapi.StormNodesV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

async get()[source]

async post()[source]

class synapse.lib.httpapi.StormV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: StormHandler

async get()[source]

async post()[source]

class synapse.lib.httpapi.StormVarsGetV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async get()[source]

class synapse.lib.httpapi.StormVarsPopV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.StormVarsSetV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

async post()[source]

class synapse.lib.httpapi.StreamHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: Handler

Subclass for Tornado streaming uploads.

Notes

Async method prepare() is called after headers are read but before body processing.
Sync method on_finish() can be used to cleanup after a request.
Sync method on_connection_close() can be used to cleanup after a client disconnect.
Async methods post(), put(), etc are called after the streaming has completed.

async data_received(chunk)[source]

Implement this method to handle streamed request data.

Requires the .stream_request_body decorator.

May be a coroutine for flow control.

class synapse.lib.httpapi.WebSocket(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: HandlerBase, WebSocketHandler

async xmit(name, **info)[source]

synapse.lib.ingest module

synapse.lib.interval module

A few utilities for dealing with intervals.

synapse.lib.interval.fold(*vals)[source]

Initialize a new (min,max) tuple interval from values.

Parameters:: *vals ([int,...]) – A list of values (or Nones)
Returns:: A (min,max) interval tuple or None
Return type:: ((int,int))

synapse.lib.interval.overlap(ival0, ival1)[source]

Determine if two interval tuples have overlap.

Parameters:

ival0 ((int,int)) – An interval tuple
ival1 ((int,int)) –

Returns:

True if the intervals overlap, otherwise False

Return type:

(bool)

synapse.lib.interval.parsetime(text)[source]

Parse an interval time string and return a (min,max) tuple.

Parameters:: text (str) – A time interval string
Returns:: A epoch millis epoch time string
Return type:: ((int,int))

synapse.lib.jsonstor module

class synapse.lib.jsonstor.JsonStor[source]

Bases: Base

A filesystem like storage mechanism that allows hirarchical lookup of reference counted “objects” that have individually editable properties.

#TODO json validation by path glob matches? (persists?) #TODO GUID ACCESS with index generation by type #TODO registered types jsonschema with optional write-back validation

async cmpDelPathObjProp(path, prop, valu)[source]

async copyPathObj(oldp, newp)[source]

async copyPathObjs(paths)[source]

async delPathObj(path)[source]: Remove a path and decref the object it references.

async delPathObjProp(path, prop)[source]

async getPathList(path)[source]

async getPathObj(path)[source]

async getPathObjProp(path, prop)[source]

async getPathObjs(path)[source]

async hasPathObj(path)[source]

async popPathObjProp(path, prop, defv=None)[source]

async setPathLink(srcpath, dstpath)[source]: Add a link from the given srcpath to the dstpath. NOTE: This causes the item at dstpath to be incref’d

async setPathObj(path, item)[source]

Set (and/or reinitialize) the object at the given path.

NOTE: This will break any links by creating a new object.

async setPathObjProp(path, prop, valu)[source]

class synapse.lib.jsonstor.JsonStorApi[source]

Bases: CellApi

async addQueue(name, info)[source]

addUserNotif(useriden, mesgtype, mesgdata=None)[source]

async cmpDelPathObjProp(path, name, valu)[source]

async copyPathObj(oldp, newp)[source]

async copyPathObjs(paths)[source]

async cullQueue(name, offs)[source]

async delPathObj(path)[source]

async delPathObjProp(path, name)[source]

async delQueue(name)[source]

delUserNotif(indx)[source]

async getPathList(path)[source]

async getPathObj(path)[source]

async getPathObjProp(path, prop)[source]

async getPathObjs(path)[source]

getUserNotif(indx)[source]

async getsQueue(name, offs, size=None, cull=True, wait=True)[source]

async hasPathObj(path)[source]

iterUserNotifs(useriden, size=None)[source]

async popPathObjProp(path, prop)[source]

async putsQueue(name, items)[source]

async setPathLink(srcpath, dstpath)[source]

async setPathObj(path, item)[source]

async setPathObjProp(path, prop, valu)[source]

watchAllUserNotifs(offs=None)[source]

class synapse.lib.jsonstor.JsonStorCell[source]

Bases: Cell

async addQueue(name, info)[source]

async addUserNotif(useriden, mesgtype, mesgdata=None)[source]

cellapi: alias of JsonStorApi

async cmpDelPathObjProp(path, name, valu)[source]

async copyPathObj(oldp, newp)[source]

async copyPathObjs(paths)[source]

async cullQueue(name, offs)[source]

async delPathObj(path)[source]

async delPathObjProp(path, name)[source]

async delQueue(name)[source]

async delUserNotif(indx)[source]

classmethod getEnvPrefix()[source]: Get a list of envar prefixes for config resolution.

async getPathList(path)[source]

async getPathObj(path)[source]

async getPathObjProp(path, prop)[source]

async getPathObjs(path)[source]

async getUserNotif(indx)[source]

async getsQueue(name, offs, size=None, cull=True, wait=True)[source]

async hasPathObj(path)[source]

async initServiceStorage()[source]

async iterUserNotifs(useriden, size=None)[source]

async popPathObjProp(path, prop)[source]

async putsQueue(name, items)[source]

async setPathLink(srcpath, dstpath)[source]

async setPathObj(path, item)[source]

async setPathObjProp(path, prop, valu)[source]

async watchAllUserNotifs(offs=None)[source]

synapse.lib.jupyter module

class synapse.lib.jupyter.CmdrCore[source]

Bases: Base

A helper for jupyter/cmdr CLI interaction

async addFeedData(name, items, *, viewiden=None)[source]: Add feed data to the cortex.

async eval(text, opts=None, num=None, cmdr=False)[source]

A helper for executing a storm command and getting a list of packed nodes.

Parameters:

text (str) – Storm command to execute.
opts (dict) – Opt to pass to the cortex during execution.
num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.
cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True.

Returns:: A list of packed nodes.
Return type:: list

async runCmdLine(text)[source]: Run a line of text directly via cmdr.

async storm(text, opts=None, num=None, cmdr=False, suppress_logging=False)[source]

A helper for executing a storm command and getting a list of storm messages.

Parameters:

text (str) – Storm command to execute.
opts (dict) – Opt to pass to the cortex during execution.
num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.
cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.
suppress_logging (bool) – If True, suppresses some logging related to Storm runtime exceptions.

Notes

The opts dictionary will not be used if cmdr=True.

Returns:: A list of storm messages.
Return type:: list

suppress_logging(suppress)[source]: Context manager to suppress specific loggers.

class synapse.lib.jupyter.StormCore[source]

Bases: Base

A helper for jupyter/storm CLI interaction

async runCmdLine(text, opts=None)[source]: Run a line of text directly via storm cli.

async storm(text, opts=None, num=None, cli=False, suppress_logging=False)[source]

A helper for executing a storm command and getting a list of storm messages.

Parameters:

text (str) – Storm command to execute.
opts (dict) – Opt to pass to the cortex during execution.
num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.
cli (bool) – If True, executes the line via the Storm CLI and will send output to outp.
suppress_logging (bool) – If True, suppresses some logging related to Storm runtime exceptions.

Notes

The opts dictionary will not be used if cmdr=True.

Returns:: A list of storm messages.
Return type:: list

suppress_logging(suppress)[source]: Context manager to suppress specific loggers.

synapse.lib.jupyter.genTempCoreProxy(mods=None)[source]: Get a temporary cortex proxy.

synapse.lib.jupyter.genTempStormsvcProxy(cmdrcore, svcname, svcctor, conf=None)[source]

synapse.lib.jupyter.getDocData(fp, root=None)[source]

Parameters:

fp (str) – Name of the file to retrieve the data of.
root (str) – Optional root path to look for a docdata directory in.

Notes

Will detect json/jsonl/yaml/mpk extensions and automatically decode that data if found; otherwise it returns bytes.

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns:: May be deserialized data or bytes.
Return type:: data
Raises:: ValueError if the file does not exist or directory traversal attempted.. –

synapse.lib.jupyter.getDocPath(fn, root=None)[source]

Helper for getting a documentation data file paths.

Parameters:

fn (str) – Name of the file to retrieve the full path for.
root (str) – Optional root path to look for a docdata in.

Notes

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns:: A file path.
Return type:: str
Raises:: ValueError if the file does not exist or directory traversal attempted.. –

async synapse.lib.jupyter.getItemCmdr(prox, outp=None, locs=None)[source]: Get a Cmdr instance with prepopulated locs

async synapse.lib.jupyter.getItemStorm(prox, outp=None)[source]: Get a Storm CLI instance with prepopulated locs

async synapse.lib.jupyter.getTempCoreCmdr(mods=None, outp=None)[source]

Get a CmdrCore instance which is backed by a temporary Cortex.

Parameters:

mods (list) – A list of additional CoreModules to load in the Cortex.
outp – A output helper. Will be used for the Cmdr instance.

Notes

The CmdrCore returned by this should be fini()’d to tear down the temporary Cortex.

Returns:: A CmdrCore instance.
Return type:: CmdrCore

async synapse.lib.jupyter.getTempCoreCmdrStormsvc(svcname, svcctor, svcconf=None, outp=None)[source]

Get a proxy to a Storm service and a CmdrCore instance backed by a temporary Cortex with the service added.

Parameters:

svcname (str) – Storm service name
svcctor – Storm service constructor (e.g. Example.anit)
svcconf – Optional conf for the Storm service
outp – A output helper for the Cmdr instance

Notes

Both the CmdrCore and Storm service proxy should be fini()’d for proper teardown

Returns:: A CmdrCore instance and proxy to the Storm service
Return type:: (CmdrCore, Proxy)

async synapse.lib.jupyter.getTempCoreProx(mods=None)[source]

Get a Telepath Proxt to a Cortex instance which is backed by a temporary Cortex.

Parameters:: mods (list) – A list of additional CoreModules to load in the Cortex.

Notes

The Proxy returned by this should be fini()’d to tear down the temporary Cortex.

Returns:: s_telepath.Proxy

async synapse.lib.jupyter.getTempCoreStorm(mods=None, outp=None)[source]

Get a StormCore instance which is backed by a temporary Cortex.

Parameters:

mods (list) – A list of additional CoreModules to load in the Cortex.
outp – A output helper. Will be used for the Cmdr instance.

Notes

The StormCore returned by this should be fini()’d to tear down the temporary Cortex.

Returns:: A StormCore instance.
Return type:: StormCore

async synapse.lib.jupyter.getTempCoreStormStormsvc(svcname, svcctor, svcconf=None, outp=None)[source]

Get a proxy to a Storm service and a StormCore instance backed by a temporary Cortex with the service added.

Parameters:

svcname (str) – Storm service name
svcctor – Storm service constructor (e.g. Example.anit)
svcconf – Optional conf for the Storm service
outp – A output helper for the Cmdr instance

Notes

Both the StormCore and Storm service proxy should be fini()’d for proper teardown

Returns:: A StormCore instance and proxy to the Storm service
Return type:: (StormCore, Proxy)

synapse.lib.jupyter.suppress_logging(suppress)[source]: Context manager to suppress specific loggers.

synapse.lib.layer module

The Layer 2.0 archtecture introduces several optimized node/message serialization formats used by the layers to optimize returning primitives and facilitate efficient node construction:

Note

This interface is subject to change between minor revisions.

Storage Types (<stortype>)

In Layers 2.0, each node property from the model has an associated “storage type”. Each storage type determines how the data is indexed and represented within the Layer. This formalizes the separation of “data model” from “storage model”. Each data model type has a “stortype” property which coresponds to one of the STOR_TYPE_XXX values. The knowledge of the mapping of data model types to storage types is the responsibility of the data model, making the Layer implementation fully decoupled from the data model.

Node Edits / Edits

A node edit consists of a (<buid>, <form>, [edits]) tuple. An edit is Tuple of (<type>, <info>, List[NodeEdits]) where the first element is an int that matches to an EDIT_* constant below, the info is a tuple that varies depending on the first element, and the third element is a list of dependent NodeEdits that will only be applied if the edit actually makes a change.

Storage Node (<sode>)

A storage node is a layer/storage optimized node representation which is similar to a “packed node”. A storage node may be partial ( as it is produced by a given layer ) and are joined by the view/snap into “full” storage nodes which are used to construct Node() instances.

Sode format:
(<buid>, {

    'ndef': (<formname>, <formvalu>),

    'props': {
        <propname>: <propvalu>,
    }

    'tags': {
        <tagname>: <tagvalu>,
    }

    'tagprops: {
        <tagname>: {
            <propname>: <propvalu>,
        },
    }

    # changes that were *just* made.
    'edits': [
        <edit>
    ]

}),

class synapse.lib.layer.IndxBy(layr, abrv, db)[source]

Bases: object

IndxBy sub-classes encapsulate access methods and encoding details for various types of properties within the layer to be lifted/compared by storage types.

buidsByDups(indx)[source]

buidsByPref(indx=b'')[source]

buidsByRange(minindx, maxindx)[source]

buidsByRangeBack(minindx, maxindx)[source]

getNodeValu(buid)[source]

hasIndxBuid(indx, buid)[source]

keyBuidsByDups(indx)[source]

keyBuidsByDupsBack(indx)[source]

keyBuidsByPref(indx=b'')[source]

keyBuidsByPrefBack(indx=b'')[source]

keyBuidsByRange(minindx, maxindx)[source]

keyBuidsByRangeBack(minindx, maxindx)[source]: Yields backwards from maxindx to minindx

scanByDups(indx)[source]

scanByPref(indx=b'')[source]

scanByPrefBack(indx=b'')[source]

scanByRange(minindx, maxindx)[source]

scanByRangeBack(minindx, maxindx)[source]

class synapse.lib.layer.IndxByForm(layr, form)[source]

Bases: IndxBy

getNodeValu(buid)[source]

class synapse.lib.layer.IndxByProp(layr, form, prop)[source]

Bases: IndxBy

getNodeValu(buid)[source]

class synapse.lib.layer.IndxByPropArray(layr, form, prop)[source]

Bases: IndxBy

getNodeValu(buid)[source]

class synapse.lib.layer.IndxByTag(layr, form, tag)[source]

Bases: IndxBy

getNodeValuForm(buid)[source]

class synapse.lib.layer.IndxByTagProp(layr, form, tag, prop)[source]

Bases: IndxBy

getNodeValu(buid)[source]

class synapse.lib.layer.Layer[source]

Bases: Pusher

The base class for a cortex layer.

async clone(newdirn)[source]: Copy the contents of this layer to a new layer

async delete()[source]: Delete the underlying storage

getAbrvProp(abrv)[source]

async getEdgeVerbs()[source]

async getEdges(verb=None)[source]

async getEditIndx()[source]: Returns what will be the next (i.e. 1 past the last) nodeedit log index.

async getEditOffs()[source]: Return the offset of the last recorded log entry. Returns -1 if nodeedit log is disabled or empty.

async getEditSize()[source]

async getFormCounts()[source]

getFormProps()[source]

getIdenFutu(iden=None)[source]

async getLayerSize()[source]: Get the total storage size for the layer.

async getMirrorStatus()[source]

async getModelVers()[source]

async getNodeData(buid, name)[source]: Return a single element of a buid’s node data

getNodeEditWindow()[source]

async getNodeForm(buid)[source]

async getNodeTag(buid, tag)[source]

async getNodeValu(buid, prop=None)[source]: Retrieve either the form valu or a prop valu for the given node by buid.

getPropAbrv(form, prop)[source]

async getPropArrayCount(formname, propname=None)[source]: Return the number of invidiual value rows in the layer for the given array form/prop.

getPropArrayValuCount(formname, propname, stortype, valu)[source]

async getPropCount(formname, propname=None, maxsize=None)[source]: Return the number of property rows in the layer for the given form/prop.

getPropValuCount(formname, propname, stortype, valu)[source]

getStorIndx(stortype, valu)[source]

async getStorNode(buid)[source]

getStorNodeCount()[source]

async getStorNodes()[source]: Yield (buid, sode) tuples for all the nodes with props/tags/tagprops stored in this layer.

async getTagCount(tagname, formname=None)[source]: Return the number of tag rows in the layer for the given tag/form.

getTagPropAbrv(*args)[source]

async getTagPropCount(form, tag, prop)[source]: Return the number of property rows in the layer for the given form/tag/prop.

getTagPropValuCount(form, tag, prop, stortype, valu)[source]

getTagProps()[source]

async getUnivPropCount(propname, maxsize=None)[source]: Return the number of universal property rows in the layer for the given prop.

async hasNodeData(buid, name)[source]

async hasNodeEdge(buid1, verb, buid2)[source]

async hasTagProp(name)[source]

async initLayerActive()[source]

async initLayerPassive()[source]

async initUpstreamSync(url)[source]

async iterEdgeVerbs(n1buid, n2buid)[source]

async iterFormRows(form, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes of a single form, optionally (re)starting at startvalu.

Parameters:

form (str) – A form name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns:

AsyncIterator[Tuple(buid, valu)]

async iterLayerNodeEdits()[source]: Scan the full layer and yield artificial sets of nodeedits.

async iterNodeData(buid)[source]: Return a generator of all a buid’s node data

async iterNodeDataKeys(buid)[source]: Return a generator of all a buid’s node data keys

async iterNodeEdgesN1(buid, verb=None)[source]

async iterNodeEdgesN2(buid, verb=None)[source]

async iterNodeEditLog(offs=0)[source]: Iterate the node edit log and yield (offs, edits, meta) tuples.

async iterNodeEditLogBack(offs=0)[source]: Iterate the node edit log and yield (offs, edits, meta) tuples in reverse.

async iterPropRows(form, prop, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes with a particular secondary property, optionally (re)starting at startvalu.

Parameters:

form (str) – A form name.
prop (str) – A universal property name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns:

AsyncIterator[Tuple(buid, valu)]

async iterTagPropRows(tag, prop, form=None, stortype=None, startvalu=None)[source]

Yields (buid, valu) that match a tag:prop, optionally (re)starting at startvalu.

Parameters:

tag (str) – tag name
prop (str) – prop name
form (Optional[str]) – optional form name
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns:

AsyncIterator[Tuple(buid, valu)]

async iterTagRows(tag, form=None, starttupl=None)[source]

Yields (buid, (valu, form)) values that match a tag and optional form, optionally (re)starting at starttupl.

Parameters:

tag (str) – the tag to match
form (Optional[str]) – if present, only yields buids of nodes that match the form.
starttupl (Optional[Tuple[buid, form]]) – if present, (re)starts the stream of values there.

Returns:

AsyncIterator[Tuple(buid, (valu, form))]

Note

This yields (buid, (tagvalu, form)) instead of just buid, valu in order to allow resuming an interrupted call by feeding the last value retrieved into starttupl

async iterUnivRows(prop, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes with a particular universal property, optionally (re)starting at startvalu.

Parameters:

prop (str) – A universal property name.
stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns:

AsyncIterator[Tuple(buid, valu)]

async iterWipeNodeEdits()[source]

async liftByDataName(name)[source]

async liftByFormValu(form, cmprvals, reverse=False)[source]

async liftByProp(form, prop, reverse=False)[source]

async liftByPropArray(form, prop, cmprvals, reverse=False)[source]

async liftByPropValu(form, prop, cmprvals, reverse=False)[source]

async liftByTag(tag, form=None, reverse=False)[source]

async liftByTagProp(form, tag, prop, reverse=False)[source]

async liftByTagPropValu(form, tag, prop, cmprvals, reverse=False)[source]: Note: form may be None

async liftByTagValu(tag, cmpr, valu, form=None, reverse=False)[source]

async liftTagProp(name)[source]

mayDelBuid(buid, sode)[source]

nodeeditctor: alias of SlabSeqn

async pack()[source]

async saveNodeEdits(edits, meta)[source]

Save node edits to the layer and return a tuple of (nexsoffs, changes).

Note: nexsoffs will be None if there are no changes.

async setLayerInfo(name, valu)[source]

async setModelVers(vers)[source]

setPropAbrv(form, prop)[source]

setSodeDirty(buid, sode, form)[source]

setTagPropAbrv(*args)[source]

async stat()[source]

async storNodeEdits(nodeedits, meta)[source]

async storNodeEditsNoLift(nodeedits, meta)[source]

Execute a series of node edit operations.

Does not return the updated nodes.

async syncIndexEvents(offs, matchdef, wait=True)[source]

Yield (offs, (buid, form, ETYPE, VALS, META)) tuples from the nodeedit log starting from the given offset. Only edits that match the filter in matchdef will be yielded.

Notes

ETYPE is an constant EDIT_* above. VALS is a tuple whose format depends on ETYPE, outlined in the comment next to the constant. META is a dict that may contain keys ‘user’ and ‘time’ to represent the iden of the user that initiated the change, and the time that it took place, respectively.

Additionally, every 1000 entries, an entry (offs, (None, None, EDIT_PROGRESS, (), ())) message is emitted.

The matchdef dict may contain the following keys: forms, props, tags, tagprops. The value must be a sequence of strings. Each key/val combination is treated as an “or”, so each key and value yields more events. forms: EDIT_NODE_ADD and EDIT_NODE_DEL events. Matches events for nodes with forms in the value list. props: EDIT_PROP_SET and EDIT_PROP_DEL events. Values must be in form:prop or .universal form tags: EDIT_TAG_SET and EDIT_TAG_DEL events. Values must be the raw tag with no #. tagprops: EDIT_TAGPROP_SET and EDIT_TAGPROP_DEL events. Values must be just the prop or tag:prop.

Will not yield any values if this layer was not created with logedits enabled

Parameters:

offs (int) – starting nexus/editlog offset
matchdef (Dict[str, Sequence[str]]) – a dict describing which events are yielded
wait (bool) – whether to pend and stream value until this layer is fini’d

async syncNodeEdits(offs, wait=True)[source]: Identical to syncNodeEdits2, but doesn’t yield meta

async syncNodeEdits2(offs, wait=True)[source]

Once caught up with storage, yield them in realtime.

Returns:: Tuple of offset(int), nodeedits, meta(dict)

async verify(config=None)[source]

async verifyAllBuids(scanconf=None)[source]

async verifyAllProps(scanconf=None)[source]

async verifyAllTagProps(scanconf=None)[source]

async verifyAllTags(scanconf=None)[source]

async verifyBuidTag(buid, formname, tagname, tagvalu)[source]

async verifyByBuid(buid, sode)[source]

async verifyByProp(form, prop, autofix=None)[source]

async verifyByPropArray(form, prop, autofix=None)[source]

async verifyByTag(tag, autofix=None)[source]

async verifyByTagProp(form, tag, prop, autofix=None)[source]

async waitEditOffs(offs, timeout=None)[source]: Wait for the node edit log to write an entry at/past the given offset.

async waitForHot()[source]: Wait for the layer’s slab to be prefaulted and locked into memory if lockmemory is true, otherwise return.

async waitUpstreamOffs(iden, offs)[source]

class synapse.lib.layer.LayerApi[source]

Bases: CellApi

async getEditIndx()[source]: Returns what will be the next nodeedit log index.

async getEditSize()[source]: Return the total number of (edits, meta) pairs in the layer changelog.

async getIden()[source]

async iterLayerNodeEdits()[source]: Scan the full layer and yield artificial nodeedit sets.

saveNodeEdits(edits, meta)[source]

Save node edits to the layer and return a tuple of (nexsoffs, changes).

Note: nexsoffs will be None if there are no changes.

async storNodeEdits(nodeedits, meta=None)[source]

async storNodeEditsNoLift(nodeedits, meta=None)[source]

async syncNodeEdits(offs, wait=True)[source]

Yield (offs, nodeedits) tuples from the nodeedit log starting from the given offset.

Once caught up with storage, yield them in realtime.

async syncNodeEdits2(offs, wait=True)[source]

class synapse.lib.layer.StorType(layr, stortype)[source]

Bases: object

decodeIndx(valu)[source]

indx(valu)[source]

async indxBy(liftby, cmpr, valu, reverse=False)[source]

async indxByForm(form, cmpr, valu, reverse=False)[source]

async indxByProp(form, prop, cmpr, valu, reverse=False)[source]

async indxByPropArray(form, prop, cmpr, valu, reverse=False)[source]

async indxByTagProp(form, tag, prop, cmpr, valu, reverse=False)[source]

async verifyBuidProp(buid, form, prop, valu)[source]

class synapse.lib.layer.StorTypeFloat(layr, stortype, size=8)[source]

Bases: StorType

FloatPackNegMax = b'\x80\x00\x00\x00\x00\x00\x00\x00'

FloatPackNegMin = b'\xff\xf0\x00\x00\x00\x00\x00\x00'

FloatPackPosMax = b'\x7f\xf0\x00\x00\x00\x00\x00\x00'

FloatPackPosMin = b'\x00\x00\x00\x00\x00\x00\x00\x00'

FloatPacker = <_struct.Struct object>

decodeIndx(bytz)[source]

fpack()

S.pack(v1, v2, …) -> bytes

Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.

indx(valu)[source]

class synapse.lib.layer.StorTypeFqdn(layr)[source]

Bases: StorTypeUtf8

decodeIndx(bytz)[source]

indx(norm)[source]

class synapse.lib.layer.StorTypeGuid(layr)[source]

Bases: StorType

decodeIndx(bytz)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeHier(layr, stortype, sepr='.')[source]

Bases: StorType

decodeIndx(bytz)[source]

getHierIndx(valu)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeHugeNum(layr, stortype)[source]

Bases: StorType

decodeIndx(bytz)[source]

getHugeIndx(norm)[source]

indx(norm)[source]

class synapse.lib.layer.StorTypeInt(layr, stortype, size, signed)[source]

Bases: StorType

decodeIndx(bytz)[source]

getIntIndx(valu)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeIpv6(layr)[source]

Bases: StorType

decodeIndx(bytz)[source]

getIPv6Indx(valu)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeIval(layr)[source]

Bases: StorType

decodeIndx(bytz)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeLatLon(layr)[source]

Bases: StorType

decodeIndx(bytz)[source]

indx(valu)[source]

class synapse.lib.layer.StorTypeLoc(layr)[source]: Bases: StorTypeHier

class synapse.lib.layer.StorTypeMsgp(layr)[source]

Bases: StorType

indx(valu)[source]

class synapse.lib.layer.StorTypeTag(layr)[source]

Bases: StorTypeHier

static getTagFilt(cmpr, valu)[source]

class synapse.lib.layer.StorTypeTime(layr)[source]: Bases: StorTypeInt

class synapse.lib.layer.StorTypeUtf8(layr)[source]

Bases: StorType

decodeIndx(bytz)[source]

indx(valu)[source]

synapse.lib.layer.getFlatEdits(nodeedits)[source]

synapse.lib.layer.getNodeEditPerms(nodeedits)[source]: Yields (offs, perm) tuples that can be used in user.allowed()

synapse.lib.link module

class synapse.lib.link.Link[source]

Bases: Base

A Link() is created to wrap a socket reader/writer.

feed(byts)[source]: Used by Plex() to unpack bytes.

get(name, defval=None)[source]: Get a property from the Link info.

getAddrInfo()[source]: Get a summary of address information related to the link.

async getSpawnInfo()[source]

getTlsPeerCn()[source]

async recv(size)[source]

async recvsize(size)[source]

async rx()[source]

async send(byts)[source]

set(name, valu)[source]: Set a property in the Link info.

async tx(mesg)[source]: Async transmit routine which will wait for writer drain().

txfini()[source]

async synapse.lib.link.connect(host, port, ssl=None, hostname=None, linkinfo=None)[source]: Async connect and return a Link().

async synapse.lib.link.fromspawn(spawninfo)[source]

async synapse.lib.link.linkfile(mode='wb')[source]: Connect a socketpair to a file-object and return (link, file).

async synapse.lib.link.linksock(forceclose=False)[source]: Connect a Link, socket pair.

async synapse.lib.link.listen(host, port, onlink, ssl=None)[source]

Listen on the given host/port and fire onlink(Link).

Returns a server object that contains the listening sockets

async synapse.lib.link.unixconnect(path)[source]: Connect to a PF_UNIX server listening on the given path.

async synapse.lib.link.unixlisten(path, onlink)[source]: Start an PF_UNIX server listening on the given path.

synapse.lib.lmdbslab module

class synapse.lib.lmdbslab.GuidStor(slab, name)[source]

Bases: object

async del_(iden)[source]

async dict(iden)[source]

gen(iden)[source]

has(iden)[source]

set(iden, name, valu)[source]

class synapse.lib.lmdbslab.Hist(slab, name)[source]

Bases: object

A class for storing items in a slab by time.

Each added item is inserted into the specified db within the slab using the current epoch-millis time stamp as the key.

add(item, tick=None)[source]

carve(tick, tock=None)[source]

class synapse.lib.lmdbslab.HotCount[source]

Bases: HotKeyVal

Like HotKeyVal, but optimized for integer/count vals

static DecFunc(b): Decode a signed 64-bit int from 8 byte big-endian

static EncFunc(i): Encode a signed 64-bit int into 8 byte big-endian bytes

get(name: str, defv=0)[source]

inc(name: str, valu=1)[source]

set(name: str, valu)[source]

class synapse.lib.lmdbslab.HotKeyVal[source]

Bases: Base

A hot-loop capable keyval that only syncs on commit.

static DecFunc(byts, use_list=False)

Use msgpack to de-serialize a python object.

Parameters:: byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns:: The de-serialized object
Return type:: obj

static EncFunc(item)

Use msgpack to serialize a compatible python object.

Parameters:: item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns:: The serialized bytes in msgpack format.
Return type:: bytes

delete(name: str)[source]

get(name: str, defv=None)[source]

pack()[source]

set(name: str, valu)[source]

sync()[source]

class synapse.lib.lmdbslab.LmdbBackup[source]

Bases: Base

async saveto(dstdir)[source]

class synapse.lib.lmdbslab.MultiQueue[source]

Bases: Base

Allows creation/consumption of multiple durable queues in a slab.

async add(name, info)[source]

async cull(name, offs)[source]: Remove up-to (and including) the queue entry at offs.

async dele(name, minoffs, maxoffs)[source]: Remove queue entries from minoffs, up-to (and including) the queue entry at maxoffs.

exists(name)[source]

async get(name, offs, wait=False, cull=True)[source]: Return (nextoffs, item) tuple or (-1, None) for the given offset.

async gets(name, offs, size=None, cull=False, wait=False)[source]: Yield (offs, item) tuples from the message queue.

list()[source]

offset(name)[source]

async pop(name, offs)[source]: Pop a single entry from the named queue by offset.

async put(name, item, reqid=None)[source]

async puts(name, items, reqid=None)[source]

async rem(name)[source]

async sets(name, offs, items)[source]: Overwrite queue entries with the values in items, starting at offs.

size(name)[source]

status(name)[source]

class synapse.lib.lmdbslab.Scan(slab, db)[source]

Bases: object

A state-object used by Slab. Not to be instantiated directly.

Parameters:

slab (Slab) – which slab the scan is over
db (str) – name of open database on the slab

bump()[source]

first()[source]

isatitem()[source]: Returns if the cursor is at the value in atitem

iterfunc()[source]

iternext()[source]

resume()[source]

set_key(lkey)[source]

set_range(lkey, valu=None)[source]

class synapse.lib.lmdbslab.ScanBack(slab, db)[source]

Bases: Scan

A state-object used by Slab. Not to be instantiated directly.

Scans backwards.

first()[source]

iterfunc()[source]

resume()[source]

set_key(lkey)[source]

set_range(lkey)[source]

class synapse.lib.lmdbslab.ScanKeys(slab, db)[source]

Bases: Scan

An iterator over the keys of the database. If the database is dupsort, a key with multiple values with be yielded once for each value.

isatitem()[source]: Returns if the cursor is at the value in atitem

iterfunc()[source]

iternext()[source]

resume()[source]

class synapse.lib.lmdbslab.Slab[source]

Bases: Base

A “monolithic” LMDB instance for use in a asyncio loop thread.

COMMIT_PERIOD = 0.2

DEFAULT_GROWSIZE = None

DEFAULT_MAPSIZE = 1073741824

WARN_COMMIT_TIME_MS = 1000

addResizeCallback(callback)[source]

allslabs = {}

copydb(sourcedbname, destslab, destdbname=None, progresscb=None)[source]

Copy an entire database in this slab to a new database in potentially another slab.

Parameters:

sourcedbname (str) – name of the db in the source environment
destslab (LmdbSlab) – which slab to copy rows to
destdbname (str) – the name of the database to copy rows to in destslab
progresscb (Callable[int]) – if not None, this function will be periodically called with the number of rows completed

Returns:

the number of rows copied

Return type:

(int)

Note

If any rows already exist in the target database, this method returns an error. This means that one cannot use destdbname=None unless there are no explicit databases in the destination slab.

async copyslab(dstpath, compact=True)[source]

count(lkey, db=None)[source]

async countByPref(byts, db=None, maxsize=None)[source]: Return the number of rows in the given db with the matching prefix bytes.

dbexists(name)[source]: The DB exists already if there’s a key in the default DB with the name of the database

delete(lkey, val=None, db=None)[source]

dropdb(name)[source]: Deletes an entire database (i.e. a table), losing all data.

async fini()[source]

Shut down the object and notify any onfini() coroutines.

Returns:: Remaining ref count

firstkey(db=None)[source]: Return the first key or None from the given db.

forcecommit()[source]: Note

This method may raise a MapFullError

get(lkey, db=None)[source]

async getHotCount(name)[source]

async getMultiQueue(name, nexsroot=None)[source]

getNameAbrv(name)[source]

getSeqn(name)[source]

async classmethod getSlabStats()[source]

classmethod getSlabsInDir(dirn)[source]: Returns all open slabs under a directory

has(lkey, db=None)[source]

hasdup(lkey, lval, db=None)[source]

async classmethod initSyncLoop(inst)[source]

initdb(name, dupsort=False, integerkey=False, dupfixed=False)[source]

last(db=None)[source]: Return the last key/value pair from the given db.

lastkey(db=None)[source]: Return the last key or None from the given db.

pop(lkey, db=None)[source]

prefexists(byts, db=None)[source]: Returns True if a prefix exists in the db.

put(lkey, lval, dupdata=False, overwrite=True, append=False, db=None)[source]

putmulti(kvpairs, dupdata=False, append=False, db=None)[source]

Returns:: Tuple of number of items consumed, number of items added

rangeexists(lmin, lmax=None, db=None)[source]: Returns True if at least one key exists in the range.

replace(lkey, lval, db=None)[source]: Like put, but returns the previous value if existed

scanByDups(lkey, db=None)[source]

scanByDupsBack(lkey, db=None)[source]

scanByFull(db=None)[source]

scanByFullBack(db=None)[source]

scanByPref(byts, startkey=None, startvalu=None, db=None)[source]

Parameters:

byts (bytes) – prefix to match on
startkey (Optional[bytes]) – if present, will start scanning at key=byts+startkey
startvalu (Optional[bytes]) – if present, will start scanning at (key+startkey, startvalu)

Notes

startvalu only makes sense if byts+startkey matches an entire key. startvalu is only value for dupsort=True dbs

scanByPrefBack(byts, db=None)[source]

scanByRange(lmin, lmax=None, db=None)[source]

scanByRangeBack(lmax, lmin=None, db=None)[source]

scanKeys(db=None)[source]

scanKeysByPref(byts, db=None)[source]

stat(db=None)[source]

statinfo()[source]

async sync()[source]

async classmethod syncLoopOnce()[source]

async classmethod syncLoopTask()[source]

syncevnt = None

synctask = None

async trash()[source]: Deletes underlying storage

class synapse.lib.lmdbslab.SlabAbrv(slab, name)[source]

Bases: object

A utility for translating arbitrary bytes into fixed with id bytes

abrvToByts(abrv)[source]

abrvToName(byts)[source]

bytsToAbrv(byts)[source]

keys()[source]

nameToAbrv(name)[source]

names()[source]

setBytsToAbrv(byts)[source]

class synapse.lib.lmdbslab.SlabDict(slab, db=None, pref=b'')[source]

Bases: object

A dictionary-like object which stores its props in a slab via a prefix.

It is assumed that only one SlabDict with a given prefix exists at any given time, but it is up to the caller to cache them.

get(name, defval=None)[source]

Get a name from the SlabDict.

Parameters:

name (str) – The key name.
defval (obj) – The default value to return.

Returns:

The return value, or None.

Return type:

(obj)

inc(name, valu=1)[source]

items()[source]

Return a tuple of (prop, valu) tuples from the SlabDict.

Returns:: Tuple of (name, valu) tuples.
Return type:: (((str, object), …))

keys()[source]

pop(name, defval=None)[source]

Pop a name from the SlabDict.

Parameters:

name (str) – The name to remove.
defval (obj) – The default value to return if the name is not present.

Returns:

The object stored in the SlabDict, or defval if the object was not present.

Return type:

object

set(name, valu)[source]

Set a name in the SlabDict.

Parameters:

name (str) – The key name.
valu (obj) – A msgpack compatible value.

Returns:

None

synapse.lib.modelrev module

class synapse.lib.modelrev.ModelRev(core)[source]

Bases: object

async revCoreLayers()[source]

async revModel20210126(layers)[source]

async revModel20210312(layers)[source]

async revModel20210528(layers)[source]

async revModel20210801(layers)[source]

async revModel20211112(layers)[source]

async revModel20220307(layers)[source]

async revModel20220315(layers)[source]

async revModel20220509(layers)[source]

async revModel20220706(layers)[source]

async revModel20220803(layers)[source]

async revModel20220901(layers)[source]

async revModel20221025(layers)[source]

async revModel20221123(layers)[source]

async revModel20221212(layers)[source]

async revModel20221220(layers)[source]

async revModel20230209(layers)[source]

async revModel_0_2_18(layers)[source]

async revModel_0_2_19(layers)[source]

async revModel_0_2_20(layers)[source]

async revModel_0_2_21(layers)[source]

async revModel_0_2_22(layers)[source]

async revModel_0_2_23(layers)[source]

async revModel_0_2_24(layers)[source]

async runStorm(text, opts=None)[source]

Run storm code in a schedcoro and log the output messages.

Parameters:

text (str) – Storm query to execute.
opts – Storm opts.

Returns:

None

synapse.lib.module module

class synapse.lib.module.CoreModule(core, conf=None)[source]

Bases: object

confdefs = ()

getConfPath()[source]

Get the path to the module specific config file (conf.yaml).

Notes

This creates the parent directory for the conf.yaml file if it does not exist. This API exists to allow a implementor to get the conf path during initCoreModule and drop a example config if needed. One use case of that is for missing configuration values, an example config can be written to the file and a exception raised.

Returns:: Path to where the conf file is located at.
Return type:: str

getModDir()[source]

Get the path to the module specific directory.

Notes

This creates the directory if it did not previously exist.

Returns:: The filepath to the module specific directory.
Return type:: str

getModName()[source]

Return the lowercased name of this module.

Notes

This pulls the mod_name attribute on the class. This allows an implementer to set a arbitrary name for the module. If this attribute is not set, it defaults to self.__class__.__name__.lower() and sets mod_name to that value.

Returns:: The module name.
Return type:: (str)

getModPath(*paths)[source]

Construct a path relative to this module’s working directory.

Parameters:: *paths – A list of path strings

Notes

This creates the module specific directory if it does not exist.

Returns:: The full path (or None if no cortex dir is configured).
Return type:: (str)

getModelDefs()[source]

getStormCmds()[source]

Module implementers may override this to provide a list of Storm commands which will be loaded into the Cortex.

Returns:: A list of Storm Command classes (not instances).
Return type:: list

async initCoreModule()[source]

Module implementers may override this method to initialize the module after the Cortex has completed and is accessible to perform storage operations.

Notes

This is the preferred function to override for implementing custom code that needs to be executed during Cortex startup.

Any exception raised within this method will remove the module from the list of currently loaded modules.

This is called for modules after getModelDefs() and getStormCmds() has been called, in order to allow for model loading and storm command loading prior to code execution offered by initCoreModule.

A failure during initCoreModule will not unload data model or storm commands registered by the module.

Returns:: None

mod_name = None

async preCoreModule()[source]

Module implementers may override this method to execute code immediately after a module has been loaded.

Notes

The initCoreModule function is preferred for overriding instead of preCoreModule().

No Cortex layer/storage operations will function in preCoreModule.

Any exception raised within this method will halt additional loading of the module.

Returns:: None

synapse.lib.modules module

Module which implements the synapse module API/convention.

synapse.lib.msgpack module

class synapse.lib.msgpack.Unpk[source]

Bases: object

An extension of the msgpack streaming Unpacker which reports sizes.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

feed(byts)[source]

Feed bytes to the unpacker and return completed objects.

Parameters:: byts (bytes) – Bytes to unpack.

Notes

It is intended that this function is called multiple times with bytes from some sort of a stream, as it will unpack and return objects as they are available.

Returns:: List of tuples containing the item size and the unpacked item.
Return type:: list

synapse.lib.msgpack.deepcopy(item, use_list=False)[source]: Copy a msgpack serializable by packing then unpacking it. For complex primitives, this runs in about 1/3 the time of copy.deepcopy()

synapse.lib.msgpack.dumpfile(item, path)[source]

Dump an object to a file by path.

Parameters:

item (object) – The object to serialize.
path (str) – The file path to save.

Returns:

None

synapse.lib.msgpack.en(item)[source]

Use msgpack to serialize a compatible python object.

Parameters:: item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns:: The serialized bytes in msgpack format.
Return type:: bytes

synapse.lib.msgpack.getvars(varz)[source]

synapse.lib.msgpack.isok(item)[source]: Returns True if the item can be msgpacked (by testing packing).

synapse.lib.msgpack.iterfd(fd)[source]

Generator which unpacks a file object of msgpacked content.

Parameters:: fd – File object to consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields:: Objects from a msgpack stream.

synapse.lib.msgpack.iterfile(path, since=-1)[source]

Generator which yields msgpack objects from a file path.

Parameters:: path – File path to open and consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields:: Objects from a msgpack stream.

synapse.lib.msgpack.loadfile(path)[source]

Load and upack the msgpack bytes from a file by path.

Parameters:: path (str) – The file path to a message pack file.
Raises:: msgpack.exceptions.ExtraData – If the file contains multiple objects.
Returns:: The decoded python object.
Return type:: (obj)

synapse.lib.msgpack.un(byts, use_list=False)[source]

Use msgpack to de-serialize a python object.

Parameters:: byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns:: The de-serialized object
Return type:: obj

synapse.lib.multislabseqn module

class synapse.lib.multislabseqn.MultiSlabSeqn[source]

Bases: Base

An append-optimized sequence of byte blobs stored across multiple slabs for fast rotating/culling

async add(item: Any, indx=None) → int[source]: Add a single item to the sequence.

async cull(offs: int) → bool[source]: Remove entries up to (and including) the given offset.

async get(offs: int) → Any[source]: Retrieve a single row by offset

getOffsetEvent(offs: int) → Event[source]: Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

async gets(offs, wait=True) → AsyncIterator[Tuple[int, Any]][source]: Just like iter, but optionally waits for new entries once the end is reached.

index() → int[source]: Return the current index to be used

async iter(offs: int) → AsyncIterator[Tuple[int, Any]][source]

Iterate over items in a sequence from a given offset.

Parameters:: offs (int) – The offset to begin iterating from.
Yields:: (indx, valu) – The index and valu of the item.

async last() → Tuple[int, Any] | None[source]

async rotate() → int[source]

Rotate the Nexus log at the current index.

Note

After this executes the tailseqn will be empty. Waiting for this indx to be written will indicate when it is possible to cull 1 minus the return value such that the rotated seqn is deleted.

Returns:: The starting index of the new seqn
Return type:: int

setIndex(indx: int) → None[source]

static slabFilename(dirn: str, indx: int)[source]

async waitForOffset(offs: int, timeout=None) → bool[source]

Returns:: true if the event got set, False if timed out

synapse.lib.nexus module

class synapse.lib.nexus.ChangeDist[source]

Bases: Base

A utility class to distribute new change entries to mirrors/followers

update() → bool[source]

class synapse.lib.nexus.NexsRoot[source]

Bases: Base

async addWriteHold(reason)[source]

async cull(offs)[source]

async delWriteHold(reason)[source]

async eat(nexsiden, event, args, kwargs, meta)[source]: Actually mutate for the given nexsiden instance.

async enNexsLog()[source]

getChangeDist(offs: int) → AsyncIterator[ChangeDist][source]

async index()[source]

async isNexsReady()[source]

async issue(nexsiden, event, args, kwargs, meta=None)[source]: If I’m not a follower, mutate, otherwise, ask the leader to make the change and wait for the follower loop to hand me the result through a future.

async iter(offs: int, tellready=False) → AsyncIterator[Any][source]: Returns an iterator of change entries in the log

async promote()[source]

async recover() → None[source]

Replays the last entry in the nexus log in case we crashed between writing the log and applying it.

Notes

This must be called at cell startup after subsystems are initialized but before any write transactions might happen.

The log can only have recorded 1 entry ahead of what is applied. All log actions are idempotent, so replaying the last action that (might have) already happened is harmless.

reqNotReadOnly()[source]

async rotate()[source]

async runMirrorLoop(proxy)[source]

async setNexsReady(status)[source]

async setindex(indx)[source]

async startup()[source]

async waitOffs(offs, timeout=None)[source]

class synapse.lib.nexus.Pusher[source]

Bases: Base

A mixin-class to manage distributing changes where one might plug in mirroring or consensus protocols

classmethod onPush(event: str, passitem=False) → Callable[source]

Decorator that registers a method to be a handler for a named event

Parameters:

event – string that distinguishes one handler from another. Must be unique per Pusher subclass
passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”

classmethod onPushAuto(event: str, passitem=False) → Callable[source]

Decorator that does the same as onPush, except automatically creates the top half method

Parameters:

event – string that distinguishes one handler from another. Must be unique per Pusher subclass
passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”

async saveToNexs(name, *args, **kwargs)[source]

setNexsRoot(nexsroot)[source]

class synapse.lib.nexus.RegMethType(name: str, bases: List[type], attrs: Dict[str, Any])[source]

Bases: type

Metaclass that collects all methods in class with _regme prop into a class member called _regclstupls

synapse.lib.node module

class synapse.lib.node.Node(snap, sode, bylayer=None)[source]

Bases: object

A Cortex hypergraph node.

NOTE: This object is for local Cortex use during a single Xact.

async addEdge(verb, n2iden)[source]

async addTag(tag, valu=(None, None))[source]

Add a tag to a node.

Parameters:

tag (str) – The tag to add to the node.
valu – The optional tag value. If specified, this must be a value that norms as a valid time interval as an ival.

Returns:

This returns None.

Return type:

None

async delEdge(verb, n2iden)[source]

async delTag(tag, init=False)[source]: Delete a tag from the node.

async delTagProp(tag, name)[source]

async delete(force=False)[source]

Delete a node from the cortex.

The following tear-down operations occur in order:

validate that you have permissions to delete the node

validate that you have permissions to delete all tags

validate that there are no remaining references to the node.

delete all the tags (bottom up)

fire onDelTag() handlers

delete tag properties from storage

delete all secondary properties

fire onDelProp handler

delete secondary property from storage

delete the primary property

fire onDel handlers for the node

delete primary property from storage

async filter(runt, text, opts=None, path=None)[source]

get(name)[source]

Return a secondary property value from the Node.

Parameters:: name (str) – The name of a secondary property.
Returns:: The secondary property value or None.
Return type:: (obj)

getByLayer()[source]: Return a dictionary that translates the node’s bylayer dict to a primitive.

async getData(name, defv=None)[source]

async getEmbeds(embeds)[source]: Return a dictionary of property embeddings.

getNodeRefs()[source]: Return a list of (prop, (form, valu)) refs out for the node.

async getStorNodes()[source]: Return a list of the raw storage nodes for each layer.

getTag(name, defval=None)[source]

getTagProp(tag, prop, defval=None)[source]: Return the value (or defval) of the given tag property.

getTagProps(tag)[source]

getTags(leaf=False)[source]

has(name)[source]

async hasData(name)[source]

hasTag(name)[source]

hasTagProp(tag, prop)[source]: Check if a #foo.bar:baz tag property exists on the node.

iden()[source]

async iterData()[source]

async iterDataKeys()[source]

async iterEdgeVerbs(n2buid)[source]

async iterEdgesN1(verb=None)[source]

async iterEdgesN2(verb=None)[source]

pack(dorepr=False)[source]

Return the serializable/packed version of the node.

Parameters:: dorepr (bool) – Include repr information for human readable versions of properties.
Returns:: An (ndef, info) node tuple.
Return type:: (tuple)

async pop(name, init=False)[source]: Remove a property from a node and return the value

async popData(name)[source]

repr(name=None, defv=None)[source]

reprs()[source]: Return a dictionary of repr values for props whose repr is different than the system mode value.

async set(name, valu, init=False)[source]

Set a property on the node.

Parameters:

name (str) – The name of the property.
valu (obj) – The value of the property.
init (bool) – Set to True to disable read-only enforcement

Returns:

True if the property was changed.

Return type:

(bool)

async setData(name, valu)[source]

async setTagProp(tag, name, valu)[source]: Set the value of the given tag property.

async storm(runt, text, opts=None, path=None)[source]

Parameters:: path (Path) – If set, then vars from path are copied into the new runtime, and vars are copied back out into path at the end

Note

If opts is not None and opts[‘vars’] is set and path is not None, then values of path vars take precedent

tagpropreprs()[source]: Return a dictionary of repr values for tagprops whose repr is different than the system mode value.

class synapse.lib.node.Path(vars, nodes)[source]

Bases: object

A path context tracked through the storm runtime.

clone()[source]

finiframe()[source]: Pop a scope frame from the path, restoring runt if at the top :param runt: A storm runtime to restore if we’re at the top :type runt: Runtime :param merge: Set to true to merge vars back up into the next frame :type merge: bool

fork(node)[source]

getVar(name, defv=<synapse.common.NoValu object>)[source]

initframe(initvars=None)[source]

meta(name, valu)[source]: Add node specific metadata to be returned with the node.

async pack(path=False)[source]

async popVar(name)[source]

async setVar(name, valu)[source]

synapse.lib.node.iden(pode)[source]

Return the iden (buid) of the packed node.

Parameters:: pode (tuple) – A packed node.
Returns:: The node iden.
Return type:: str

synapse.lib.node.ndef(pode)[source]

Return a node definition (<form>,<valu>) tuple from the node.

Parameters:: pode (tuple) – A packed node.
Returns:: The (<form>,<valu>) tuple for the node
Return type:: ((str,obj))

synapse.lib.node.prop(pode, prop)[source]

Return the valu of a given property on the node.

Parameters:

pode (tuple) – A packed node.
prop (str) – Property to retrieve.

Notes

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns:

synapse.lib.node.props(pode)[source]

Get the props from the node.

Parameters:: pode (tuple) – A packed node.

Notes

This will include any universal props present on the node.

Returns:: A dictionary of properties.
Return type:: dict

synapse.lib.node.reprNdef(pode)[source]

Get the ndef of the pode with a human readable value.

Parameters:: pode (tuple) – A packed node.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

Returns:: A tuple of form and the human readable value.
Return type:: (str, str)

synapse.lib.node.reprProp(pode, prop)[source]

Get the human readable value for a secondary property from the pode.

Parameters:

pode (tuple) – A packed node.
prop –

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns:: The human readable property value. If the property is not present, returns None.
Return type:: str

synapse.lib.node.reprTag(pode, tag)[source]

Get the human readable value for the tag timestamp from the pode.

Parameters:

pode (tuple) – A packed node.
tag (str) – The tag to get the value for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have a timestamp, this returns a empty string.

Returns:: The human readable value for the tag. If the tag is not present, returns None.
Return type:: str

synapse.lib.node.reprTagProps(pode, tag)[source]

Get the human readable values for any tagprops on a tag for a given node.

Parameters:

pode (tuple) – A packed node.
tag (str) – The tag to get the tagprops reprs for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have any tagprops associated with it, this returns an empty list.

Returns:: A list of tuples, containing the name of the tagprop and the repr value.
Return type:: list

synapse.lib.node.tagged(pode, tag)[source]

Check if a packed node has a given tag.

Parameters:

pode (tuple) – A packed node.
tag (str) – The tag to check.

Examples

Check if a node is tagged with “woot” and dostuff if it is.

if s_node.tagged(node,’woot’):
dostuff()

Notes

If the tag starts with #, this is removed prior to checking.

Returns:: True if the tag is present. False otherwise.
Return type:: bool

synapse.lib.node.tags(pode, leaf=False)[source]

Get all the tags for a given node.

Parameters:

pode (tuple) – A packed node.
leaf (bool) – If True, only return leaf tags

Returns:

A list of tag strings.

Return type:

list

synapse.lib.node.tagsnice(pode)[source]

Get all the leaf tags and the tags that have values or tagprops.

Parameters:: pode (tuple) – A packed node.
Returns:: A list of tag strings.
Return type:: list

synapse.lib.oauth module

class synapse.lib.oauth.OAuthMixin[source]

Bases: Pusher

Mixin for Cells to organize and execute OAuth token refreshes.

async addOAuthProvider(conf)[source]

async clearOAuthAccessToken(provideriden, useriden)[source]: Remove a client access token by clearing the configuration. This will prevent further refreshes (if scheduled), and a new auth code will be required the next time an access token is requested.

async delOAuthProvider(iden)[source]

async getOAuthAccessToken(provideriden, useriden)[source]

async getOAuthClient(provideriden, useriden)[source]

async getOAuthProvider(iden)[source]

listOAuthClients()[source]

Returns:: List of (provideriden, useriden, conf) for each client.
Return type:: list

async listOAuthProviders()[source]

async setOAuthAuthCode(provideriden, useriden, authcode, code_verifier=None)[source]: Typically set as the end result of a successful OAuth flow. An initial access token and refresh token will be immediately requested, and the client will be loaded into the schedule to be background refreshed.

synapse.lib.oauth.normOAuthTokenData(issued_at, data)[source]: Normalize timestamps to be in epoch millis and set expires_at/refresh_at.

synapse.lib.output module

Tools for easily hookable output from cli-like tools.

class synapse.lib.output.OutPut[source]

Bases: object

printf(mesg, addnl=True)[source]

class synapse.lib.output.OutPutBytes[source]: Bases: OutPutFd

class synapse.lib.output.OutPutFd(fd, enc='utf8')[source]: Bases: OutPut

class synapse.lib.output.OutPutStr[source]: Bases: OutPut

synapse.lib.parser module

class synapse.lib.parser.AstConverter(text)[source]

Bases: Transformer

Convert AST from parser into synapse AST, depth first.

If a method with a name that matches the current rule exists, that will be called, otherwise __default__ will be used

cmdrargs(meta, kids)[source]

embedquery(meta, kids)[source]

evalvalu(meta, kids)[source]

exprdict(meta, kids)[source]

exprlist(meta, kids)[source]

funcargs(meta, kids)[source]: A list of function parameters (as part of a function definition)

funccall(meta, kids)[source]

liftreverse(meta, kids)[source]

metaToAstInfo(meta, isterm=False)[source]

operrelprop_join(meta, kids)[source]

operrelprop_pivot(meta, kids, isjoin=False)[source]

raiseBadSyntax(mesg, astinfo)[source]

stormcmdargs(meta, kids)[source]

subquery(meta, kids)[source]

switchcase(meta, kids)[source]

varderef(meta, kids)[source]

varlist(meta, kids)[source]

yieldvalu(meta, kids)[source]

class synapse.lib.parser.AstInfo(text, soff, eoff, sline, eline, scol, ecol, isterm)

Bases: tuple

ecol: Alias for field number 6

eline: Alias for field number 4

eoff: Alias for field number 2

isterm: Alias for field number 7

scol: Alias for field number 5

sline: Alias for field number 3

soff: Alias for field number 1

text: Alias for field number 0

class synapse.lib.parser.CmdStringer(visit_tokens: bool = True)[source]

Bases: Transformer

alist(meta, kids)[source]

cmdstring(meta, kids)[source]

valu(meta, kids)[source]

class synapse.lib.parser.Parser(text, offs=0)[source]

Bases: object

Storm query parser

cmdrargs()[source]: Parse command args that might have storm queries as arguments

eval()[source]

lookup()[source]

query()[source]

Parse the storm query

Returns (s_ast.Query): instance of parsed query

search()[source]

synapse.lib.parser.format_unescape(valu)[source]

synapse.lib.parser.massage_vartokn(astinfo, x)[source]

synapse.lib.parser.parseEval(text)[source]

synapse.lib.parser.parseQuery(text, mode='storm')[source]: Parse a storm query and return the Lark AST. Cached here to speed up unit tests

synapse.lib.parser.parse_cmd_string(text, off)[source]: Parse a command line string which may be quoted.

synapse.lib.parser.unescape(valu)[source]: Parse a string for backslash-escaped characters and omit them. The full list of escaped characters can be found at https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

synapse.lib.queue module

class synapse.lib.queue.AQueue[source]

Bases: Base

An async queue with chunk optimized sync compatible consumer.

put(item)[source]: Add an item to the queue.

async slice()[source]

class synapse.lib.queue.Queue(maxsize=None)[source]

Bases: object

An asyncio Queue with batch methods and graceful close.

async close()[source]

async put(item)[source]

async puts(items)[source]

async size()[source]

async slice(size=1000)[source]

async slices(size=1000)[source]

class synapse.lib.queue.Window[source]

Bases: Base

A Queue like object which yields added items. If the queue ever reaches its maxsize, it will be fini()d. On fini(), the Window will continue to yield results until empty and then return.

async put(item)[source]: Add a single item to the Window.

async puts(items)[source]: Add multiple items to the window.

synapse.lib.ratelimit module

class synapse.lib.ratelimit.RateLimit(rate, per)[source]

Bases: object

A RateLimit class may be used to detect/enforce rate limits.

Example

# allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10)

Notes

It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”.

allows()[source]

Returns True if the rate limit has not been reached.

Example

if not rlimit.allows():: rasie RateExceeded()

# ok to go…

synapse.lib.reflect module

synapse.lib.reflect.getClsNames(item)[source]

Return a list of “fully qualified” class names for an instance.

Example

for name in getClsNames(foo):: print(name)

synapse.lib.reflect.getItemLocals(item)[source]

Iterate the locals of an item and yield (name,valu) pairs.

Example

for name,valu in getItemLocals(item):: dostuff()

synapse.lib.reflect.getMethName(meth)[source]: Return a fully qualified string for the <mod>.<class>.<func> name of a given method.

synapse.lib.reflect.getShareInfo(item)[source]

Get a dictionary of special annotations for a Telepath Proxy.

Parameters:: item – Item to inspect.

Notes

This will set the _syn_telemeth attribute on the item and the items class, so this data is only computed once.

Returns:: A dictionary of methods requiring special handling by the proxy.
Return type:: dict

synapse.lib.rstorm module

class synapse.lib.rstorm.OutPutRst[source]

Bases: OutPutStr

Rst specific helper for output intended to be indented in RST text as a literal block.

prefix = ' '

printf(mesg, addnl=True)[source]

class synapse.lib.rstorm.StormCliOutput[source]

Bases: StormCli

async handleErr(mesg)[source]

printf(mesg, addnl=True, color=None)[source]

async runRstCmdLine(text, ctx, stormopts=None)[source]

class synapse.lib.rstorm.StormOutput(core, ctx, stormopts=None, opts=None)[source]

Bases: StormCmd

Produce standard output from a stream of storm runtime messages. Must be instantiated for a single query with a rstorm context.

printf(mesg, addnl=True, color=None)[source]

async runCmdLine(line)[source]

Run a line of command input for this command.

Parameters:: line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:: opts (dict) – Options dictionary.

class synapse.lib.rstorm.StormRst[source]

Bases: Base

async run()[source]

Parses the specified RST file with Storm directive handling.

Returns:: List of line strings for the RST output
Return type:: list

synapse.lib.rstorm.getCell(ctor, conf)[source]

synapse.lib.schemas module

synapse.lib.scope module

class synapse.lib.scope.Scope(*frames, **vals)[source]

Bases: object

The Scope object assists in creating nested varible scopes.

Example

with Scope() as scope:

scope.set(‘foo’,10)

with scope:
scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20…

dostuff(scope) # ‘foo’ is 10 again…

add(name, *vals)[source]: Add values as iter() compatible items in the current scope frame.

copy()[source]

Create a shallow copy of the current Scope.

Returns:: A new scope which is a copy of the current scope.
Return type:: Scope

enter(vals=None)[source]: Add an additional scope frame.

get(name, defval=None)[source]: Retrieve a value from the closest scope frame.

iter(name)[source]: Iterate through values added with add() from each scope frame.

leave()[source]: Pop the current scope frame.

pop(name, defval=None)[source]

Pop and return a value (from the last frame) of the scope.

Parameters:: name (str) – The name of the scope variable.
Returns:: The scope variable value or None
Return type:: obj

set(name, valu)[source]: Set a value in the current scope frame.

update(vals)[source]: Set multiple values in the current scope frame.

synapse.lib.scope.clone(task: Task) → None[source]

Clone the current task Scope onto the provided task.

Parameters:: task (asyncio.Task) – The task object to attach the scope too.

Notes

This must be run from an asyncio IO loop.

If the current task does not have a scope, we clone the default global Scope.

This will enter() the scope, and add a task callback to leave() the scope.

Returns:: None

synapse.lib.scope.ctor(name, func, *args, **kwargs)[source]: Add a ctor callback to the global scope.

synapse.lib.scope.enter(vals=None)[source]: Return the task’s local scope for use in a with block

synapse.lib.scope.get(name, defval=None)[source]: Access this task’s scope with default values from glob.

synapse.lib.scope.pop(name)[source]

Pop and return a task scope variable. :param name: The task scope variable name. :type name: str

Returns:: The scope value or None
Return type:: obj

synapse.lib.scope.set(name, valu)[source]: Set a value in the current frame of the local task scope.

synapse.lib.scope.update(vals)[source]

synapse.lib.scrape module

synapse.lib.scrape.contextScrape(text, form=None, refang=True, first=False)[source]

Scrape types from a blob of text and yield info dictionaries.

Parameters:

text (str) – Text to scrape.
form (str) – Optional form to scrape. If present, only scrape items which match the provided form.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.

Notes

The dictionaries yielded by this function contains the following keys:

match
The raw matching text found in the input text.

offset
The offset into the text where the match was found.

valu
The resulting value.

form
The corresponding form for the valu.

Returns:: Yield info dicts of results.
Return type:: (dict)

async synapse.lib.scrape.contextScrapeAsync(text, form=None, refang=True, first=False)[source]

Scrape types from a blob of text and yield info dictionaries, potentially in a spawned process.

Parameters:

text (str) – Text to scrape.
form (str) – Optional form to scrape. If present, only scrape items which match the provided form.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.

Notes

The dictionaries yielded by this function contains the following keys:

match
The raw matching text found in the input text.

offset
The offset into the text where the match was found.

valu
The resulting value.

form
The corresponding form for the valu.

Returns:: Yield info dicts of results.
Return type:: (dict)

synapse.lib.scrape.cve_check(match: Match)[source]

synapse.lib.scrape.fqdn_check(match: Match)[source]

synapse.lib.scrape.fqdn_prefix_check(match: Match)[source]

synapse.lib.scrape.genFangRegex(fangs, flags=RegexFlag.I)[source]

synapse.lib.scrape.genMatches(text: str, regx: compile, opts: dict)[source]

Generate regular expression matches for a blob of text.

Parameters:

text (str) – The text to generate matches for.
regx (regex.Regex) – A compiled regex object. The regex must contained a named match group for valu.
opts (dict) – An options dictionary.

Notes

The dictionaries yielded by this function contains the following keys:

raw_valu
The raw matching text found in the input text.

offset
The offset into the text where the match was found.

valu
The resulting value - this may be altered by callbacks.

The options dictionary can contain a callback key. This function is expected to take a single argument, a regex.Match object, and return a tuple of the new valu and info dictionary. The new valu is used as the valu key in the returned dictionary, and any other information in the info dictionary is pushed into the return dictionary as well.

Yields:: dict – A dictionary of match results.

async synapse.lib.scrape.genMatchesAsync(text: str, regx: compile, opts: dict)[source]

Generate regular expression matches for a blob of text, potentially in a spawned process.

Parameters:

text (str) – The text to generate matches for.
regx (regex.Regex) – A compiled regex object. The regex must contained a named match group for valu.
opts (dict) – An options dictionary.

Notes

The dictionaries yielded by this function contains the following keys:

raw_valu
The raw matching text found in the input text.

offset
The offset into the text where the match was found.

valu
The resulting value - this may be altered by callbacks.

The options dictionary can contain a callback key. This function is expected to take a single argument, a regex.Match object, and return a tuple of the new valu and info dictionary. The new valu is used as the valu key in the returned dictionary, and any other information in the info dictionary is pushed into the return dictionary as well.

Yields:: dict – A dictionary of match results.

synapse.lib.scrape.getForms()[source]

Get a list of forms recognized by the scrape APIs.

Returns:: A list of form values.
Return type:: list

synapse.lib.scrape.inet_server_check(match: Match)[source]

synapse.lib.scrape.ipv6_check(match: Match)[source]

synapse.lib.scrape.linux_path_check(match: Match)[source]

synapse.lib.scrape.refang_text(txt)[source]

Remove address de-fanging in text blobs, .e.g. example[.]com to example.com

Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM

Returns:: Re-fanged text blob
Return type:: (str)

synapse.lib.scrape.refang_text2(txt: str, re: compile = regex.Regex('fxp:|fxps:|hxxp:|hxxps:|fxp\\[s\\]:|hxxp\\[s\\]:|ftp\\[:\\]|fxp\\[:\\]|ftps\\[:\\]|fxps\\[:\\]|http\\[:\\]|hxxp\\[:\\]|https\\[:\\]|hxxps\\[:\\]|ftp\\[://\\]|fxp\\[://\\]|ftps\\[://\\]|fxps\\[://\\]|http\\[://\\]|hxxp\\[://\\]|https\\[://\\]|hxxps\\[://\\]|ftp\\[:|fxp\\[:|ftps\\[:|fxps\\[:|http\\[:|hxxp\\[:|https\\[:|hxxps\\[:|ftp\$:\$|fxp\$:\$|ftps\$:\$|fxps\$:\$|http\$:\$|hxxp\$:\$|https\$:\$|hxxps\$:\$|\\[\\.\\]|\\.\\]|\\[\\.|\\[．\\]|\\[。\\]|\\[｡\\]|\$\\.\$|\$．\$|\$。\$|\$｡\$|\\[dot\\]|\\[:\\]|\\[at\\]|\\[@\\]|\\\\\\.', flags=regex.I | regex.V0), fangs: dict = {'(.)': '.', '(。)': '。', '(．)': '．', '(｡)': '｡', '.]': '.', '[.': '.', '[.]': '.', '[:]': ':', '[@]': '@', '[at]': '@', '[dot]': '.', '[。]': '。', '[．]': '．', '[｡]': '｡', '\\.': '.', 'ftp(:)': 'ftp:', 'ftp[:': 'ftp:', 'ftp[://]': 'ftp://', 'ftp[:]': 'ftp:', 'ftps(:)': 'ftps:', 'ftps[:': 'ftps:', 'ftps[://]': 'ftps://', 'ftps[:]': 'ftps:', 'fxp(:)': 'ftp:', 'fxp:': 'ftp:', 'fxp[:': 'ftp:', 'fxp[://]': 'ftp://', 'fxp[:]': 'ftp:', 'fxp[s]:': 'ftps:', 'fxps(:)': 'ftps:', 'fxps:': 'ftps:', 'fxps[:': 'ftps:', 'fxps[://]': 'ftps://', 'fxps[:]': 'ftps:', 'http(:)': 'http:', 'http[:': 'http:', 'http[://]': 'http://', 'http[:]': 'http:', 'https(:)': 'https:', 'https[:': 'https:', 'https[://]': 'https://', 'https[:]': 'https:', 'hxxp(:)': 'http:', 'hxxp:': 'http:', 'hxxp[:': 'http:', 'hxxp[://]': 'http://', 'hxxp[:]': 'http:', 'hxxp[s]:': 'https:', 'hxxps(:)': 'https:', 'hxxps:': 'https:', 'hxxps[:': 'https:', 'hxxps[://]': 'https://', 'hxxps[:]': 'https:'})[source]

Remove address de-fanging in text blobs, .e.g. example[.]com to example.com

Notes

Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM

Parameters:: txt (str) – The text to re-fang.
Returns:: A tuple containing the new text, and a dictionary containing offset information where the new text was altered with respect to the original text.
Return type:: tuple(str, dict)

synapse.lib.scrape.scrape(text, ptype=None, refang=True, first=False)[source]

Scrape types from a blob of text and return node tuples.

Parameters:

text (str) – Text to scrape.
ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.

Returns:

Yield tuples of node ndef values.

Return type:

(str, object)

async synapse.lib.scrape.scrapeAsync(text, ptype=None, refang=True, first=False)[source]

Scrape types from a blob of text and return node tuples, potentially in a spawned process.

Parameters:

text (str) – Text to scrape.
ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.
refang (bool) – Whether to remove de-fanging schemes from text before scraping.
first (bool) – If true, only yield the first item scraped.

Returns:

Yield tuples of node ndef values.

Return type:

(str, object)

synapse.lib.scrape.unc_path_check(match: Match)[source]

synapse.lib.scrape.windows_path_check(match: Match)[source]

synapse.lib.slaboffs module

class synapse.lib.slaboffs.SlabOffs(slab: Slab, db: str)[source]

Bases: object

A helper for storing offset integers by iden.

As with all slab objects, this is meant for single-thread async loop use.

delete(iden)[source]

get(iden)[source]

set(iden, offs)[source]

synapse.lib.slabseqn module

class synapse.lib.slabseqn.SlabSeqn(slab, name: str)[source]

Bases: object

An append optimized sequence of byte blobs.

Parameters:

lenv (lmdb.Environment) – The LMDB Environment.
name (str) – The name of the sequence.

add(item, indx=None)[source]: Add a single item to the sequence.

async aiter(offs, wait=False, timeout=None)[source]

Iterate over items in a sequence from a given offset.

Parameters:

offs (int) – The offset to begin iterating from.
wait (boolean) – Once caught up, yield new results in realtime.
timeout (int) – Max time to wait for a new item.

Yields:

(indx, valu) – The index and valu of the item.

async cull(offs)[source]: Remove entries up to (and including) the given offset.

first()[source]

get(offs)[source]: Retrieve a single row by offset

getByIndxByts(indxbyts)[source]

getOffsetEvent(offs)[source]: Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

getraw(byts)[source]

async gets(offs, wait=True)[source]

Returns an async generator of indx/valu tuples, optionally waiting and continuing to yield them as new entries are added

Parameters:

offs (int) – The offset to begin iterating from.
wait (bool) – Whether to continue yielding tupls when it hits the end of the sequence.

Yields:

(indx, valu) – The index and valu of the item.

index()[source]: Return the current index to be used

iter(offs)[source]

Iterate over items in a sequence from a given offset.

Parameters:: offs (int) – The offset to begin iterating from.
Yields:: (indx, valu) – The index and valu of the item.

iterBack(offs)[source]

Iterate backwards over items in a sequence from a given offset.

Parameters:: offs (int) – The offset to begin iterating from.
Yields:: (indx, valu) – The index and valu of the item.

last()[source]

nextindx()[source]

Determine the next insert offset according to storage.

Returns:: The next insert offset.
Return type:: int

pop(offs)[source]: Pop a single entry at the given offset.

rows(offs)[source]: Iterate over raw indx, bytes tuples from a given offset.

save(items)[source]

Save a series of items to a sequence.

Parameters:: items (tuple) – The series of items to save into the sequence.
Returns:: The index of the first item

slice(offs, size)[source]

sliceBack(offs, size)[source]

stat()[source]

trim(offs)[source]: Delete entries starting at offset and moving forward.

async waitForOffset(offs, timeout=None)[source]

Returns:: true if the event got set, False if timed out

synapse.lib.snap module

class synapse.lib.snap.ProtoNode(ctx, buid, form, valu, node)[source]

Bases: object

A prototype node used for staging node adds using a SnapEditor.

TODO: This could eventually fully mirror the synapse.lib.node.Node API and be used: to slipstream into sections of the pipeline to facilitate a bulk edit / transaction

async addEdge(verb, n2iden)[source]

async addTag(tag, valu=(None, None), tagnode=None)[source]

async delEdge(verb, n2iden)[source]

get(name)[source]

async getData(name)[source]

getNodeEdit()[source]

async getSetOps(name, valu, norminfo=None)[source]

getTag(tag)[source]

getTagProp(tag, name)[source]

iden()[source]

async set(name, valu, norminfo=None)[source]

async setData(name, valu)[source]

async setTagProp(tag, name, valu)[source]

class synapse.lib.snap.Scrubber(rules)[source]

Bases: object

scrub(pode)[source]

class synapse.lib.snap.Snap[source]

Bases: Base

A “snapshot” is a transaction across multiple Cortex layers.

The Snap object contains the bulk of the Cortex API to facilitate performance through careful use of transaction boundaries.

Transactions produce the following EventBus events:

(‘print’, {}),

async addFeedData(name, items)[source]

async addFeedNodes(name, items)[source]

Call a feed function and return what it returns (typically yields Node()s).

Parameters:

name (str) – The name of the feed record type.
items (list) – A list of records of the given feed type.

Returns:

The return value from the feed function. Typically Node() generator.

Return type:

(object)

async addNode(name, valu, props=None, norminfo=None)[source]

Add a node by form name and value with optional props.

Parameters:

name (str) – The form of node to add.
valu (obj) – The value for the node.
props (dict) – Optional secondary properties for the node.

Notes

If a props dictionary is provided, it may be mutated during node construction.

Returns:: A Node object. It may return None if the snap is unable to add or lift the node.
Return type:: s_node.Node

async addNodes(nodedefs)[source]

Add/merge nodes in bulk.

The addNodes API is designed for bulk adds which will also set properties, add tags, add edges, and set nodedata to existing nodes. Nodes are specified as a list of the following tuples:

( (form, valu), {‘props’:{}, ‘tags’:{}})

Parameters:: nodedefs (list) – A list of nodedef tuples.
Returns:: A list of xact messages.
Return type:: (list)

async addStormRuntime(query, opts=None, user=None)[source]

async applyNodeEdit(edit, nodecache=None)[source]

async applyNodeEdits(edits, nodecache=None)[source]: Sends edits to the write layer and evaluates the consequences (triggers, node object updates)

buidcachesize = 100000

async clearCache()[source]

clearCachedNode(buid)[source]

async eval(text, opts=None, user=None)[source]: Run a storm query and yield Node() objects.

getEditor()[source]

async getNodeByBuid(buid)[source]

Retrieve a node tuple by binary id.

Parameters:: buid (bytes) – The binary ID for the node.
Returns:: The node object or None.
Return type:: Optional[s_node.Node]

async getNodeByNdef(ndef)[source]

Return a single Node by (form,valu) tuple.

Parameters:

ndef ((str,obj)) – A (form,valu) ndef tuple. valu must be
normalized. –

Returns:

The Node or None.

Return type:

(synapse.lib.node.Node)

async getNodeData(buid, name, defv=None)[source]: Get nodedata from closest to write layer, no merging involved

getNodeEditor(node)[source]

async getRuntNodes(full, valu=None, cmpr=None)[source]

async getSnapMeta()[source]: Retrieve snap metadata to store along side nodeEdits.

getStormRuntime(query, opts=None, user=None)[source]

async getTagNode(name)[source]: Retrieve a cached tag node. Requires name is normed. Does not add.

async getTagNorm(tagname)[source]

async hasNodeData(buid, name)[source]: Return True if the buid has nodedata set on it under the given name False otherwise

async hasNodeEdge(buid1, verb, buid2)[source]

async iterEdgeVerbs(n1buid, n2buid)[source]

async iterNodeData(buid)[source]: Returns: Iterable[Tuple[str, Any]]

async iterNodeDataKeys(buid)[source]: Yield each data key from the given node by buid.

async iterNodeEdgesN1(buid, verb=None)[source]

async iterNodeEdgesN2(buid, verb=None)[source]

async iterStormPodes(text, opts, user=None)[source]: Yield packed node tuples for the given storm query text.

async keepalive(period)[source]

async nodes(text, opts=None, user=None)[source]

async nodesByDataName(name)[source]

async nodesByProp(full, reverse=False)[source]

async nodesByPropArray(full, cmpr, valu, reverse=False)[source]

async nodesByPropTypeValu(name, valu, reverse=False)[source]

async nodesByPropValu(full, cmpr, valu, reverse=False)[source]

async nodesByTag(tag, form=None, reverse=False)[source]

async nodesByTagProp(form, tag, name, reverse=False)[source]

async nodesByTagPropValu(form, tag, name, cmpr, valu, reverse=False)[source]

async nodesByTagValu(tag, cmpr, valu, form=None, reverse=False)[source]

async printf(mesg)[source]

async saveNodeEdits(edits, meta)[source]

async storm(text, opts=None, user=None)[source]: Execute a storm query and yield (Node(), Path()) tuples.

tagcachesize = 1000

async warn(mesg, log=True, **info)[source]

async warnonce(mesg, log=True, **info)[source]

class synapse.lib.snap.SnapEditor(snap)[source]

Bases: object

A SnapEditor allows tracking node edits with subs/deps as a transaction.

async addNode(formname, valu, props=None, norminfo=None)[source]

async getAddNodeOps(formname, valu, props=None, norminfo=None)[source]

async getNodeByBuid(buid)[source]

getNodeEdits()[source]

loadNode(node)[source]

synapse.lib.spooled module

class synapse.lib.spooled.Dict[source]

Bases: Spooled

get(key, defv=None)[source]

has(key)[source]

items()[source]

keys()[source]

pop(key, defv=None)[source]

async set(key, val)[source]

class synapse.lib.spooled.Set[source]

Bases: Spooled

A minimal set-like implementation that will spool to a slab on large growth.

async add(valu)[source]

discard(valu)[source]

has(key)[source]

class synapse.lib.spooled.Spooled[source]

Bases: Base

A Base class that can be used to implement objects which fallback to lmdb.

These objects are intended to fallback from Python to lmbd slabs, which aligns them together. Under memory pressure, these objects have a better shot of getting paged out.

synapse.lib.storm module

class synapse.lib.storm.BackgroundCmd(runt, runtsafe)[source]

Bases: Cmd

Execute a query pipeline as a background task. NOTE: Variables are passed through but nodes are not

async execStormCmd(runt, genr)[source]: Abstract base method

async execStormTask(query, opts)[source]

getArgParser()[source]

name = 'background'

class synapse.lib.storm.BatchCmd(runt, runtsafe)[source]

Bases: Cmd

Run a query with batched sets of nodes.

The batched query will have the set of inbound nodes available in the variable $nodes.

This command also takes a conditional as an argument. If the conditional evaluates to true, the nodes returned by the batched query will be yielded, if it evaluates to false, the inbound nodes will be yielded after executing the batched query.

NOTE: This command is intended to facilitate use cases such as queries to external: APIs with aggregate node values to reduce quota consumption. As this command interrupts the node stream, it should be used carefully to avoid unintended slowdowns in the pipeline.

Example

// Execute a query with batches of 5 nodes, then yield the inbound nodes batch $lib.false –size 5 { $lib.print($nodes) }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'batch'

class synapse.lib.storm.Cmd(runt, runtsafe)[source]

Bases: object

A one line description of the command.

Command usage details and long form description.

Example

cmd –help

Notes

Python Cmd implementers may override the forms attribute with a dictionary to provide information about Synapse forms which are possible input and output nodes that a Cmd may recognize. A list of (key, form) tuples may also be added to provide information about forms which may have additional nodedata added to them by the Cmd.

Example:

{
    'input': (
        'inet:ipv4',
        'tel:mob:telem',
    ),
    'output': (
        'geo:place',
    ),
    'nodedata': (
        ('foodata', 'inet:http:request'),
        ('bardata', 'inet:ipv4'),
    ),
}

asroot = False

async execStormCmd(runt, genr)[source]: Abstract base method

forms = {}

getArgParser()[source]

classmethod getCmdBrief()[source]

getDescr()[source]

getName()[source]

classmethod getStorNode(form)[source]

isReadOnly()[source]

name = 'cmd'

pkgname = ''

readonly = False

async setArgv(argv)[source]

svciden = ''

class synapse.lib.storm.CopyToCmd(runt, runtsafe)[source]

Bases: Cmd

Copy nodes from the current view into another view.

Examples

// Copy all nodes tagged with #cno.mal.redtree to the target view.

#cno.mal.redtree | copyto 33c971ac77943da91392dadd0eec0571

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'copyto'

class synapse.lib.storm.CountCmd(runt, runtsafe)[source]

Bases: Cmd

Iterate through query results, and print the resulting number of nodes which were lifted. This does not yield the nodes counted, unless the –yield switch is provided.

Example

# Count the number of IPV4 nodes with a given ASN. inet:ipv4:asn=20 | count

# Count the number of IPV4 nodes with a given ASN and yield them. inet:ipv4:asn=20 | count –yield

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'count'

readonly = True

class synapse.lib.storm.DelNodeCmd(runt, runtsafe)[source]

Bases: Cmd

Delete nodes produced by the previous query logic.

(no nodes are returned)

Example

inet:fqdn=vertex.link | delnode

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'delnode'

class synapse.lib.storm.DiffCmd(runt, runtsafe)[source]

Bases: Cmd

Generate a list of nodes with changes in the top layer of the current view.

Examples

// Lift all nodes with any changes

diff

// Lift ou:org nodes that were added in the top layer.

diff –prop ou:org

// Lift inet:ipv4 nodes with the :asn property modified in the top layer.

diff –prop inet:ipv4:asn

// Lift the nodes with the tag #cno.mal.redtree added in the top layer.

diff –tag cno.mal.redtree

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'diff'

readonly = True

class synapse.lib.storm.DivertCmd(runt, runtsafe)[source]

Bases: Cmd

Either consume a generator or yield it’s results based on a conditional.

NOTE: This command is purpose built to facilitate the –yield convention: common to storm commands.
NOTE: The genr argument must not be a function that returns, else it will: be invoked for each inbound node.

Example

divert $cmdopts.yield $fooBarBaz()

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'divert'

class synapse.lib.storm.DmonManager[source]

Bases: Base

Manager for StormDmon objects.

async addDmon(iden, ddef)[source]

getDmon(iden)[source]

getDmonDef(iden)[source]

getDmonDefs()[source]

getDmonRunlog(iden)[source]

async popDmon(iden)[source]: Remove the dmon and fini it if its exists.

async start()[source]: Start all the dmons.

async stop()[source]: Stop all the dmons.

class synapse.lib.storm.EdgesDelCmd(runt, runtsafe)[source]

Bases: Cmd

Bulk delete light edges from input nodes.

Examples

# Delete all “foo” light edges from an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo

# Delete light edges with any verb from a node inet:ipv4=1.2.3.4 | edges.del *

# Delete all “foo” light edges to an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo –n2

async delEdges(node, verb, n2=False)[source]

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'edges.del'

class synapse.lib.storm.GraphCmd(runt, runtsafe)[source]

Bases: Cmd

Generate a subgraph from the given input nodes and command line options.

Example

Using the graph command:

inet:fqdn | graph
            --degrees 2
            --filter { -#nope }
            --pivot { -> meta:seen }
            --form-pivot inet:fqdn {<- * | limit 20}
            --form-pivot inet:fqdn {-> * | limit 20}
            --form-filter inet:fqdn {-inet:fqdn:issuffix=1}
            --form-pivot syn:tag {-> *}
            --form-pivot * {-> #}

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'graph'

class synapse.lib.storm.HelpCmd(runt, runtsafe)[source]

Bases: Cmd

List available information about Storm and brief descriptions of different items.

Notes

If an item is provided, this can be a string or a function.

Examples

// Get all available commands, libraries, types, and their brief descriptions.

help

// Only get commands which have “model” in the name.

help model

// Get help about the base Storm library

help $lib

// Get detailed help about a specific library or library function

help –verbose $lib.print

// Get detailed help about a named Storm type

help –verbose str

// Get help about a method from a $node object

<inbound $node> help $node.tags

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'help'

class synapse.lib.storm.IdenCmd(runt, runtsafe)[source]

Bases: Cmd

Lift nodes by iden.

Example

iden b25bc9eec7e159dce879f9ec85fb791f83b505ac55b346fcb64c3c51e98d1175 | count

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'iden'

readonly = True

class synapse.lib.storm.IntersectCmd(runt, runtsafe)[source]

Bases: Cmd

Yield an intersection of the results of running inbound nodes through a pivot.

Note

This command must consume the entire inbound stream to produce the intersection. This type of stream consuming before yielding results can cause the query to appear laggy in comparison with normal incremental stream operations.

Examples

// Show the it:mitre:attack:technique nodes common to several groups

it:mitre:attack:group*in=(G0006, G0007) | intersect { -> it:mitre:attack:technique }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'intersect'

class synapse.lib.storm.LiftByVerb(runt, runtsafe)[source]

Bases: Cmd

Lift nodes from the current view by an light edge verb.

Examples

# Lift all the n1 nodes for the light edge “foo” lift.byverb “foo”

# Lift all the n2 nodes for the light edge “foo” lift.byverb –n2 “foo”

Notes

Only a single instance of a node will be yielded from this command when that node is lifted via the light edge membership.

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

async iterEdgeNodes(verb, idenset, n2=False)[source]

name = 'lift.byverb'

class synapse.lib.storm.LimitCmd(runt, runtsafe)[source]

Bases: Cmd

Limit the number of nodes generated by the query in the given position.

Example

inet:ipv4 | limit 10

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'limit'

readonly = True

class synapse.lib.storm.MaxCmd(runt, runtsafe)[source]

Bases: Cmd

Consume nodes and yield only the one node with the highest value for an expression.

Examples

// Yield the file:bytes node with the highest :size property file:bytes#foo.bar | max :size

// Yield the file:bytes node with the highest value for $tick file:bytes#foo.bar +.seen ($tick, $tock) = .seen | max $tick

// Yield the it:dev:str node with the longest length it:dev:str | max $lib.len($node.value())

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'max'

readonly = True

class synapse.lib.storm.MergeCmd(runt, runtsafe)[source]

Bases: Cmd

Merge edits from the incoming nodes down to the next layer.

NOTE: This command requires the current view to be a fork.

NOTE: The arguments for including/excluding tags can accept tag glob: expressions for specifying tags. For more information on tag glob expressions, check the Synapse documentation for $node.globtags().

Examples

// Having tagged a new #cno.mal.redtree subgraph in a forked view…

#cno.mal.redtree | merge –apply

// Print out what the merge command would do but dont.

#cno.mal.redtree | merge

// Merge any org nodes with changes in the top layer.

diff | +ou:org | merge –apply

// Merge all tags other than cno.* from ou:org nodes with edits in the // top layer.

diff | +ou:org | merge –only-tags –exclude-tags cno.** –apply

// Merge only tags rep.vt.* and rep.whoxy.* from ou:org nodes with edits // in the top layer.

diff | +ou:org | merge –include-tags rep.vt.* rep.whoxy.* –apply

// Lift only inet:ipv4 nodes with a changed :asn property in top layer // and merge all changes.

diff –prop inet:ipv4:asn | merge –apply

// Lift only nodes with an added #cno.mal.redtree tag in the top layer and merge them.

diff –tag cno.mal.redtree | merge –apply

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'merge'

class synapse.lib.storm.MinCmd(runt, runtsafe)[source]

Bases: Cmd

Consume nodes and yield only the one node with the lowest value for an expression.

Examples

// Yield the file:bytes node with the lowest :size property file:bytes#foo.bar | min :size

// Yield the file:bytes node with the lowest value for $tick file:bytes#foo.bar +.seen ($tick, $tock) = .seen | min $tick

// Yield the it:dev:str node with the shortest length it:dev:str | min $lib.len($node.value())

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'min'

readonly = True

class synapse.lib.storm.MoveNodesCmd(runt, runtsafe)[source]

Bases: Cmd

Move storage nodes between layers.

Storage nodes will be removed from the source layers and the resulting storage node in the destination layer will contain the merged values (merged in bottom up layer order by default).

Examples

// Move storage nodes for ou:org nodes to the top layer

ou:org | movenodes –apply

// Print out what the movenodes command would do but dont.

ou:org | movenodes

// In a view with many layers, only move storage nodes from the bottom layer // to the top layer.

$layers = $lib.view.get().layers $top = $layers.0.iden $bot = $layers.”-1”.iden

ou:org | movenodes –srclayers $bot –destlayer $top

// In a view with many layers, move storage nodes to the top layer and // prioritize values from the bottom layer over the other layers.

$layers = $lib.view.get().layers $top = $layers.0.iden $mid = $layers.1.iden $bot = $layers.2.iden

ou:org | movenodes –precedence $bot $top $mid

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'movenodes'

class synapse.lib.storm.MoveTagCmd(runt, runtsafe)[source]

Bases: Cmd

Rename an entire tag tree and preserve time intervals.

Example

movetag foo.bar baz.faz.bar

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'movetag'

class synapse.lib.storm.OnceCmd(runt, runtsafe)[source]

Bases: Cmd

The once command is used to filter out nodes which have already been processed via the use of a named key. It includes an optional parameter to allow the node to pass the filter again after a given amount of time.

For example, to run an enrichment command on a set of nodes just once:

file:bytes#my.files | once enrich:foo | enrich.foo

The once command filters out any nodes which have previously been through any other use of the “once” command using the same <name> (in this case “enrich:foo”).

You may also specify the –asof option to allow nodes to pass the filter after a given amount of time. For example, the following command will allow any given node through every 2 days:

file:bytes#my.files | once enrich:foo –asof “-2 days” | enrich.foo

Use of “–asof now” or any future date or positive relative time offset will always allow the node to pass the filter.

State tracking data for the once command is stored as nodedata which is stored in your view’s write layer, making it view-specific. So if you have two views, A and B, and they do not share any layers between them, and you execute this query in view A:

inet:ipv4=8.8.8.8 | once enrich:address | enrich.baz

And then you run it in view B, the node will still pass through the once command to the enrich.baz portion of the query because the tracking data for the once command does not yet exist in view B.

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'once'

class synapse.lib.storm.ParallelCmd(runt, runtsafe)[source]

Bases: Cmd

Execute part of a query pipeline in parallel. This can be useful to minimize round-trip delay during enrichments.

Examples

inet:ipv4#foo | parallel { $place = $lib.import(foobar).lookup(:latlong) [ :place=$place ] }

NOTE: Storm variables set within the parallel query pipelines do not interact.

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'parallel'

async nextitem(inq)[source]

async pipeline(runt, query, inq, outq)[source]

readonly = True

class synapse.lib.storm.Parser(prog=None, descr=None, root=None)[source]

Bases: object

add_argument(*names, **opts)[source]

help(mesg=None)[source]

parse_args(argv)[source]

set_inputs(idefs)[source]

class synapse.lib.storm.PureCmd(cdef, runt, runtsafe)[source]

Bases: Cmd

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

getDescr()[source]

getName()[source]

readonly = True

class synapse.lib.storm.ReIndexCmd(runt, runtsafe)[source]

Bases: Cmd

Use admin privileges to re index/normalize node properties.

NOTE: Currently does nothing but is reserved for future use.

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'reindex'

class synapse.lib.storm.RunAsCmd(runt, runtsafe)[source]

Bases: Cmd

Execute a storm query as a specified user.

NOTE: This command requires admin privileges.

Examples

// Create a node as another user. runas someuser { [ inet:fqdn=foo.com ] }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'runas'

class synapse.lib.storm.Runtime[source]

Bases: Base

A Runtime represents the instance of a running query.

The runtime should maintain a firm API boundary using the snap. Parallel query execution requires that the snap be treated as an opaque object which is called through, but not dereferenced.

addInput(node)[source]: Add a Node() object as input to the query runtime.

allowed(perms, gateiden=None, default=None)[source]

allowedEasyPerm(item, perm)[source]

allowedReason(perms, gateiden=None, default=None)[source]: Similar to allowed, but always prefer the default value specified by the caller. Default values are still pulled from permdefs if there is a match there; but still prefer caller default. This results in a ternary response that can be used to know if a rule had a positive/negative or no match. The matching reason metadata is also returned.

cancel()[source]

confirm(perms, gateiden=None, default=None)[source]

Raise AuthDeny if the user doesn’t have the permission.

Notes

An elevated runtime with asroot=True will always return True.

Parameters:

perms (tuple) – The permission tuple.
gateiden (str) – The gateiden.
default (bool) – The default value.

Returns:

If the permission is allowed.

Return type:

True

Raises:

AuthDeny – If the user does not have the permission.

confirmEasyPerm(item, perm, mesg=None)[source]

confirmPropDel(prop, layriden=None)[source]

confirmPropSet(prop, layriden=None)[source]

async coreDynCall(todo, perm=None)[source]

async dyncall(iden, todo, gatekeys=())[source]

async dyniter(iden, todo, gatekeys=())[source]

async emit(item)[source]

async emitter()[source]

async execute(genr=None)[source]

getCmdRuntime(query, opts=None)[source]: Yield a runtime with proper scoping for use in executing a pure storm command.

getGraph()[source]

async getInput()[source]

async getModRuntime(query, opts=None)[source]: Construct a non-context managed runtime for use in module imports.

async getOneNode(propname, valu, filt=None, cmpr='=')[source]: Return exactly 1 node by <prop> <cmpr> <valu>

getOpt(name, defval=None)[source]

getScopeVars()[source]: Return a dict of all the vars within this and all parent scopes.

async getStormQuery(text)[source]

getSubRuntime(query, opts=None)[source]: Yield a runtime with shared scope that will populate changes upward.

async getTeleProxy(url, **opts)[source]

getVar(name, defv=None)[source]

initPath(node)[source]

async initSubRuntime(query, opts=None)[source]: Construct and return sub-runtime with a shared scope. ( caller must fini )

isAdmin(gateiden=None)[source]

isRuntVar(name)[source]

layerConfirm(perms)[source]

async popVar(name)[source]

async printf(mesg)[source]

reqAdmin(gateiden=None, mesg=None)[source]

async reqGateKeys(gatekeys)[source]

async reqUserCanReadLayer(layriden)[source]

setGraph(gdef)[source]

setOpt(name, valu)[source]

async setVar(name, valu)[source]

async storm(text, opts=None, genr=None)[source]: Execute a storm runtime which inherits from this storm runtime.

tick()[source]

async warn(mesg, **info)[source]

async warnonce(mesg, **info)[source]

class synapse.lib.storm.ScrapeCmd(runt, runtsafe)[source]

Bases: Cmd

Use textual properties of existing nodes to find other easily recognizable nodes.

Examples

# Scrape properties from inbound nodes and create standalone nodes. inet:search:query | scrape

# Scrape properties from inbound nodes and make refs light edges to the scraped nodes. inet:search:query | scrape –refs

# Scrape only the :engine and :text props from the inbound nodes. inet:search:query | scrape :text :engine

# Scrape properties inbound nodes and yield newly scraped nodes. inet:search:query | scrape –yield

# Skip re-fanging text before scraping. inet:search:query | scrape –skiprefang

# Limit scrape to specific forms. inet:search:query | scrape –forms (inet:fqdn, inet:ipv4)

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'scrape'

class synapse.lib.storm.SleepCmd(runt, runtsafe)[source]

Bases: Cmd

Introduce a delay between returning each result for the storm query.

NOTE: This is mostly used for testing / debugging.

Example

#foo.bar | sleep 0.5

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'sleep'

readonly = True

class synapse.lib.storm.SpinCmd(runt, runtsafe)[source]

Bases: Cmd

Iterate through all query results, but do not yield any. This can be used to operate on many nodes without returning any.

Example

foo:bar:size=20 [ +#hehe ] | spin

async execStormCmd(runt, genr)[source]: Abstract base method

name = 'spin'

readonly = True

class synapse.lib.storm.StormDmon[source]

Bases: Base

A background storm runtime which is restarted by the cortex.

async bump()[source]

async dmonloop()[source]

pack()[source]

async run()[source]

async stop()[source]

class synapse.lib.storm.TagPruneCmd(runt, runtsafe)[source]

Bases: Cmd

Prune a tag (or tags) from nodes.

This command will delete the tags specified as parameters from incoming nodes, as well as all of their parent tags that don’t have other tags as children.

For example, given a node with the tags:

#parent #parent.child #parent.child.grandchild

Pruning the parent.child.grandchild tag would remove all tags. If the node had the tags:

#parent #parent.child #parent.child.step #parent.child.grandchild

Pruning the parent.child.grandchild tag will only remove the parent.child.grandchild tag as the parent tags still have other children.

Examples

# Prune the parent.child.grandchild tag inet:ipv4=1.2.3.4 | tag.prune parent.child.grandchild

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

hasChildTags(node, tag)[source]

name = 'tag.prune'

class synapse.lib.storm.TeeCmd(runt, runtsafe)[source]

Bases: Cmd

Execute multiple Storm queries on each node in the input stream, joining output streams together.

Commands are executed in order they are given; unless the --parallel switch is provided.

Examples

# Perform a pivot out and pivot in on a inet:ivp4 node inet:ipv4=1.2.3.4 | tee { -> * } { <- * }

# Also emit the inbound node inet:ipv4=1.2.3.4 | tee –join { -> * } { <- * }

# Execute multiple enrichment queries in parallel. inet:ipv4=1.2.3.4 | tee -p { enrich.foo } { enrich.bar } { enrich.baz }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'tee'

async pipeline(runt, outq, genr=None)[source]

readonly = True

class synapse.lib.storm.TreeCmd(runt, runtsafe)[source]

Bases: Cmd

Walk elements of a tree using a recursive pivot.

Examples

# pivot upward yielding each FQDN inet:fqdn=www.vertex.link | tree { :domain -> inet:fqdn }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'tree'

readonly = True

class synapse.lib.storm.UniqCmd(runt, runtsafe)[source]

Bases: Cmd

Filter nodes by their uniq iden values. When this is used a Storm pipeline, only the first instance of a given node is allowed through the pipeline.

A relative property or variable may also be specified, which will cause this command to only allow through the first node with a given value for that property or value rather than checking the node iden.

Examples

# Filter duplicate nodes after pivoting from inet:ipv4 nodes tagged with #badstuff #badstuff +inet:ipv4 ->* | uniq

# Unique inet:ipv4 nodes by their :asn property #badstuff +inet:ipv4 | uniq :asn

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'uniq'

readonly = True

class synapse.lib.storm.ViewExecCmd(runt, runtsafe)[source]

Bases: Cmd

Execute a storm query in a different view.

NOTE: Variables are passed through but nodes are not. The behavior of this command may be non-intuitive in relation to the way storm normally operates. For further information on behavior and limitations when using view.exec, reference the view.exec section of the Synapse User Guide: https://v.vtx.lk/view-exec.

Examples

// Move some tagged nodes to another view inet:fqdn#foo.bar $fqdn=$node.value() | view.exec 95d5f31f0fb414d2b00069d3b1ee64c6 { [ inet:fqdn=$fqdn ] }

async execStormCmd(runt, genr)[source]: Abstract base method

getArgParser()[source]

name = 'view.exec'

readonly = True

synapse.lib.storm_format module

class synapse.lib.storm_format.StormLexer(parser)[source]

Bases: Lexer

get_tokens_unprocessed(text)[source]

This method should process the text and return an iterable of (index, tokentype, value) tuples where index is the starting position of the token within the input text.

It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness.

synapse.lib.storm_format.highlight_storm(parser, text)[source]: Prints a storm query with syntax highlighting

synapse.lib.stormctrl module

exception synapse.lib.stormctrl.StormBreak(item=None)[source]: Bases: StormCtrlFlow

exception synapse.lib.stormctrl.StormContinue(item=None)[source]: Bases: StormCtrlFlow

exception synapse.lib.stormctrl.StormCtrlFlow(item=None)[source]: Bases: Exception

exception synapse.lib.stormctrl.StormExit(item=None)[source]: Bases: StormCtrlFlow

exception synapse.lib.stormctrl.StormReturn(item=None)[source]: Bases: StormCtrlFlow

exception synapse.lib.stormctrl.StormStop(item=None)[source]: Bases: StormCtrlFlow

synapse.lib.stormhttp module

class synapse.lib.stormhttp.HttpResp(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a HTTP response.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormhttp.LibHttp(runt, name=())[source]

Bases: Lib

A Storm Library exposing an HTTP client API.

For APIs that accept an ssl_opts argument, the dictionary may contain the following values:

{
    'verify': <bool> - Perform SSL/TLS verification. Is overridden by the ssl_verify argument.
    'client_cert': <str> - PEM encoded full chain certificate for use in mTLS.
    'client_key': <str> - PEM encoded key for use in mTLS. Alternatively, can be included in client_cert.
}

async codereason(code)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async inetHttpConnect(url, headers=None, ssl_verify=True, timeout=300, params=None, proxy=None, ssl_opts=None)[source]

strify(item)[source]

async urldecode(text)[source]

async urlencode(text)[source]

class synapse.lib.stormhttp.WebSocket[source]

Bases: Base, StormType

Implements the Storm API for a Websocket.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async rx(timeout=None)[source]

async tx(mesg)[source]

synapse.lib.stormsvc module

class synapse.lib.stormsvc.StormSvc[source]

Bases: object

The StormSvc mixin class used to make a remote storm service with commands.

async getStormSvcInfo()[source]

async getStormSvcPkgs()[source]

class synapse.lib.stormsvc.StormSvcClient[source]

Bases: Base

A StormService is a wrapper for a telepath proxy to a service accessible from the storm runtime.

synapse.lib.stormtypes module

class synapse.lib.stormtypes.Bool(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a boolean instance.

class synapse.lib.stormtypes.Bytes(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a Bytes object.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.CmdOpts(valu, path=None)[source]

Bases: Dict

A dictionary like object that holds a reference to a command options namespace. ( This allows late-evaluation of command arguments rather than forcing capture )

async deref(name)[source]

async iter()[source]

async setitem(name, valu)[source]

async stormrepr()[source]

async value()[source]

class synapse.lib.stormtypes.CronJob(runt, cdef, path=None)[source]

Bases: Prim

Implements the Storm api for a cronjob instance.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.Dict(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a Dictionary object.

async deref(name)[source]

async iter()[source]

async setitem(name, valu)[source]

async stormrepr()[source]

async value()[source]

class synapse.lib.stormtypes.Layer(runt, ldef, path=None)[source]

Bases: Prim

Implements the Storm api for a layer instance.

async getEdges()[source]

async getEdgesByN1(nodeid)[source]

async getEdgesByN2(nodeid)[source]

async getMirrorStatus()[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async getStorNode(nodeid)[source]

async getStorNodes()[source]

async liftByProp(propname, propvalu=None, propcmpr='=')[source]

async liftByTag(tagname, formname=None)[source]

async verify(config=None)[source]

class synapse.lib.stormtypes.Lib(runt, name=())[source]

Bases: StormType

A collection of storm methods under a name

addLibFuncs()[source]

async deref(name)[source]

async dyncall(iden, todo, gatekeys=())[source]

async dyniter(iden, todo, gatekeys=())[source]

async initLibAsync()[source]

async stormrepr()[source]

class synapse.lib.stormtypes.LibAxon(runt, name=())[source]

Bases: Lib

A Storm library for interacting with the Cortex’s Axon.

For APIs that accept an ssl_opts argument, the dictionary may contain the following values:

{
    'verify': <bool> - Perform SSL/TLS verification. Is overridden by the ssl argument.
    'client_cert': <str> - PEM encoded full chain certificate for use in mTLS.
    'client_key': <str> - PEM encoded key for use in mTLS. Alternatively, can be included in client_cert.
}

async csvrows(sha256, dialect='excel', errors='ignore', **fmtparams)[source]

async del_(sha256)[source]

async dels(sha256s)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async has(sha256)[source]

async hashset(sha256)[source]

async jsonlines(sha256, errors='ignore')[source]

async list(offs=0, wait=False, timeout=None)[source]

async metrics()[source]

async put(byts)[source]

async readlines(sha256, errors='ignore')[source]

async size(sha256)[source]

strify(item)[source]

async upload(genr)[source]

async urlfile(*args, **kwargs)[source]

async wget(url, headers=None, params=None, method='GET', json=None, body=None, ssl=True, timeout=None, proxy=None, ssl_opts=None)[source]

async wput(sha256, url, headers=None, params=None, method='PUT', ssl=True, timeout=None, proxy=None, ssl_opts=None)[source]

class synapse.lib.stormtypes.LibBase(runt, name=())[source]

Bases: Lib

The Base Storm Library. This mainly contains utility functionality.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async trycast(name, valu)[source]

class synapse.lib.stormtypes.LibBase64(runt, name=())[source]

Bases: Lib

A Storm Library for encoding and decoding base64 data.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibBytes(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with bytes storage. This Library is deprecated; use $lib.axon.* instead.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibCron(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Cron Jobs in the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibCsv(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with csvtool.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibDict(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with dictionaries.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibDmon(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with StormDmons.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibExport(runt, name=())[source]

Bases: Lib

A Storm Library for exporting data.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async toaxon(query, opts=None)[source]

class synapse.lib.stormtypes.LibFeed(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Cortex feed functions.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibGlobals(runt, name)[source]

Bases: Lib

A Storm Library for interacting with global variables which are persistent across the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibJsonStor(runt, name=())[source]

Bases: Lib

Implements cortex JSON storage.

addLibFuncs()[source]

async cachedel(path, key)[source]

async cacheget(path, key, asof='now', envl=False)[source]

async cacheset(path, key, valu)[source]

async get(path, prop=None)[source]

async has(path)[source]

async iter(path=None)[source]

async set(path, valu, prop=None)[source]

class synapse.lib.stormtypes.LibLayer(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Layers in the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibLift(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with lift helpers.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibPipe(runt, name=())[source]

Bases: Lib

A Storm library for interacting with non-persistent queues.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibPkg(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Storm Packages.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibPs(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with running tasks on the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibQueue(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with persistent Queues in the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibRegx(runt, name=())[source]

Bases: Lib

A Storm library for searching/matching with regular expressions.

async escape(text)[source]

async findall(pattern, text, flags=0)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async matches(pattern, text, flags=0)[source]

async replace(pattern, replace, text, flags=0)[source]

async search(pattern, text, flags=0)[source]

class synapse.lib.stormtypes.LibService(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Storm Services.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibStr(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with strings.

async concat(*args)[source]

async format(text, **kwargs)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async join(sepr, items)[source]

class synapse.lib.stormtypes.LibTags(runt, name=())[source]

Bases: Lib

Storm utility functions for tags.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async prefix(names, prefix, ispart=False)[source]

class synapse.lib.stormtypes.LibTelepath(runt, name=())[source]

Bases: Lib

A Storm Library for making Telepath connections to remote services.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibTime(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with timestamps.

async day(tick)[source]

async dayofmonth(tick)[source]

async dayofweek(tick)[source]

async dayofyear(tick)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async hour(tick)[source]

async minute(tick)[source]

async month(tick)[source]

async monthofyear(tick)[source]

async second(tick)[source]

async toUTC(tick, timezone)[source]

async year(tick)[source]

class synapse.lib.stormtypes.LibTrigger(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Triggers in the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibVars(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with runtime variables.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.LibView(runt, name=())[source]

Bases: Lib

A Storm Library for interacting with Views in the Cortex.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.List(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a List instance.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async iter()[source]

async setitem(name, valu)[source]

async stormrepr()[source]

async value()[source]

class synapse.lib.stormtypes.Node(node, path=None)[source]

Bases: Prim

Implements the Storm api for a node instance.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.NodeData(node, path=None)[source]

Bases: Prim

A Storm Primitive representing the NodeData stored for a Node.

async cacheget(name, asof='now')[source]

async cacheset(name, valu)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.NodeProps(node, path=None)[source]

Bases: Prim

A Storm Primitive representing the properties on a Node.

async get(name)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async iter()[source]

async list()[source]

async set(prop, valu)[source]

async setitem(name, valu)[source]

Set a property on a Node.

Parameters:

name (str) – The name of the property to set.
valu – The value being set.

Raises:

s_exc – NoSuchProp: If the property being set is not valid for the node.
s_exc.BadTypeValu – If the value of the property fails to normalize.

value()[source]

class synapse.lib.stormtypes.Number(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a Number instance.

Storm Numbers are high precision fixed point decimals corresponding to the the hugenum storage type.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async stormrepr()[source]

class synapse.lib.stormtypes.Path(node, path=None)[source]

Bases: Prim

Implements the Storm API for the Path object.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.PathMeta(path)[source]

Bases: Prim

Put the storm deref/setitem/iter convention on top of path meta information.

async deref(name)[source]

async iter()[source]

async setitem(name, valu)[source]

class synapse.lib.stormtypes.PathVars(path)[source]

Bases: Prim

Put the storm deref/setitem/iter convention on top of path variables.

async deref(name)[source]

async iter()[source]

async setitem(name, valu)[source]

class synapse.lib.stormtypes.Pipe(runt, size)[source]

Bases: StormType

A Storm Pipe provides fast ephemeral queues.

async close()[source]: Close the pipe for writing. This will cause the slice()/slices() API to return once drained.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.Prim(valu, path=None)[source]

Bases: StormType

The base type for all Storm primitive values.

async bool()[source]

async iter()[source]

async nodes()[source]

async stormrepr()[source]

value()[source]

class synapse.lib.stormtypes.Proxy(runt, proxy, path=None)[source]

Bases: StormType

Implements the Storm API for a Telepath proxy.

These can be created via $lib.telepath.open(). Storm Service objects are also Telepath proxy objects.

Methods called off of these objects are executed like regular Telepath RMI calls.

An example of calling a method which returns data:

$prox = $lib.telepath.open($url)
$result = $prox.doWork($data)
return ( $result )

An example of calling a method which is a generator:

$prox = $lib.telepath.open($url)
for $item in $prox.genrStuff($data) {
    $doStuff($item)
}

async deref(name)[source]

async stormrepr()[source]

class synapse.lib.stormtypes.ProxyGenrMethod(meth, path=None)[source]

Bases: StormType

Implements the generator methods for the telepath:proxy.

An example of calling a method which is a generator:

$prox = $lib.telepath.open($url)
for $item in $prox.genrStuff($data) {
    $doStuff($item)
}

async stormrepr()[source]

class synapse.lib.stormtypes.ProxyMethod(runt, meth, path=None)[source]

Bases: StormType

Implements the call methods for the telepath:proxy.

An example of calling a method which returns data:

$prox = $lib.telepath.open($url)
$result = $prox.doWork($data)
$doStuff($result)

async stormrepr()[source]

class synapse.lib.stormtypes.Query(text, varz, runt, path=None)[source]

Bases: Prim

A storm primitive representing an embedded query.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async iter()[source]

async nodes()[source]

async stormrepr()[source]

class synapse.lib.stormtypes.Queue(runt, name, info)[source]

Bases: StormType

A StormLib API instance of a named channel in the Cortex multiqueue.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async stormrepr()[source]

class synapse.lib.stormtypes.Service(runt, ssvc)[source]

Bases: Proxy

async deref(name)[source]

class synapse.lib.stormtypes.Set(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a Set object.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async iter()[source]

async stormrepr()[source]

class synapse.lib.stormtypes.StormHiveDict(runt, info)[source]

Bases: Prim

A Storm Primitive representing a HiveDict.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async iter()[source]

value()[source]

class synapse.lib.stormtypes.StormType(path=None)[source]

Bases: object

The base type for storm runtime value objects.

async deref(name)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

ismutable()[source]

async setitem(name, valu)[source]

class synapse.lib.stormtypes.StormTypesRegistry[source]

Bases: object

addStormLib(path, ctor)[source]

addStormType(path, ctor)[source]

base_undefined_types = ('any', 'int', 'lib', 'null', 'time', 'prim', 'undef', 'float', 'generator')

delStormLib(path)[source]

delStormType(path)[source]

getLibDocs(lib=None)[source]

getTypeDocs(styp: str = None)[source]

iterLibs()[source]

iterTypes()[source]

known_types = {'aha:pool', 'auth:gate', 'auth:role', 'auth:user', 'auth:user:json', 'auth:user:profile', 'auth:user:vars', 'boolean', 'bytes', 'cache:fixed', 'cmdopts', 'cronjob', 'dict', 'hive:dict', 'http:api', 'http:api:methods', 'http:api:perms', 'http:api:request', 'http:api:request:headers', 'http:api:vars', 'inet:http:oauth:v1:client', 'inet:http:resp', 'inet:http:socket', 'inet:imap:server', 'inet:smtp:message', 'json:schema', 'layer', 'list', 'model:form', 'model:property', 'model:tagprop', 'model:type', 'node', 'node:data', 'node:path', 'node:path:meta', 'node:path:vars', 'node:props', 'number', 'pipe', 'proj:comment', 'proj:comments', 'proj:epic', 'proj:epics', 'proj:project', 'proj:sprint', 'proj:sprints', 'proj:ticket', 'proj:tickets', 'queue', 'set', 'spooled:set', 'stat:tally', 'stix:bundle', 'storm:query', 'str', 'telepath:proxy', 'telepath:proxy:genrmethod', 'telepath:proxy:method', 'text', 'trigger', 'vault', 'vault:data', 'view', 'xml:element'}

registerLib(ctor)[source]: Decorator to register a StormLib

registerType(ctor)[source]: Decorator to register a StormPrim

rtypes = {}

undefined_types = {'any', 'float', 'generator', 'int', 'lib', 'null', 'prim', 'time', 'undef'}

class synapse.lib.stormtypes.Str(valu, path=None)[source]

Bases: Prim

Implements the Storm API for a String object.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.Text(valu, path=None)[source]

Bases: Prim

A mutable text type for simple text construction.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

class synapse.lib.stormtypes.Trigger(runt, tdef)[source]

Bases: Prim

Implements the Storm API for a Trigger.

async deref(name)[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async move(viewiden)[source]

async pack()[source]

async set(name, valu)[source]

class synapse.lib.stormtypes.Undef[source]

Bases: object

async stormrepr()[source]

class synapse.lib.stormtypes.View(runt, vdef, path=None)[source]

Bases: Prim

Implements the Storm api for a View instance.

async addNode(form, valu, props=None)[source]

async delMergeRequest()[source]

async delMergeVote(useriden=None)[source]

async detach()[source]

async getMergeRequest()[source]

async getMergeRequestSummary()[source]

async getMerges()[source]

async getMergingViews()[source]

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

async setMergeComment(comment)[source]

async setMergeRequest(comment=None)[source]

async setMergeVote(approved=True, comment=None)[source]

async setMergeVoteComment(comment)[source]

async viewDynCall(todo, perm)[source]

async viewDynIter(todo, perm)[source]

synapse.lib.stormtypes.allowed(perm, gateiden=None)[source]

synapse.lib.stormtypes.allowedEasyPerm(item, perm)[source]

synapse.lib.stormtypes.confirm(perm, gateiden=None)[source]

synapse.lib.stormtypes.confirmEasyPerm(item, perm, mesg=None)[source]

synapse.lib.stormtypes.fromprim(valu, path=None, basetypes=True)[source]

synapse.lib.stormtypes.getCallSig(func) → Signature[source]: Get the callsig of a function, stripping self if present.

synapse.lib.stormtypes.getDoc(obj, errstr)[source]: Helper to get __doc__

synapse.lib.stormtypes.intify(x)[source]

synapse.lib.stormtypes.ismutable(valu)[source]

async synapse.lib.stormtypes.kwarg_format(_text, **kwargs)[source]: Replaces instances curly-braced argument names in text with their values

synapse.lib.stormtypes.stormfunc(readonly=False)[source]

async synapse.lib.stormtypes.tobool(valu, noneok=False)[source]

async synapse.lib.stormtypes.tobuidhex(valu, noneok=False)[source]

async synapse.lib.stormtypes.tocmprvalu(valu)[source]

async synapse.lib.stormtypes.toint(valu, noneok=False)[source]

async synapse.lib.stormtypes.toiter(valu, noneok=False)[source]

async synapse.lib.stormtypes.tonumber(valu, noneok=False)[source]

async synapse.lib.stormtypes.toprim(valu, path=None)[source]

async synapse.lib.stormtypes.torepr(valu, usestr=False)[source]

async synapse.lib.stormtypes.tostor(valu)[source]

async synapse.lib.stormtypes.tostr(valu, noneok=False)[source]

async synapse.lib.stormtypes.totype(valu, basetypes=False) → str[source]

Convert a value to its Storm type string.

Parameters:

valu – The object to check.
basetypes (bool) – If True, return the base Python class name as a fallback.

Returns:

The type name.

Return type:

str

Raises:

StormRuntimeError – If the valu does not resolve to a known type and basetypes=False.

async synapse.lib.stormtypes.typeerr(name, reqt)[source]

synapse.lib.stormwhois module

class synapse.lib.stormwhois.LibWhois(runt, name=())[source]

Bases: Lib

A Storm Library for providing a consistent way to generate guids for WHOIS / Registration Data in Storm.

getObjLocals()[source]

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns:: A key/value pairs.
Return type:: dict

synapse.lib.structlog module

class synapse.lib.structlog.JsonFormatter(*args, **kwargs)[source]

Bases: Formatter

format(record: LogRecord)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

synapse.lib.task module

class synapse.lib.task.Task[source]

Bases: Base

The synapse Task object implements concepts similar to process trees for asyncio.Task instances.

async kill()[source]

pack()[source]

async worker(coro, name='worker')[source]

synapse.lib.task.current()[source]: Return the current synapse task.

async synapse.lib.task.executor(func, *args, **kwargs)[source]

Execute a function in an executor thread.

Parameters:: todo ((func,args,kwargs)) – A todo tuple.

synapse.lib.task.loop()[source]

synapse.lib.task.user()[source]: Return the current task user.

synapse.lib.task.username()[source]: Return the current task user name.

synapse.lib.task.vardefault(name, func)[source]

Add a default constructor for a particular task-local variable

All future calls to taskVarGet with the same name will return the result of calling func

synapse.lib.task.varget(name, defval=None, task=None)[source]

Access a task local variable by name

Precondition:: If task is None, this must be called from task context

synapse.lib.task.varinit(task=None)[source]

Initializes (or re-initializes for testing purposes) all of a task’s task-local variables

Precondition:: If task is None, this must be called from task context

synapse.lib.task.varset(name, valu, task=None)[source]

Set a task-local variable

Parameters:: task – If task is None, uses current task

Precondition:: If task is None, this must be called from task context

synapse.lib.thishost module

synapse.lib.thishost.get(prop)[source]

Retrieve a property from the hostinfo dictionary.

Example

import synapse.lib.thishost as s_thishost

if s_thishost.get(‘platform’) == ‘windows’:: dostuff()

synapse.lib.thishost.hostaddr(dest='8.8.8.8')[source]

Retrieve the ipv4 address for this host ( optionally as seen from dest ). .. rubric:: Example

addr = s_socket.hostaddr()

synapse.lib.thisplat module

synapse.lib.threads module

synapse.lib.threads.current()[source]

synapse.lib.threads.iden()[source]

synapse.lib.time module

Time related utilities for synapse “epoch millis” time values.

synapse.lib.time.day(tick)[source]

synapse.lib.time.dayofmonth(tick)[source]

synapse.lib.time.dayofweek(tick)[source]

synapse.lib.time.dayofyear(tick)[source]

synapse.lib.time.delta(text)[source]: Parse a simple time delta string and return the delta.

synapse.lib.time.hour(tick)[source]

synapse.lib.time.ival(*times)[source]

synapse.lib.time.minute(tick)[source]

synapse.lib.time.month(tick)[source]

synapse.lib.time.parse(text, base=None, chop=False)[source]

Parse a time string into an epoch millis value.

Parameters:

text (str) – Time string to parse
base (int or None) – Milliseconds to offset the time from
chop (bool) – Whether to chop the digit-only string to 17 chars

Returns:

Epoch milliseconds

Return type:

int

synapse.lib.time.parsetz(text)[source]

Parse timezone from time string, with UTC as the default.

Parameters:: text (str) – Time string
Returns:: A tuple of text with tz chars removed and base milliseconds to offset time.
Return type:: tuple

synapse.lib.time.repr(tick, pack=False)[source]

Return a date string for an epoch-millis timestamp.

Parameters:: tick (int) – The timestamp in milliseconds since the epoch.
Returns:: A date time string
Return type:: (str)

synapse.lib.time.second(tick)[source]

synapse.lib.time.toUTC(tick, fromzone)[source]

synapse.lib.time.wildrange(text)[source]: Parse an interval from a wild card time stamp: 2021/10/31*

synapse.lib.time.year(tick)[source]

synapse.lib.trigger module

class synapse.lib.trigger.Trigger(view, tdef)[source]

Bases: object

async execute(node, vars=None)[source]: Actually execute the query

get(name)[source]

getStorNode(form)[source]

pack()[source]

async set(name, valu)[source]: Set one of the dynamic elements of the trigger definition.

class synapse.lib.trigger.Triggers(view)[source]

Bases: object

Manages “triggers”, conditions where changes in data result in new storm queries being executed.

Note

These methods should not be called directly under normal circumstances. Use the owning “View” object to ensure that mirrors/clusters members get the same changes.

get(iden)[source]

list()[source]

async load(tdef)[source]

pop(iden)[source]

async runEdgeAdd(n1, verb, n2)[source]

async runEdgeDel(n1, verb, n2)[source]

async runNodeAdd(node)[source]

async runNodeDel(node)[source]

async runPropSet(node, prop, oldv)[source]

async runTagAdd(node, tag)[source]

async runTagDel(node, tag)[source]

synapse.lib.trigger.reqValidTdef(conf)[source]

synapse.lib.types module

class synapse.lib.types.Array(modl, name, info, opts)[source]

Bases: Type

isarray = True

postTypeInit()[source]

repr(valu)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

class synapse.lib.types.Bool(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(valu)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 2

class synapse.lib.types.Comp(modl, name, info, opts)[source]

Bases: Type

getCompOffs(name)[source]: If this type is a compound, return the field offset for the given property name or None.

postTypeInit()[source]

repr(valu)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Data(modl, name, info, opts)[source]

Bases: Type

norm(valu)[source]

Normalize the value for a given type.

Parameters:: valu (obj) – The value to normalize.
Returns:: The normalized valu, info tuple.
Return type:: ((obj,dict))

Notes

The info dictionary uses the following key conventions:: subs (dict): The normalized sub-fields as name: valu entries.

postTypeInit()[source]

stortype: int = 13

class synapse.lib.types.Duration(modl, name, info, opts)[source]

Bases: IntBase

postTypeInit()[source]

repr(valu)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 5

class synapse.lib.types.Edge(modl, name, info, opts)[source]

Bases: Type

getCompOffs(name)[source]: If this type is a compound, return the field offset for the given property name or None.

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.FieldHelper(modl, tname, fields)[source]

Bases: defaultdict

Helper for Comp types. Performs Type lookup/creation upon first use.

class synapse.lib.types.Float(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 22

class synapse.lib.types.Guid(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

stortype: int = 10

class synapse.lib.types.Hex(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

stortype: int = 1

class synapse.lib.types.HugeNum(modl, name, info, opts)[source]

Bases: Type

norm(valu)[source]

Normalize the value for a given type.

Parameters:: valu (obj) – The value to normalize.
Returns:: The normalized valu, info tuple.
Return type:: ((obj,dict))

Notes

The info dictionary uses the following key conventions:: subs (dict): The normalized sub-fields as name: valu entries.

stortype: int = 23

class synapse.lib.types.Int(modl, name, info, opts)[source]

Bases: IntBase

merge(oldv, newv)[source]

Allow types to “merge” data from two sources based on value precedence.

Parameters:

valu (object) – The current value.
newv (object) – The updated value.

Returns:

The merged value.

Return type:

(object)

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

class synapse.lib.types.IntBase(modl, name, info, opts)[source]: Bases: Type

class synapse.lib.types.Ival(modl, name, info, opts)[source]

Bases: Type

An interval, i.e. a range, of times

merge(oldv, newv)[source]

Allow types to “merge” data from two sources based on value precedence.

Parameters:

valu (object) – The current value.
newv (object) – The updated value.

Returns:

The merged value.

Return type:

(object)

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 12

class synapse.lib.types.Loc(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stems(valu)[source]

stortype: int = 15

class synapse.lib.types.Ndef(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.NodeProp(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

stortype: int = 13

class synapse.lib.types.Range(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Str(modl, name, info, opts)[source]

Bases: Type

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 1

class synapse.lib.types.Tag(modl, name, info, opts)[source]

Bases: Str

postTypeInit()[source]

class synapse.lib.types.TagPart(modl, name, info, opts)[source]

Bases: Str

postTypeInit()[source]

class synapse.lib.types.Taxon(modl, name, info, opts)[source]

Bases: Str

postTypeInit()[source]

class synapse.lib.types.Taxonomy(modl, name, info, opts)[source]

Bases: Str

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

class synapse.lib.types.Time(modl, name, info, opts)[source]

Bases: IntBase

getTickTock(vals)[source]

Get a tick, tock time pair.

Parameters:: vals (list) – A pair of values to norm.
Returns:: A ordered pair of integers.
Return type:: (int, int)

merge(oldv, newv)[source]

Allow types to “merge” data from two sources based on value precedence.

Parameters:

valu (object) – The current value.
newv (object) – The updated value.

Returns:

The merged value.

Return type:

(object)

postTypeInit()[source]

repr(valu)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 11

class synapse.lib.types.TimeEdge(modl, name, info, opts)[source]

Bases: Edge

getCompOffs(name)[source]: If this type is a compound, return the field offset for the given property name or None.

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Type(modl, name, info, opts)[source]

Bases: object

clone(opts)[source]

Create a new instance of this type with the specified options.

Parameters:: opts (dict) – The type specific options for the new instance.

cmpr(val1, name, val2)[source]: Compare the two values using the given type specific comparator.

extend(name, opts, info)[source]

Extend this type to construct a sub-type.

Parameters:

name (str) – The name of the new sub-type.
opts (dict) – The type options for the sub-type.
info (dict) – The type info for the sub-type.

Returns:

A new sub-type instance.

Return type:

(synapse.types.Type)

getCmprCtor(name)[source]

getCompOffs(name)[source]: If this type is a compound, return the field offset for the given property name or None.

getLiftHintCmpr(valu, cmpr)[source]

getLiftHintCmprCtor(name)[source]

getStorCmprs(cmpr, valu)[source]

getStorNode(form)[source]

getTypeDef()[source]

getTypeVals(valu)[source]

isarray = False

merge(oldv, newv)[source]

Allow types to “merge” data from two sources based on value precedence.

Parameters:

valu (object) – The current value.
newv (object) – The updated value.

Returns:

The merged value.

Return type:

(object)

norm(valu)[source]

Normalize the value for a given type.

Parameters:: valu (obj) – The value to normalize.
Returns:: The normalized valu, info tuple.
Return type:: ((obj,dict))

Notes

The info dictionary uses the following key conventions:: subs (dict): The normalized sub-fields as name: valu entries.

pack()[source]

postTypeInit()[source]

repr(norm)[source]: Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

setCmprCtor(name, func)[source]

Set a comparator ctor for a given named comparison operation.

Parameters:

name (str) – Name of the comparison operation.
func – Function which returns a comparator.

Notes

Comparator ctors should expect to get the right-hand-side of the comparison as their argument, and the returned function should expect to get the left hand side of the comparison and return a boolean from there.

setLiftHintCmprCtor(name, func)[source]

setNormFunc(typo, func)[source]

Register a normalizer function for a given python type.

Parameters:

typo (type) – A python type/class to normalize.
func (function) – A callback which normalizes a python value.

stortype: int = None

class synapse.lib.types.Velocity(modl, name, info, opts)[source]

Bases: IntBase

oflight = 299792458000

postTypeInit()[source]

stortype: int = 9

synapse.lib.urlhelp module

synapse.lib.urlhelp.chopurl(url)[source]

A sane “stand alone” url parser.

Example

info = chopurl(url)

synapse.lib.urlhelp.sanitizeUrl(url)[source]

Returns a URL with the password (if present) replaced with ****

RFC 3986 3.2.1 ‘Applications should not render as clear text any data after the first colon (“:”) character found within a userinfo subcomponent unless the data after the colon is the empty string (indicating no password)’

Essentially, replace everything between the 2nd colon (if it exists) and the first succeeding at sign. Return the original string otherwise.

Note: this depends on this being a reasonably-well formatted URI that starts with a scheme (e.g. http) and ‘//:’ Failure of this condition yields the original string.

synapse.lib.version module

Synapse utilites for dealing with Semvar versioning. This includes the Synapse version information.

synapse.lib.version.fmtVersion(*vsnparts)[source]

Join a string of parts together with a . separator.

Parameters:: *vsnparts –

Returns:

synapse.lib.version.matches(vers, cmprvers)[source]: Check if a version string matches a version comparison string.

synapse.lib.version.packVersion(major, minor=0, patch=0)[source]

Pack a set of major/minor/patch integers into a single integer for storage.

Parameters:

major (int) – Major version level integer.
minor (int) – Minor version level integer.
patch (int) – Patch version level integer.

Returns:

System normalized integer value to represent a software version.

Return type:

int

synapse.lib.version.parseSemver(text)[source]

Parse a Semantic Version string into is component parts.

Parameters:

text (str) – A text string to parse into semver components. This string has whitespace and leading ‘v’
it. (characters stripped off of) –

Examples

Parse a string into it semvar parts:

parts = parseSemver('v1.2.3')

Returns:: The dictionary will contain the keys ‘major’, ‘minor’ and ‘patch’ pointing to integer values. The dictionary may also contain keys for ‘build’ and ‘pre’ information if that data is parsed out of a semver string. None is returned if the string is not a valid Semver string.
Return type:: dict

synapse.lib.version.parseVersionParts(text, seps=('.', '-', '_', '+'))[source]

Extract a list of major/minor/version integer strings from a string.

Parameters:

text (str) – String to parse
seps (tuple) – A tuple or list of separators to use when parsing the version string.

Examples

Parse a simple version string into a major and minor parts:

parts = parseVersionParts('1.2')

Parse a complex version string into a major and minor parts:

parts = parseVersionParts('wowsoft_1.2')

Parse a simple version string into a major, minor and patch parts. Parts after the “3.” are dropped from the results:

parts = parseVersionParts('1.2.3.4.5')

Notes

This attempts to brute force out integers from the version string by stripping any leading ascii letters and part separators, and then regexing out numeric parts optionally followed by part separators. It will stop at the first mixed-character part encountered. For example, “1.2-3a” would only parse out the “1” and “2” from the string.

Returns:: Either a empty dictionary or dictionary containing up to three keys, ‘major’, ‘minor’ and ‘patch’.
Return type:: dict

synapse.lib.version.reqVersion(valu, reqver, exc=<class 'synapse.exc.BadVersion'>, mesg='Provided version does not match required version.')[source]

Require a given version tuple is valid for a given requirements string.

Parameters:

Optional[Tuple[int (valu) – Major, minor and patch value to check.
int – Major, minor and patch value to check.
int]] – Major, minor and patch value to check.
reqver (str) – A requirements version string.
exc (s_exc.SynErr) – The synerr class to raise.
mesg (str) – The message to pass in the exception.

Returns:

If the value is in bounds of minver and maxver.

Return type:

None

Raises:

s_exc.BadVersion – If a precondition is incorrect or a version value is out of bounds.

synapse.lib.version.unpackVersion(ver)[source]

Unpack a system normalized integer representing a softare version into its component parts.

Parameters:: ver (int) – System normalized integer value to unpack into a tuple.
Returns:: A tuple containing the major, minor and patch values shifted out of the integer.
Return type:: (int, int, int)

synapse.lib.view module

class synapse.lib.view.View[source]

Bases: Pusher

A view represents a cortex as seen from a specific set of layers.

The view class is used to implement Copy-On-Write layers as well as interact with a subset of the layers configured in a Cortex.

async addLayer(layriden, indx=None)[source]

async addNode(form, valu, props=None, user=None)[source]

async addNodeEdits(edits, meta)[source]

A telepath compatible way to apply node edits to a view.

NOTE: This does cause trigger execution.

async addTrigQueue(triginfo, nexsitem)[source]

async addTrigger(tdef)[source]: Adds a trigger to the view.

async callStorm(text, opts=None)[source]

async callStormIface(name, todo)[source]

async delMergeRequest()[source]

async delMergeVote(useriden)[source]

async delTrigQueue(offs)[source]

async delTrigger(iden)[source]

async delete()[source]

Delete the metadata for this view.

Note: this does not delete any layer storage.

async detach()[source]: Detach the view from its parent but do not change the layers. ( this is not reversible! )

async eval(text, opts=None, log_info=None)[source]: Evaluate a storm query and yield Nodes only.

async finiMergeTask()[source]

async finiTrigTask()[source]

async fork(ldef=None, vdef=None)[source]

Make a new view inheriting from this view with the same layers and a new write layer on top

Parameters:

ldef – layer parameter dict
vdef – view parameter dict
cortex.addLayer (Passed through to) –

Returns:

new view object, with an iden the same as the new write layer iden

async getEdgeVerbs()[source]

async getEdges(verb=None)[source]

async getFormCounts()[source]

getMergeRequest()[source]

async getMergeVotes()[source]

async getMerges()[source]: Yield the historical merges into this view.

async getMergingViews()[source]

async getPropArrayCount(propname, valu=<synapse.common.NoValu object>)[source]

async getPropCount(propname, valu=<synapse.common.NoValu object>)[source]

async getStorNodes(buid)[source]: Return a list of storage nodes for the given buid in layer order.

async getTagPropCount(form, tag, propname, valu=<synapse.common.NoValu object>)[source]

async getTrigger(iden)[source]

hasKids()[source]

init2()[source]: We have a second round of initialization so the views can get a handle to their parents which might not be initialized yet

async initMergeTask()[source]

async initTrigTask()[source]

isForkOf(viewiden)[source]

async isMergeReady()[source]

isafork()[source]

async iterStormPodes(text, opts=None)[source]

async listTriggers()[source]: List all the triggers in the view.

async merge(useriden=None, force=False)[source]: Merge this view into its parent. All changes made to this view will be applied to the parent. Parent’s triggers will be run.

async mergeAllowed(user=None, force=False)[source]

Check whether a user can merge a view into its parent.

NOTE: This API may not be used to check for merges based on quorum votes.

async mergeStormIface(name, todo)[source]: Allow an interface which specifies a generator use case to yield (priority, value) tuples and merge results from multiple generators yielded in ascending priority order.

async nodes(text, opts=None)[source]: A simple non-streaming way to return a list of nodes.

async pack()[source]

reqNoParentQuorum()[source]

reqParentQuorum()[source]

reqValidVoter(useriden)[source]

async runEdgeAdd(n1, edge, n2)[source]

async runEdgeDel(n1, edge, n2)[source]

async runNodeAdd(node)[source]

async runNodeDel(node)[source]

async runPropSet(node, prop, oldv)[source]: Handle when a prop set trigger event fired

async runTagAdd(node, tag, valu)[source]

async runTagDel(node, tag, valu)[source]

async runViewMerge()[source]

async scrapeIface(text, unique=False, refang=True)[source]

async setLayers(layers)[source]: Set the view layers from a list of idens. NOTE: view layers are stored “top down” (the write layer is self.layers[0])

async setMergeComment(comment)[source]

async setMergeRequest(mergeinfo)[source]

async setMergeVote(vote)[source]

async setMergeVoteComment(useriden, comment)[source]

async setTriggerInfo(iden, name, valu)[source]

async setViewInfo(name, valu)[source]: Set a mutable view property.

async snap(user)[source]

async classmethod snapctor(*args, **kwargs)

async storNodeEdits(edits, meta)[source]

async storm(text, opts=None)[source]: Evaluate a storm query and yield result messages. :Yields: ((str,dict)) – Storm messages.

async stormlist(text, opts=None)[source]

async tryToMerge(tick)[source]

async wipeAllowed(user=None)[source]: Check whether a user can wipe the write layer in the current view.

async wipeLayer(useriden=None)[source]: Delete the data in the write layer by generating del nodeedits. Triggers will be run.

class synapse.lib.view.ViewApi[source]

Bases: CellApi

async getCellIden()[source]

async getEditSize()[source]

saveNodeEdits(edits, meta)[source]

async storNodeEdits(edits, meta)[source]

async syncNodeEdits2(offs, wait=True)[source]

synapse.lib package

Subpackages

Submodules

synapse.lib.agenda module

synapse.lib.aha module

synapse.lib.ast module

synapse.lib.autodoc module

synapse.lib.base module

synapse.lib.boss module

synapse.lib.cache module

synapse.lib.cell module

synapse.lib.certdir module

synapse.lib.chop module

synapse.lib.cli module

synapse.lib.cmd module

synapse.lib.cmdr module

synapse.lib.config module

synapse.lib.const module

synapse.lib.coro module

synapse.lib.datfile module

synapse.lib.dyndeps module

synapse.lib.encoding module

synapse.lib.gis module

synapse.lib.grammar module

synapse.lib.hashitem module

synapse.lib.hashset module

synapse.lib.health module

synapse.lib.hive module

synapse.lib.hiveauth module

synapse.lib.httpapi module

synapse.lib.ingest module

synapse.lib.interval module

synapse.lib.jsonstor module

synapse.lib.jupyter module

synapse.lib.layer module

synapse.lib.link module

synapse.lib.lmdbslab module

synapse.lib.modelrev module

synapse.lib.module module

synapse.lib.modules module

synapse.lib.msgpack module

synapse.lib.multislabseqn module

synapse.lib.nexus module

synapse.lib.node module

synapse.lib.oauth module

synapse.lib.output module

synapse.lib.parser module

synapse.lib.queue module

synapse.lib.ratelimit module

synapse.lib.reflect module

synapse.lib.rstorm module

synapse.lib.schemas module

synapse.lib.scope module

synapse.lib.scrape module

synapse.lib.share module

synapse.lib.slaboffs module

synapse.lib.slabseqn module

synapse.lib.snap module

synapse.lib.spooled module

synapse.lib.storm module

synapse.lib.storm_format module

synapse.lib.stormctrl module

synapse.lib.stormhttp module

synapse.lib.stormsvc module

synapse.lib.stormtypes module

synapse.lib.stormwhois module

synapse.lib.structlog module

synapse.lib.task module

synapse.lib.thishost module

synapse.lib.thisplat module

synapse.lib.threads module

synapse.lib.time module

synapse.lib.trigger module

synapse.lib.types module

synapse.lib.urlhelp module

synapse.lib.version module

synapse.lib.view module