synapse.lib package

Subpackages

• synapse.lib.crypto package
  • Submodules
  • synapse.lib.crypto.coin module
  • synapse.lib.crypto.ecc module
  • synapse.lib.crypto.passwd module
  • synapse.lib.crypto.rsa module
  • synapse.lib.crypto.tinfoil module
• synapse.lib.platforms package
  • Submodules
  • synapse.lib.platforms.common module
  • synapse.lib.platforms.darwin module
  • synapse.lib.platforms.freebsd module
  • synapse.lib.platforms.linux module
  • synapse.lib.platforms.windows module
• synapse.lib.stormlib package
  • Submodules
  • synapse.lib.stormlib.auth module
  • synapse.lib.stormlib.backup module
  • synapse.lib.stormlib.basex module
  • synapse.lib.stormlib.cell module
  • synapse.lib.stormlib.ethereum module
  • synapse.lib.stormlib.gen module
  • synapse.lib.stormlib.graph module
  • synapse.lib.stormlib.hashes module
  • synapse.lib.stormlib.hex module
  • synapse.lib.stormlib.imap module
  • synapse.lib.stormlib.infosec module
  • synapse.lib.stormlib.ipv6 module
  • synapse.lib.stormlib.iters module
  • synapse.lib.stormlib.json module
  • synapse.lib.stormlib.log module
  • synapse.lib.stormlib.macro module
  • synapse.lib.stormlib.math module
  • synapse.lib.stormlib.mime module
  • synapse.lib.stormlib.model module
  • synapse.lib.stormlib.modelext module
  • synapse.lib.stormlib.notifications module
  • synapse.lib.stormlib.oauth module
  • synapse.lib.stormlib.project module
  • synapse.lib.stormlib.random module
  • synapse.lib.stormlib.scrape module
  • synapse.lib.stormlib.smtp module
  • synapse.lib.stormlib.stix module
  • synapse.lib.stormlib.storm module
  • synapse.lib.stormlib.version module
  • synapse.lib.stormlib.xml module
  • synapse.lib.stormlib.yaml module

Submodules

synapse.lib.agenda module

class synapse.lib.agenda.Agenda

Bases: synapse.lib.base.Base

Organize and execute all the scheduled storm queries in a cortex.

async add(cdef)

Persistently adds an appointment

Parameters

cdef (dict) – Dictionary containing the Cron definition.

Notes

The cron definition may contain the following keys:

creator (str)

Iden of the creating user.

iden (str)

Iden of the appointment.

storm (str)

The Storm query to run.

reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…])

One or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.

incunit (Union[None, TimeUnit])

The unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.

incvals (Union[None, int, Iterable[int])

Count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)

If the values for req and incvals are both lists, all combinations of all values (the product) are used.

Returns

Packed appointment definition

async delete(iden)

Delete an appointment

async disable(iden)

async enable(iden)

async get(iden)

list()

async mod(iden, query)

Change the query of an appointment

async move(croniden, viewiden)

Move a cronjob from one view to another

async start()

Enable cron jobs to start running, start the scheduler loop

Go through all the appointments, making sure the query is valid, and remove the ones that aren’t. (We can’t evaluate queries until enabled because not all the modules are loaded yet.)

async stop()

Cancel the scheduler loop, and set self.enabled to False.

class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)

Bases: object

Represents a single element of a single combination of an appointment

nexttime(lastts)

Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

pack()

Make ApptRec json/msgpack-friendly

classmethod unpack(val)

Convert from json/msgpack-friendly

class synapse.lib.agenda.TimeUnit(value)

Bases: enum.IntEnum

Unit of time that recurring and required parts of appointments are made of

DAY = 5

DAYOFMONTH = 3

DAYOFWEEK = 4

HOUR = 6

MINUTE = 7

MONTH = 2

NOW = 8

YEAR = 1

classmethod fromString(s)

synapse.lib.aha module

class synapse.lib.aha.AhaApi

Bases: synapse.lib.cell.CellApi

async addAhaSvc(name, info, network=None)

Register a service with the AHA discovery server.

NOTE: In order for the service to remain marked “up” a caller

must maintain the telepath link.

addAhaSvcProv(name, provinfo=None)

Provision the given relative service name within the configured network name.

addAhaUserEnroll(name, userinfo=None, again=False)

Create and return a one-time user enroll key.

async delAhaSvc(name, network=None)

Remove an AHA service entry.

delAhaSvcProv(iden)

Remove a previously added provisioning entry by iden.

delAhaUserEnroll(iden)

Remove a previously added enrollment entry by iden.

async genCaCert(network)

async getAhaSvc(name, filters=None)

Return an AHA service description dictionary for a service name.

async getAhaSvcMirrors(name)

Return list of AHA svcinfo dictionaries for mirrors of a service.

async getAhaSvcs(network=None)

Yield AHA svcinfo dictionaries.

Parameters

network (str) – Optionally specify a network to filter on.

async getAhaUrls()

async getCaCert(network)

async modAhaSvcInfo(name, svcinfo)

async signHostCsr(csrtext, signas=None, sans=None)

async signUserCsr(csrtext, signas=None)

class synapse.lib.aha.AhaCell

Bases: synapse.lib.cell.Cell

async addAhaSvc(name, info, network=None)

async addAhaSvcProv(name, provinfo=None)

async addAhaUserEnroll(name, userinfo=None, again=False)

cellapi

alias of synapse.lib.aha.AhaApi

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class.  Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': False, 'hidedocs': False, 'type': ['string', 'null']}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk.  Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. (may take a while)', 'type': 'boolean'}}

confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'provision:listen': {'description': 'A telepath URL for the AHA provisioning listener.', 'type': ['string', 'null']}}

async delAhaSvc(name, network=None)

async delAhaSvcProv(iden)

async delAhaUserEnroll(iden)

async genCaCert(network)

async getAhaSvc(name, filters=None)

async getAhaSvcMirrors(iden, network=None)

async getAhaSvcProv(iden)

async getAhaSvcs(network=None)

async getAhaUserEnroll(iden)

async getCaCert(network)

classmethod getEnvPrefix()

Get a list of envar prefixes for config resolution.

async initServiceNetwork()

async initServiceRuntime()

async initServiceStorage()

async modAhaSvcInfo(name, svcinfo)

async saveCaCert(name, cakey, cacert)

async saveHostCert(name, hostkey, hostcert)

async saveUserCert(name, userkey, usercert)

async setAhaSvcDown(name, linkiden, network=None)

async signHostCsr(csrtext, signas=None, sans=None)

async signUserCsr(csrtext, signas=None)

class synapse.lib.aha.AhaProvisionServiceV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.aha.EnrollApi(aha, userinfo)

Bases: object

async getCaCert()

async getUserInfo()

async signUserCsr(byts)

class synapse.lib.aha.ProvApi(aha, provinfo)

Bases: object

async getCaCert()

async getProvInfo()

async signHostCsr(byts)

async signUserCsr(byts)

class synapse.lib.aha.ProvDmon

Bases: synapse.daemon.Daemon

synapse.lib.ast module

class synapse.lib.ast.AbsProp(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.AbsPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AndCond(kids=())

Bases: synapse.lib.ast.Cond

<cond> and <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.ArgvQuery(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

validate(runt)

class synapse.lib.ast.ArrayCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AstNode(kids=())

Bases: object

Base class for all nodes in the Storm abstract syntax tree.

addKid(astn)

format(depth=0)

getRuntVars(runt)

hasAstClass(clss)

hasVarName(name)

init(core)

isRuntSafe(runt)

iterright()

Yield “rightward” siblings until None.

optimize()

prepare()

repr()

sibling(offs=1)

Return sibling node by relative offset from self.

validate(runt)

class synapse.lib.ast.Bool(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.BreakOper(kids=())

Bases: synapse.lib.ast.AstNode

async run(runt, genr)

class synapse.lib.ast.CallArgs(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.CallKwarg(kids=())

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CallKwargs(kids=())

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CaseEntry(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.CatchBlock(kids=())

Bases: synapse.lib.ast.AstNode

async catches(name, runt, path=None)

errvar()

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.CmdOper(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.Cmpr(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.Cond(kids=())

Bases: synapse.lib.ast.Value

A condition that is evaluated to filter nodes.

class synapse.lib.ast.Const(valu, kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

repr()

value()

class synapse.lib.ast.ContinueOper(kids=())

Bases: synapse.lib.ast.AstNode

async run(runt, genr)

class synapse.lib.ast.DollarExpr(kids=())

Bases: synapse.lib.ast.Value

Top level node for $(…) expressions

async compute(runt, path)

class synapse.lib.ast.Edit(kids=())

Bases: synapse.lib.ast.Oper

class synapse.lib.ast.EditEdgeAdd(kids=(), n2=False)

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditEdgeDel(kids=(), n2=False)

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditNodeAdd(kids=())

Bases: synapse.lib.ast.Edit

async addFromPath(form, runt, path)

Add a node using the context from path.

NOTE: CALLER MUST CHECK PERMS

prepare()

async run(runt, genr)

class synapse.lib.ast.EditParens(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditPropDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditPropSet(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagAdd(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagPropDel(kids=())

Bases: synapse.lib.ast.Edit

[ -#foo.bar:baz ]

async run(runt, genr)

class synapse.lib.ast.EditTagPropSet(kids=())

Bases: synapse.lib.ast.Edit

[ #foo.bar:baz=10 ]

async run(runt, genr)

class synapse.lib.ast.EditUnivDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EmbedQuery(valu, kids=())

Bases: synapse.lib.ast.Const

async compute(runt, path)

getRuntVars(runt)

hasVarName(name)

validate(runt)

class synapse.lib.ast.Emit(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.ExprAndNode(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.ExprDict(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprList(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprNode(kids=())

Bases: synapse.lib.ast.Value

A binary (i.e. two argument) expression node

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprOrNode(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FiltByArray(kids=())

Bases: synapse.lib.ast.FiltOper

+:foo*[^=visi]

class synapse.lib.ast.FiltOper(kids=())

Bases: synapse.lib.ast.Oper

async getLiftHints(runt, path)

async run(runt, genr)

class synapse.lib.ast.FiniBlock(kids=())

Bases: synapse.lib.ast.AstNode

An AST node that runs only once after all nodes have been consumed.

Example

Using a fini block:

fini {
   // stuff here runs *once* after the last node yield (even if there are no nodes)
}

Notes

A fini block must be runtsafe.

async run(runt, genr)

class synapse.lib.ast.ForLoop(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.FormName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FormPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> foo:bar

async run(runt, genr)

class synapse.lib.ast.FormTagProp(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FormatString(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.FuncArgs(kids=())

Bases: synapse.lib.ast.AstNode

Represents the function arguments in a function definition

async compute(runt, path)

class synapse.lib.ast.FuncCall(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.Function(kids=())

Bases: synapse.lib.ast.AstNode

( name, args, body )

// use args/kwargs syntax function bar(x, v=$(30)) { }

# we auto-detect the behavior of the target function

# return a value function bar(x, y) { return ($(x + y)) }

# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }

$foo = $bar(10, v=20)

async callfunc(runt, argdefs, args, kwargs)

Execute a function call using the given runtime.

This function may return a value / generator / async generator

getRuntVars(runt)

isRuntSafe(runt)

prepare()

async run(runt, genr)

validate(runt)

class synapse.lib.ast.HasAbsPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.HasRelPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

async hasProp(node, runt, name)

class synapse.lib.ast.HasTagPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.IfClause(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.IfStmt(kids=())

Bases: synapse.lib.ast.Oper

prepare()

async run(runt, genr)

class synapse.lib.ast.InitBlock(kids=())

Bases: synapse.lib.ast.AstNode

An AST node that runs only once before yielding nodes.

Example

Using a init block:

init {
    // stuff here runs *once* before the first node yield (even if there are no nodes)
}

async run(runt, genr)

class synapse.lib.ast.LiftByArray(kids=())

Bases: synapse.lib.ast.LiftOper

:prop*[range=(200, 400)]

async lift(runt, path)

class synapse.lib.ast.LiftFormTag(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftFormTagProp(kids=())

Bases: synapse.lib.ast.LiftOper

hehe:haha#foo.bar:baz [ = x ]

async lift(runt, path)

class synapse.lib.ast.LiftOper(kids=())

Bases: synapse.lib.ast.Oper

async lift(runt, path)

async run(runt, genr)

class synapse.lib.ast.LiftProp(kids=())

Bases: synapse.lib.ast.LiftOper

async getRightHints(runt, path)

async lift(runt, path)

class synapse.lib.ast.LiftPropBy(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftTag(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftTagProp(kids=())

Bases: synapse.lib.ast.LiftOper

#foo.bar:baz [ = x ]

async lift(runt, path)

class synapse.lib.ast.LiftTagTag(kids=())

Bases: synapse.lib.ast.LiftOper

##foo.bar

async lift(runt, path)

class synapse.lib.ast.List(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

repr()

class synapse.lib.ast.LookList(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.Lookup(kids, autoadd=False)

Bases: synapse.lib.ast.Query

When storm input mode is “lookup”

async run(runt, genr)

class synapse.lib.ast.N1Walk(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

async walkNodeEdges(runt, node, verb=None)

class synapse.lib.ast.N1WalkNPivo(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOut

async run(runt, genr)

class synapse.lib.ast.N2Walk(kids=())

Bases: synapse.lib.ast.N1Walk

async walkNodeEdges(runt, node, verb=None)

class synapse.lib.ast.N2WalkNPivo(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotIn

async run(runt, genr)

class synapse.lib.ast.NotCond(kids=())

Bases: synapse.lib.ast.Cond

not <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.Oper(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.OrCond(kids=())

Bases: synapse.lib.ast.Cond

<cond> or <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.PivotIn(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

<- *

async getPivsIn(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

<- foo:edge

async run(runt, genr)

class synapse.lib.ast.PivotOper(kids=(), isjoin=False)

Bases: synapse.lib.ast.Oper

repr()

class synapse.lib.ast.PivotOut(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> *

async getPivsOut(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.PivotToTags(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present

async run(runt, genr)

class synapse.lib.ast.PropName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.PropPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

:foo -> bar:foo

async run(runt, genr)

class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

:prop -> *

async run(runt, genr)

class synapse.lib.ast.PropValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

async getPropAndValu(runt, path)

isRuntSafe(valu)

prepare()

class synapse.lib.ast.Query(kids=())

Bases: synapse.lib.ast.AstNode

async iterNodePaths(runt, genr=None)

async run(runt, genr)

class synapse.lib.ast.RawPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> { <varsfrompath> }

async run(runt, genr)

class synapse.lib.ast.RelProp(kids=())

Bases: synapse.lib.ast.PropName

class synapse.lib.ast.RelPropCond(kids=())

Bases: synapse.lib.ast.Cond

(:foo:bar or .univ) <cmpr> <value>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.RelPropValue(kids=())

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Return(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.Search(kids=())

Bases: synapse.lib.ast.Query

async run(runt, genr)

class synapse.lib.ast.SetItemOper(kids=())

Bases: synapse.lib.ast.Oper

$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz

async run(runt, genr)

class synapse.lib.ast.SetVarOper(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.Stop(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.SubGraph(rules)

Bases: object

An Oper like object which generates a subgraph.

Notes

The rules format for the subgraph is shaped like the following:

rules = {

    'degrees': 1,

    'edges': True,
    'filterinput': True,
    'yieldfiltered': False,

    'filters': [
        '-(#foo or #bar)',
        '-(foo:bar or baz:faz)',
    ],

    'pivots': [
        '-> * | limit 100',
        '<- * | limit 100',
    ]

    'forms': {

        'inet:fqdn':{
            'filters': [],
            'pivots': [],
        }

        '*': {
            'filters': [],
            'pivots': [],
        },
    },
}

Nodes which were original seeds have path.meta(‘graph:seed’).

All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.

async omit(runt, node)

async pivots(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.SubQuery(kids=())

Bases: synapse.lib.ast.Oper

async compute(runt, path)

Use subquery as a value. It is error if the subquery used in this way doesn’t yield exactly one node or has a return statement.

Its value is the primary property of the node yielded, or the returned value.

async compute_array(runt, path)

Use subquery as an array.

async inline(runt, genr)

Operate subquery as if it were inlined

async run(runt, genr)

class synapse.lib.ast.SubqCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.SwitchCase(kids=())

Bases: synapse.lib.ast.Oper

prepare()

async run(runt, genr)

class synapse.lib.ast.TagCond(kids=())

Bases: synapse.lib.ast.Cond

#foo.bar

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.TagMatch(kids=())

Bases: synapse.lib.ast.TagName

Like TagName, but can have asterisks

hasglob()

class synapse.lib.ast.TagName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.TagProp(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.TagPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagPropValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.TagValuCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

class synapse.lib.ast.TryCatch(kids=())

Bases: synapse.lib.ast.AstNode

async getCatchBlock(name, runt, path=None)

async getErrValu(e)

async run(runt, genr)

class synapse.lib.ast.UnaryExprNode(kids=())

Bases: synapse.lib.ast.Value

A unary (i.e. single-argument) expression node

async compute(runt, path)

prepare()

class synapse.lib.ast.UnivProp(kids=())

Bases: synapse.lib.ast.RelProp

async compute(runt, path)

class synapse.lib.ast.UnivPropValue(kids=())

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Value(kids=())

Bases: synapse.lib.ast.AstNode

The base class for all values and value expressions.

async compute(runt, path)

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

isRuntSafe(runt)

class synapse.lib.ast.VarDeref(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.VarEvalOper(kids=())

Bases: synapse.lib.ast.Oper

Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)

async run(runt, genr)

class synapse.lib.ast.VarList(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.VarListSetOper(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.VarValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

hasVarName(name)

isRuntSafe(runt)

prepare()

validate(runt)

class synapse.lib.ast.WhileLoop(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.YieldValu(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

async yieldFromValu(runt, valu)

async synapse.lib.ast.expr_add(x, y)

async synapse.lib.ast.expr_div(x, y)

async synapse.lib.ast.expr_eq(x, y)

async synapse.lib.ast.expr_ge(x, y)

async synapse.lib.ast.expr_gt(x, y)

async synapse.lib.ast.expr_le(x, y)

async synapse.lib.ast.expr_lt(x, y)

async synapse.lib.ast.expr_mul(x, y)

async synapse.lib.ast.expr_ne(x, y)

async synapse.lib.ast.expr_neg(x)

async synapse.lib.ast.expr_not(x)

async synapse.lib.ast.expr_pow(x, y)

async synapse.lib.ast.expr_prefix(x, y)

async synapse.lib.ast.expr_re(x, y)

async synapse.lib.ast.expr_sub(x, y)

synapse.lib.ast.parseNumber(x)

async synapse.lib.ast.pullone(genr)

synapse.lib.autodoc module

class synapse.lib.autodoc.RstHelp

Bases: object

addHead(name, lvl=0, link=None)

addLines(*lines)

getRstText()

synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)

Process a list of StormTypes doc information to add them to a a RstHelp object.

Notes

This will create internal hyperlink link targets for each header item. The link prefix string must be given with the linkprefix argument.

Parameters

• page (RstHelp) – The RST page to add .

• docinfo (dict) – A Stormtypes Doc.

• linkprefix (str) – The RST link prefix string to use.

• islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.

• lvl (int) – The base header level to use when adding headers to the page.

Returns

None

synapse.lib.autodoc.genCallsig(rtype)

synapse.lib.autodoc.getArgLines(rtype)

synapse.lib.autodoc.getLink(sname, linkprefix, ref=False, suffix=None)

synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None, isstor=False)

synapse.lib.autodoc.getRtypeStr(rtype, known_types, types_prefix, suffix)

synapse.lib.autodoc.ljuster(ilines)

Helper to lstrip lines of whitespace an appropriate amount.

synapse.lib.autodoc.prepareRstLines(doc)

Prepare a desc string for RST lines.

synapse.lib.autodoc.scrubLines(lines)

Remove any empty lines until we encounter non-empty linee

synapse.lib.base module

class synapse.lib.base.Base

Bases: object

Base class for Synapse objects.

Acts as an observable, enables async init and fini.

Example

class Foo(Base):

async def __anit__(self, x, y):

await Base.__anit__(self)

await stuff(x, y)

foo = await Foo.anit(10)

Note

One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

async addSignalHandlers()

Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

async classmethod anit(*args, **kwargs)

async dist(mesg)

Distribute an existing event tuple.

Parameters

mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) )

async enter_context(item)

Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.

Returns

The result of item’s own __aenter__ or __enter__() method.

async fini()

Shut down the object and notify any onfini() coroutines.

Returns

Remaining ref count

async fire(evtname, **info)

Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’):

print(‘got: %r’ % (ret,))

incref()

Increment the reference count for this base. This API may be optionally used to control fini().

link(func)

Add a callback function to receive all events.

Example

base1 = Base() base2 = Base()

base1.link( base2.dist )

# all events on base1 are also propagated on base2

async main()

Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )

Note

This API may only be used when the ioloop is also the main thread.

off(evnt, func)

Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc )

on(evnt, func, base=None)

Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.

Parameters

• evnt (str) – An event name

• func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it:

async def baz(event):

x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y

d.on(‘foo’, baz)

# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)

Return type

None

onWith(evnt, func)

A context manager which can be used to add a callback and remove it when using a with statement.

Parameters

• evnt (str) – An event name

• func (function) – A callback function to receive event tufo

onfini(func)

Add a function/coroutine/Base to be called on fini().

async postAnit()

Method called after self.__anit__() has completed, but before anit() returns the object to the caller.

schedCallSafe(func, *args, **kwargs)

Schedule a function to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on the function completion.

Parameters

• func –

• *args –

• **kwargs –

Notes

This method may be called from outside of the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.

Returns

A Future representing the eventual function execution.

Return type

concurrent.futures.Future

schedCoro(coro)

Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.

Parameters

coro – The coroutine to schedule.

Notes

This function is not threadsafe and must be run on the Base’s event loop. Tasks created by this function do inherit the synapse.lib.scope Scope from the current task.

Returns

An asyncio.Task object.

Return type

asyncio.Task

schedCoroSafe(coro)

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on coroutine completion.

Notes

This method may be run outside the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.

Returns

A Future representing the eventual coroutine execution.

Return type

concurrent.futures.Future

schedCoroSafePend(coro)

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note

This method may not be run inside an event loop

unlink(func)

Remove a callback function previously added with link()

Example

base.unlink( callback )

waiter(count, *names)

Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events…

waiter = base.waiter(10,’foo:bar’)

# .. fire task that will cause foo:bar events

events = await waiter.wait(timeout=3)

if events == None:

# handle the timeout case…

for event in events:

# parse the events if you need…

Note

Use this with caution. It’s easy to accidentally construct race conditions with this mechanism ;)

async waitfini(timeout=None)

Wait for the base to fini()

Returns

None if timed out, True if fini happened

Example

base.waitfini(timeout=30)

class synapse.lib.base.BaseRef

Bases: synapse.lib.base.Base

An object for managing multiple Base instances by name.

async gen(name)

Atomically get/gen a Base and incref. (requires ctor during BaseRef init)

Parameters

name (str) – The name/iden of the Base instance.

get(name)

Retrieve a Base instance by name.

Parameters

name (str) – The name/iden of the Base

Returns

The Base instance (or None)

Return type

(Base)

items()

pop(name)

Remove and return a Base from the BaseRef.

Parameters

name (str) – The name/iden of the Base instance

Returns

The named base ( or None )

Return type

(Base)

put(name, base)

Add a Base (or sub-class) to the BaseRef by name.

Parameters

• name (str) – The name/iden of the Base

• base (Base) – The Base instance

Returns

(None)

vals()

class synapse.lib.base.Waiter(base, count, *names)

Bases: object

A helper to wait for a given number of events on a Base.

fini()

async wait(timeout=None)

Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30)

if evnts == None:

handleTimedOut() return

for evnt in evnts:

doStuff(evnt)

async synapse.lib.base.main(coro)

async synapse.lib.base.schedGenr(genr, maxsize=100)

Schedule a generator to run on a separate task and yield results to this task (pipelined generator).

synapse.lib.boss module

class synapse.lib.boss.Boss

Bases: synapse.lib.base.Base

An object to track “promoted” async tasks.

async execute(coro, name, user, info=None, iden=None)

Create a synapse task from the given coroutine.

get(iden)

async promote(name, user, info=None, taskiden=None)

Promote the currently running task.

Parameters

• name (str) – The name of the task.

• user – The User who owns the task.

• taskiden – An optional GUID for the task.

• info – An optional information dictionary containing information about the task.

Returns

The Synapse Task object.

Return type

s_task.Task

ps()

synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully.

class synapse.lib.cache.FixedCache(callback, size=10000)

Bases: object

async aget(key)

clear()

get(key)

pop(key)

put(key, val)

class synapse.lib.cache.LruDict(size=10000)

Bases: collections.abc.MutableMapping

Maintains the last n accessed keys

get(key, default=None)

Note: we override default impl from parent to avoid costly KeyError

items() → a set-like object providing a view on D's items

values() → an object providing a view on D's values

class synapse.lib.cache.TagGlobs

Bases: object

An object that manages multiple tag globs and values for caching.

add(name, valu, base=None)

get(name)

rem(name, valu)

synapse.lib.cache.getTagGlobRegx(name)

synapse.lib.cache.memoize(size=16384)

synapse.lib.cache.memoizemethod(size=16384)

A version of memoize that doesn’t cause GC cycles when applied to a method.

synapse.lib.cache.regexizeTagGlob(tag)

Returns

a regular expression string with ** and * interpreted as tag globs

Precondition:

tag is a valid tagmatch

Notes

A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.

The returned string does not contain a starting ‘^’ or trailing ‘$’.

synapse.lib.cell module

class synapse.lib.cell.Cell

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

A Cell() implements a synapse micro-service.

A Cell has 5 phases of startup:

1. Universal cell data structures

2. Service specific storage/data (pre-nexs)

3. Nexus subsystem initialization

4. Service specific startup (with nexus)

5. Networking and mirror services

BACKUP_SPAWN_TIMEOUT = 60.0

COMMIT = ''

FREE_SPACE_CHECK_FREQ = 60.0

VERSION = (2, 125, 0)

VERSTRING = '2.125.0'

addActiveCoro(func, iden=None, base=None)

Add a function callback to be run as a coroutine when the Cell is active.

Parameters

• func (coroutine function) – The function run as a coroutine.

• iden (str) – The iden to use for the coroutine.

• base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d

Returns

A GUID string that identifies the coroutine for delActiveCoro()

Return type

str

Note

This will re-fire the coroutine if it exits and the Cell is still active.

addHealthFunc(func)

Register a callback function to get a HealthCheck object.

addHttpApi(path, ctor, info)

async addHttpSess(iden, info)

async addHttpsPort(port, host='0.0.0.0', sslctx=None)

async addRole(name)

async addRoleRule(iden, rule, indx=None, gateiden=None)

async addUser(name, passwd=None, email=None, iden=None)

async addUserRole(useriden, roleiden)

async addUserRule(iden, rule, indx=None, gateiden=None)

async behold()

beholder()

cellapi

alias of synapse.lib.cell.CellApi

checkFreeSpace()

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class.  Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': True, 'hidedocs': True, 'type': ['string', 'null']}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk.  Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. (may take a while)', 'type': 'boolean'}}

confdefs = {}

async cullNexsLog(offs)

async delActiveCoro(iden)

Remove an Active coroutine previously added with addActiveCoro().

Parameters

iden (str) – The iden returned by addActiveCoro()

async delBackup(name)

async delHttpSess(iden)

async delRole(iden)

async delRoleRule(iden, rule, gateiden=None)

async delUser(iden)

async delUserRole(useriden, roleiden)

async delUserRule(iden, rule, gateiden=None)

async dyncall(iden, todo, gatekeys=())

async dyniter(iden, todo, gatekeys=())

async classmethod execmain(argv, outp=None)

The main entry point for running the Cell as an application.

Parameters

• argv (list) – A list of command line arguments to launch the Cell with.

• outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.

Returns

None.

async feedBeholder(name, info, gates=None, perms=None)

Feed a named event onto the cell:beholder message bus that will sent to any listeners.

Parameters

• info (dict) – An information dictionary to be sent to any consumers.

• gates (list) – List of gate idens, whose details will be added to the outbound message(s).

• perms (list) – List of permission names, whose details will be added to the outbound message(s).

Returns

None

async fini()

Fini override that ensures locking teardown order.

async genHttpSess(iden)

async genUserOnepass(iden, duration=600000)

classmethod getArgParser(conf=None)

Get an argparse.ArgumentParser for the Cell.

Parameters

conf (s_config.Config) – Optional, a Config object which

Notes

Boot time configuration data is placed in the argument group called config. This adds default dirn, --telepath, --https and --name arguements to the argparser instance. Configuration values which have the hideconf or hidecmdl value set to True are not added to the argparser instance.

Returns

A ArgumentParser for the Cell.

Return type

argparse.ArgumentParser

async getAuthGate(iden)

async getAuthGates()

async getAuthRoles()

async getAuthUsers(archived=False)

async getBackupInfo()

Gets information about recent backup activity

async getBackups()

async getCellApi(link, user, path)

Get an instance of the telepath Client object for a given user, link and path.

Parameters

• link (s_link.Link) – The link object.

• user (s_hive.HiveUser) – The heavy user object.

• path (str) – The path requested.

Notes

This defaults to the self.cellapi class. Implementors may override the default class attribute for cellapi to share a different interface.

Returns

The shared object for this cell.

Return type

object

getCellIden()

async getCellInfo()

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns

A Dictionary of metadata.

Return type

Dict

getCellNexsRoot()

async getCellRunId()

classmethod getCellType()

async getConfOpt(name)

async getDmonSessions()

classmethod getEnvPrefix()

Get a list of envar prefixes for config resolution.

async getHealthCheck()

async getHiveKey(path)

Get the value of a key in the cell default hive

async getHiveKeys(path)

Return a list of (name, value) tuples for nodes under the path.

async getHttpSessDict(iden)

getLocalProxy(share='*', user='root')

getLocalUrl(share='*', user='root')

async getLogExtra(**kwargs)

Get an extra dictionary for structured logging which can be used as a extra argument for loggers.

Parameters

**kwargs – Additional key/value items to add to the log.

Returns

A dictionary

Return type

Dict

async getMirrorUrls()

async getNexsIndx()

async getNexusChanges(offs, tellready=False)

async getPermDef(perm)

async getPermDefs()

async getRoleDef(iden)

async getRoleDefByName(name)

async getRoleDefs()

async getSystemInfo()

Get info about the system in which the cell is running

Returns

• volsize - Volume where cell is running total space

• volfree - Volume where cell is running free space

• backupvolsize - Backup directory volume total space

• backupvolfree - Backup directory volume free space

• cellstarttime - Cell start time in epoch milliseconds

• celluptime - Cell uptime in milliseconds

• cellrealdisk - Cell’s use of disk, equivalent to du

• cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

• osversion - OS version/architecture

• pyversion - Python version

• totalmem - Total memory in the system

• availmem - Available memory in the system

• cpucount - Number of CPUs on system

Return type

A dictionary with the following keys. All size values are in bytes

async getTeleApi(link, mesg, path)

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

getTempDir()

async getUserDef(iden, packroles=True)

async getUserDefByName(name)

async getUserDefs()

getUserName(iden, defv='<unknown>')

Translate the user iden to a user name.

async getUserProfInfo(iden, name)

async getUserProfile(iden)

async handoff(turl, timeout=30)

Hand off leadership to a mirror in a transactional fashion.

classmethod initCellConf(conf=None)

Create a Config object for the Cell.

Parameters

conf (s_config.Config) – An optional config structure. This has _opts_data taken from it.

Notes

The Config object has a envar_prefix set according to the results of cls.getEnvPrefix().

Returns

A Config helper object.

Return type

s_config.Config

async classmethod initFromArgv(argv, outp=None)

Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.

Parameters

• argv (list) – A list of command line arguments to launch the Cell with.

• outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This does the following items:

• Create a Config object from the Cell class.

• Creates an Argument Parser from the Cell class and Config object.

• Parses the provided arguments.

• Loads configuration data from the parsed options and environment variables.

• Sets logging for the process.

• Creates the Cell from the Cell Ctor.

• Adds a Telepath listener, HTTPs port listeners and Telepath share names.

• Returns the Cell.

Returns

This returns an instance of the Cell.

Return type

Cell

async initNexusSubsystem()

async initServiceActive()

async initServiceNetwork()

async initServicePassive()

async initServiceRuntime()

async initServiceStorage()

initSslCtx(certpath, keypath)

async isCellActive()

async isRoleAllowed(iden, perm, gateiden=None)

async isUserAllowed(iden, perm, gateiden=None)

async iterBackupArchive(name, user)

async iterNewBackupArchive(user, name=None, remove=False)

async kill(user, iden)

async listHiveKey(path=None)

async loadHiveTree(tree, path=(), trim=False)

Note: this is for expert emergency use only.

modCellConf(conf)

Modify the Cell’s ondisk configuration overrides file and runtime configuration.

Parameters

conf (dict) – A dictionary of items to set.

Notes

This does require the data being set to be schema valid.

Returns

None.

popCellConf(name)

Remove a key from the Cell’s ondisk configuration overrides file and runtime configuration.

Parameters

name (str) – Name of the value to remove.

Notes

This does not modify the cell.yaml file. This does re-validate the configuration after removing the value, so if the value removed had a default populated by schema, that default would be reset.

Returns

None

async popHiveKey(path)

Remove and return the value of a key in the cell default hive.

Note: this is for expert emergency use only.

async popUserProfInfo(iden, name, default=None)

async promote(graceful=False)

Transform this cell from a passive follower to an active cell that writes changes locally.

async ps(user)

async readyToMirror()

async reqGateKeys(gatekeys)

async rotateNexsLog()

async runBackup(name=None, wait=True)

async saveHiveTree(path=())

async setCellActive(active)

async setHiveKey(path, valu)

Set or change the value of a key in the cell default hive

async setHttpSessInfo(iden, name, valu)

async setNexsIndx(indx)

async setRoleName(iden, name)

async setRoleRules(iden, rules, gateiden=None)

async setUserAdmin(iden, admin, gateiden=None)

async setUserArchived(iden, archived)

async setUserEmail(useriden, email)

async setUserLocked(iden, locked)

async setUserName(useriden, name)

async setUserPasswd(iden, passwd)

async setUserProfInfo(iden, name, valu)

async setUserRoles(useriden, roleidens)

async setUserRules(iden, rules, gateiden=None)

async sync()

no-op mutable for testing purposes. If I am follower, when this returns, I have received and applied all the writes that occurred on the leader before this call.

async trimNexsLog(consumers=None, timeout=30)

async tryUserPasswd(name, passwd)

async waitNexsOffs(offs, timeout=None)

class synapse.lib.cell.CellApi

Bases: synapse.lib.base.Base

addAuthRole(name)

addAuthRule(name, rule, indx=None, gateiden=None)

This API is deprecated.

addRole(name)

addRoleRule(iden, rule, indx=None, gateiden=None)

addUser(name, passwd=None, email=None, iden=None)

addUserRole(useriden, roleiden)

addUserRule(iden, rule, indx=None, gateiden=None)

async allowed(perm, default=None)

Check if the user has the requested permission.

Parameters

• perm – permission path components to check

• default – Value returned if no value stored

Examples

Form a path and check the permission from a remote proxy:

perm = ('node', 'add', 'inet:ipv4')
allowed = await prox.allowed(perm)
if allowed:
    dostuff()

Returns

True if the user has permission, False if explicitly denied, None if no entry

Return type

Optional[bool]

behold()

Yield Cell system messages

cullNexsLog(offs)

Remove Nexus log entries up to (and including) the given offset.

Note

If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.

Only rotated logs where the last index is less than the provided offset will be removed from disk.

Parameters

offs (int) – The offset to remove entries up to.

Returns

Whether the cull was executed

Return type

bool

delAuthRole(name)

delAuthRule(name, rule, gateiden=None)

This API is deprecated.

delAuthUser(name)

delBackup(name)

Delete a backup by name.

Parameters

name (str) – The name of the backup to delete.

delRole(iden)

delRoleRule(iden, rule, gateiden=None)

delUser(iden)

delUserRole(useriden, roleiden)

delUserRule(iden, rule, gateiden=None)

dyncall(iden, todo, gatekeys=())

dyniter(iden, todo, gatekeys=())

genUserOnepass(iden, duration=60000)

getAuthGate(iden)

getAuthGates()

getAuthInfo(name)

This API is deprecated.

getAuthRoles()

getAuthUsers(archived=False)

Parameters

archived (bool) – If true, list all users, else list non-archived users

getBackupInfo()

Get information about recent backup activity.

Returns

• currduration - If backup currently running, time in ms since backup started, otherwise None

• laststart - Last time (in epoch milliseconds) a backup started

• lastend - Last time (in epoch milliseconds) a backup ended

• lastduration - How long last backup took in ms

• lastsize - Disk usage of last backup completed

• lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive

• lastexception - Tuple of exception information if last backup failed, otherwise None

Return type

(dict) It has the following keys

Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.

getBackups()

Retrieve a list of backups.

Returns

A list of backup names.

Return type

list[str]

getCellIden()

async getCellInfo()

async getCellRunId()

getCellType()

getCellUser()

getDiagInfo()

getDmonSessions()

getGcInfo()

For diagnostic purposes only!

NOTE: This API is not supported and can be removed at any time!

async getHealthCheck()

getHiveKey(path)

getHiveKeys(path)

getMirrorUrls()

getNexsIndx()

getNexusChanges(offs, tellready=False)

async getPermDef(perm)

Return a specific permission definition.

async getPermDefs()

Return a non-comprehensive list of perm definitions.

getRoleDef(iden)

getRoleDefByName(name)

getRoleDefs()

async getRoleInfo(name)

getSystemInfo()

Get info about the system in which the cell is running

Returns

• volsize - Volume where cell is running total space

• volfree - Volume where cell is running free space

• backupvolsize - Backup directory volume total space

• backupvolfree - Backup directory volume free space

• celluptime - Cell uptime in milliseconds

• cellrealdisk - Cell’s use of disk, equivalent to du

• cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

• osversion - OS version/architecture

• pyversion - Python version

• totalmem - Total memory in the system

• availmem - Available memory in the system

Return type

A dictionary with the following keys. All size values are in bytes

getUserDef(iden, packroles=True)

getUserDefByName(name)

getUserDefs()

async getUserInfo(name)

getUserProfInfo(iden, name)

getUserProfile(iden)

handoff(turl, timeout=30)

async initCellApi()

async isCellActive()

Returns True if the cell is an active/leader cell.

isRoleAllowed(iden, perm, gateiden=None)

isUserAllowed(iden, perm, gateiden=None)

issue(nexsiden: str, event: str, args, kwargs, meta=None)

iterBackupArchive(name)

Retrieve a backup by name as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters

name (str) – The name of the backup to retrieve.

iterNewBackupArchive(name=None, remove=False)

Run a new backup and return it as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters

• name (str) – The name of the backup to retrieve.

• remove (bool) – Delete the backup after streaming.

async kill(iden)

listHiveKey(path=None)

popHiveKey(path)

popUserProfInfo(iden, name, default=None)

promote(graceful=False)

async ps()

readyToMirror()

rotateNexsLog()

Rotate the Nexus log at the current offset.

Returns

The starting index of the active Nexus log

Return type

int

runBackup(name=None, wait=True)

Run a new backup.

Parameters

• name (str) – The optional name of the backup.

• wait (bool) – On True, wait for backup to complete before returning.

Returns

The name of the newly created backup.

Return type

str

runGcCollect(generation=2)

For diagnostic purposes only!

NOTE: This API is not supported and can be removed at any time!

saveHiveTree(path=())

setAuthAdmin(name, isadmin)

This API is deprecated.

setCellUser(iden)

Switch to another user (admin only).

This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.

setHiveKey(path, valu)

setRoleRules(iden, rules, gateiden=None)

setUserAdmin(iden, admin, gateiden=None)

setUserArchived(useriden, archived)

setUserEmail(useriden, email)

setUserLocked(useriden, locked)

async setUserPasswd(iden, passwd)

setUserProfInfo(iden, name, valu)

setUserRoles(useriden, roleidens)

setUserRules(iden, rules, gateiden=None)

trimNexsLog(consumers=None, timeout=60)

Rotate and cull the Nexus log (and those of any consumers) at the current offset.

Note

If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.

Parameters

• consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.

• timeout (int) – Time in seconds to wait for downstream consumers to be caught up.

Returns

The offset that the Nexus log was culled up to and including.

Return type

int

tryUserPasswd(name, passwd)

waitNexsOffs(offs, timeout=None)

Wait for the Nexus log to write an offset.

Parameters

• offs (int) – The offset to wait for.

• timeout (int or None) – An optional timeout in seconds.

Returns

True if the offset was written, False if it timed out.

Return type

bool

synapse.lib.cell.SLAB_MAP_SIZE = 134217728

Base classes for the synapse “cell” microservice architecture.

synapse.lib.cell.adminapi(log=False)

Decorator for CellApi (and subclasses) for requiring a method to be called only by an admin user.

Parameters

log (bool) – If set to True, log the user, function and arguments.

synapse.lib.certdir module

class synapse.lib.certdir.CRL(certdir, name)

Bases: object

revoke(cert)

Revoke a certificate with the CRL.

Parameters

cert (cryto.X509) – The certificate to revoke.

Returns

None

class synapse.lib.certdir.CertDir(path=None)

Bases: object

Certificate loading/generation/signing utilities.

Features:

• Locates and load certificates, keys, and certificate signing requests (CSRs).

• Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.

• Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).

• Signs certificate signing requests (CSRs).

• Generates PKCS#12 archives for use in browser.

Parameters

path (str) – Optional path which can override the default path directory.

Notes

• All certificates will be loaded from and written to ~/.syn/certs by default. Set the environment variable SYN_CERT_DIR to override.

• All certificate generation methods create 4096 bit RSA keypairs.

• All certificate signing methods use sha256 as the signature algorithm.

• CertDir does not currently support signing CA CSRs.

addCertPath(*path)

delCertPath(*path)

genCaCert(name, signas=None, outp=None, save=True)

Generates a CA keypair.

Parameters

• name (str) – The name of the CA keypair.

• signas (str) – The CA keypair to sign the new CA with.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Make a CA named “myca”:

mycakey, mycacert = cdir.genCaCert(‘myca’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genCaCrl(name)

Get the CRL for a given CA.

Parameters

name (str) – The CA name.

Returns

The CRL object.

Return type

CRL

genClientCert(name, outp=None)

Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material.

Parameters

• name (str) – The name of the user keypair.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”:

myuserpkcs12 = cdir.genClientCert(‘myuser’)

Returns

The PKCS #12 archive.

Return type

OpenSSL.crypto.PKCS12

genCodeCert(name, signas=None, outp=None, save=True)

Generates a code signing keypair.

Parameters

• name (str) – The name of the code signing cert.

• signas (str) – The CA keypair to sign the new code keypair with.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a code signing cert for the name “The Vertex Project”:

myuserkey, myusercert = cdir.genCodeCert(‘The Vertex Project’)

Returns

Tuple containing the key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genCrlPath(name)

genHostCert(name, signas=None, outp=None, csr=None, sans=None, save=True)

Generates a host keypair.

Parameters

• name (str) – The name of the host keypair.

• signas (str) – The CA keypair to sign the new host keypair with.

• outp (synapse.lib.output.Output) – The output buffer.

• csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

• sans (list) – List of subject alternative names.

Examples

Make a host keypair named “myhost”:

myhostkey, myhostcert = cdir.genHostCert(‘myhost’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genHostCsr(name, outp=None)

Generates a host certificate signing request.

Parameters

• name (str) – The name of the host CSR.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the host key named “myhost”:

cdir.genHostCsr(‘myhost’)

Returns

The bytes of the CSR.

Return type

bytes

genUserCert(name, signas=None, outp=None, csr=None, save=True)

Generates a user keypair.

Parameters

• name (str) – The name of the user keypair.

• signas (str) – The CA keypair to sign the new user keypair with.

• outp (synapse.lib.output.Output) – The output buffer.

• csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”:

myuserkey, myusercert = cdir.genUserCert(‘myuser’)

Returns

Tuple containing the key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genUserCsr(name, outp=None)

Generates a user certificate signing request.

Parameters

• name (str) – The name of the user CSR.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the user “myuser”:

cdir.genUserCsr(‘myuser’)

Returns

The bytes of the CSR.

Return type

bytes

getCaCert(name)

Loads the X509 object for a given CA.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the certificate for the CA “myca”

mycacert = cdir.getCaCert(‘myca’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getCaCertBytes(name)

getCaCertPath(name)

Gets the path to a CA certificate.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the CA certificate for the CA “myca”:

mypath = cdir.getCACertPath(‘myca’)

Returns

The path if exists.

Return type

str

getCaCerts()

Return a list of CA certs from the CertDir.

Returns

List of CA certificates.

Return type

[OpenSSL.crypto.X509]

getCaKey(name)

Loads the PKey object for a given CA keypair.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the private key for the CA “myca”:

mycakey = cdir.getCaKey(‘myca’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getCaKeyPath(name)

Gets the path to a CA key.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”:

mypath = cdir.getCAKeyPath(‘myca’)

Returns

The path if exists.

Return type

str

getClientCert(name)

Loads the PKCS12 archive object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”:

mypkcs12 = cdir.getClientCert(‘myuser’)

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool

Returns

The PKCS12 archive, if exists.

Return type

OpenSSL.crypto.PKCS12

getClientCertPath(name)

Gets the path to a client certificate.

Parameters

name (str) – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”:

mypath = cdir.getClientCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getClientSSLContext(certname=None)

Returns an ssl.SSLContext appropriate for initiating a TLS session

Parameters

certname – If specified, use the user certificate with the matching name to authenticate to the remote service.

Returns

A SSLContext object.

Return type

ssl.SSLContext

getCodeCert(name)

getCodeCertPath(name)

getCodeKey(name)

getCodeKeyPath(name)

getCrlPath(name)

getHostCaPath(name)

Gets the path to the CA certificate that issued a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”:

mypath = cdir.getHostCaPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostCert(name)

Loads the X509 object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”:

myhostcert = cdir.getHostCert(‘myhost’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getHostCertHash(name)

getHostCertPath(name)

Gets the path to a host certificate.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”:

mypath = cdir.getHostCertPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostKey(name)

Loads the PKey object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the private key object for the host “myhost”:

myhostkey = cdir.getHostKey(‘myhost’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getHostKeyPath(name)

Gets the path to a host key.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”:

mypath = cdir.getHostKeyPath(‘myhost’)

Returns

The path if exists.

Return type

str

getServerSSLContext(hostname=None, caname=None)

Returns an ssl.SSLContext appropriate to listen on a socket

Parameters

• hostname – If None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory.

• caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.

Returns

A SSLContext object.

Return type

ssl.SSLContext

getUserCaPath(name)

Gets the path to the CA certificate that issued a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the CA cert which issue the cert for “myuser”:

mypath = cdir.getUserCaPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserCert(name)

Loads the X509 object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”:

myusercert = cdir.getUserCert(‘myuser’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getUserCertPath(name)

Gets the path to a user certificate.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”:

mypath = cdir.getUserCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserForHost(user, host)

Gets the name of the first existing user cert for a given user and host.

Parameters

• user (str) – The name of the user.

• host (str) – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”:

usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)

Returns

The cert name, if exists.

Return type

str

getUserKey(name)

Loads the PKey object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”:

myuserkey = cdir.getUserKey(‘myuser’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getUserKeyPath(name)

Gets the path to a user key.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the user key for “myuser”:

mypath = cdir.getUserKeyPath(‘myuser’)

Returns

The path if exists.

Return type

str

importFile(path, mode, outp=None)

Imports certs and keys into the Synapse cert directory

Parameters

• path (str) – The path of the file to be imported.

• mode (str) – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory.

certdir.importFile(‘mycoolca.crt’, ‘cas’)

Notes

importFile does not perform any validation on the files it imports.

Returns

None

isCaCert(name)

Checks if a CA certificate exists.

Parameters

name (str) – The name of the CA keypair.

Examples

Check if the CA certificate for “myca” exists:

exists = cdir.isCaCert(‘myca’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isClientCert(name)

Checks if a user client certificate (PKCS12) exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists:

exists = cdir.isClientCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isHostCert(name)

Checks if a host certificate exists.

Parameters

name (str) – The name of the host keypair.

Examples

Check if the host cert “myhost” exists:

exists = cdir.isUserCert(‘myhost’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isUserCert(name)

Checks if a user certificate exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the user cert “myuser” exists:

exists = cdir.isUserCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

loadCertByts(byts)

Load a X509 certificate from its PEM encoded bytes.

Parameters

byts (bytes) – The PEM encoded bytes of the certificate.

Returns

The X509 certificate.

Return type

OpenSSL.crypto.X509

Raises

BadCertBytes – If the certificate bytes are invalid.

saveCaCertByts(byts)

saveCertPem(cert, path)

Save a certificate in PEM format to a file outside the certdir.

saveHostCertByts(byts)

savePkeyPem(pkey, path)

Save a private key in PEM format to a file outside the certdir.

saveUserCertByts(byts)

selfSignCert(cert, pkey)

Self-sign a certificate.

Parameters

• cert (OpenSSL.crypto.X509) – The certificate to sign.

• pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key:

cdir.selfSignCert(mycert, myotherprivatekey)

Returns

None

signCertAs(cert, signas)

Signs a certificate with a CA keypair.

Parameters

• cert (OpenSSL.crypto.X509) – The certificate to sign.

• signas (str) – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”:

cdir.signCertAs(mycert, ‘myca’)

Returns

None

signHostCsr(xcsr, signas, outp=None, sans=None, save=True)

Signs a host CSR with a CA keypair.

Parameters

• xcsr (OpenSSL.crypto.X509Req) – The certificate signing request.

• signas (str) – The CA keypair name to sign the CSR with.

• outp (synapse.lib.output.Output) – The output buffer.

• sans (list) – List of subject alternative names.

Examples

Sign a host key with the CA “myca”:

cdir.signHostCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

signUserCsr(xcsr, signas, outp=None, save=True)

Signs a user CSR with a CA keypair.

Parameters

• xcsr (OpenSSL.crypto.X509Req) – The certificate signing request.

• signas (str) – The CA keypair name to sign the CSR with.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

cdir.signUserCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

valCodeCert(byts)

Verify a code cert is valid according to certdir’s available CAs and CRLs.

Parameters

byts (bytes) – The certificate bytes.

Returns

The certificate.

Return type

OpenSSL.crypto.X509

valUserCert(byts, cacerts=None)

Validate the PEM encoded x509 user certificate bytes and return it.

Parameters

• byts (bytes) – The bytes for the User Certificate.

• cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates.

Raises

BadCertVerify – If the certificate is not valid.

Returns

The certificate, if it is valid.

Return type

OpenSSL.crypto.X509

synapse.lib.certdir.addCertPath(path)

synapse.lib.certdir.delCertPath(path)

synapse.lib.certdir.getCertDir() → synapse.lib.certdir.CertDir

Get the singleton CertDir instance.

Returns

A certdir object.

Return type

CertDir

synapse.lib.certdir.getCertDirn() → str

Get the expanded default path used by the singleton CertDir instance.

Returns

The path string.

Return type

str

synapse.lib.certdir.getServerSSLContext() → ssl.SSLContext

Get a server SSLContext object.

This object has a minimum TLS version of 1.2, a subset of ciphers in use, and disabled client renegotiation.

This object has no certificates loaded in it.

Returns

The context object.

Return type

ssl.SSLContext

synapse.lib.certdir.iterFqdnUp(fqdn)

synapse.lib.chop module

synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)

Shared primitive routines for chopping up strings and values.

synapse.lib.chop.digits(text)

synapse.lib.chop.hexstr(text)

Ensure a string is valid hex.

Parameters

text (str) – String to normalize.

Examples

Norm a few strings:

hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.

Returns

Normalized hex string.

Return type

str

synapse.lib.chop.intstr(text)

synapse.lib.chop.onespace(text)

synapse.lib.chop.printables(text)

synapse.lib.chop.replaceUnicodeDashes(valu)

Replace unicode dashes in a string with regular dashes.

Parameters

valu (str) – A string.

Returns

A new string with replaced dashes.

Return type

str

synapse.lib.chop.stormstring(s)

Make a string storm safe by escaping backslashes and double quotes.

Parameters

s (str) – String to make storm safe.

Notes

This does not encapsulate a string in double quotes.

Returns

A string which can be embedded directly into a storm query.

Return type

str

synapse.lib.chop.tag(text)

synapse.lib.chop.tagpath(text)

synapse.lib.chop.tags(norm)

Divide a normalized tag string into hierarchical layers.

synapse.lib.chop.validateTagMatch(tag)

Raises an exception if tag is not a valid tagmatch (i.e. a tag that might have globs)

synapse.lib.cli module

class synapse.lib.cli.Cli

Bases: synapse.lib.base.Base

A modular / event-driven CLI base object.

addCmdClass(ctor, **opts)

Add a Cmd subclass to this cli.

async addSignalHandlers()

Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.

get(name, defval=None)

getCmdByName(name)

Return a Cmd instance by name.

getCmdNames()

Return a list of all the known command names for the CLI.

getCmdPrompt()

Get the command prompt.

Returns

Configured command prompt

Return type

str

histfile = 'cmdr_history'

initCmdClasses()

printf(mesg, addnl=True, color=None)

async prompt(text=None)

Prompt for user input from stdin.

async runCmdLine(line)

Run a single command line.

Parameters

line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch:

await cli.runCmdLine(‘woot –help’)

Returns

Arbitrary data from the cmd class.

Return type

object

async runCmdLoop()

Run commands from a user in an interactive fashion until fini() or EOFError is raised.

set(name, valu)

class synapse.lib.cli.Cmd(cli, **opts)

Bases: object

Base class for modular commands in the synapse CLI.

getCmdBrief()

Return the single-line description for this command.

getCmdDoc()

Return the help/doc output for this command.

getCmdItem()

Get a reference to the object we are commanding.

getCmdName()

getCmdOpts(text)

Use the _cmd_syntax def to split/parse/normalize the cmd line.

Parameters

text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type

Returns

An opts dictionary.

Return type

dict

printf(mesg, addnl=True, color=None)

async runCmdLine(line)

Run a line of command input for this command.

Parameters

line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdHelp(cli, **opts)

Bases: synapse.lib.cli.Cmd

List commands and display help output.

Example

help foocmd

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdLocals(cli, **opts)

Bases: synapse.lib.cli.Cmd

List the current locals for a given CLI object.

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdQuit(cli, **opts)

Bases: synapse.lib.cli.Cmd

Quit the current command line interpreter.

Example

quit

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

synapse.lib.cmd module

class synapse.lib.cmd.Parser(prog=None, outp=<synapse.lib.output.OutPut object>, **kwargs)

Bases: argparse.ArgumentParser

exit(status=0, message=None)

Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead.

synapse.lib.cmdr module

async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)

Construct and return a cmdr for the given remote cell.

Parameters

• cell – Cell proxy being commanded.

• outp – Output helper object.

• color (bool) – If true, enable colorized output.

• **opts – Additional options pushed into the Cmdr locs.

Examples

Get the cmdr for a proxy:

cmdr = await getItemCmdr(foo)

Returns

A Cli instance with Cmds loaeded into it.

Return type

s_cli.Cli

async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)

Create a cmdr for the given item and run the cmd loop.

Parameters

• item – Cell proxy being commanded.

• outp – Output helper object.

• color (bool) – If true, enable colorized output.

• **opts – Additional options pushed into the Cmdr locs.

Notes

This function does not return while the command loop is run.

Examples

Run the Cmdr for a proxy:

await runItemCmdr(foo)

Returns

This function returns None.

Return type

None

synapse.lib.config module

class synapse.lib.config.Config(schema, conf=None, envar_prefixes=None)

Bases: collections.abc.MutableMapping

Synapse configuration helper based on JSON Schema.

Parameters

• schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.

• conf (dict) – Optional, a set of configuration data to preload.

• envar_prefixes (list) – Optional, a list of prefix strings used when collecting configuration data from environment variables.

Notes

This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.

The default values provided in the schema must be able to be recreated from the repr() of their Python value.

Default values are not loaded into the configuration data until the reqConfValid() method is called.

asDict()

Get a copy of configuration data.

Returns

A copy of the configuration data.

Return type

dict

getArgParseArgs()

getCmdlineMapping()

classmethod getConfFromCell(cell, conf=None, envar_prefixes=None)

Get a Config object from a Cell directly (either the ctor or the instance thereof).

Returns

A Config object.

Return type

Config

getEnvarMapping(prefix=None)

Get a mapping of config values to envars.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

reqConfValid()

Validate that the loaded configuration data is valid according to the schema.

Notes

The validation set does set any default values which are not currently set for configuration options.

Returns

This returns nothing.

Return type

None

reqConfValu(key)

Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.

Parameters

key (str) – The key to require.

Returns

The requested value.

reqKeyValid(key, value)

Test if a key is valid for the provided schema it is associated with.

Parameters

• key (str) – Key to check.

• value – Value to check.

Raises

• BadArg – If the key has no associated schema.

• BadConfValu – If the data is not schema valid.

Returns

None when valid.

setConfFromEnvs()

Set configuration options from environment variables.

Notes

Environment variables are resolved from configuration options after doing the following transform:

• Replace : characters with _.

• Add a config provided prefix, if set.

• Uppercase the string.

• Resolve the environment variable

• If the environment variable is set, set the config value to the results of yaml.yaml_safeload() on the value.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

Examples

For the configuration value auth:passwd, the environment variable is resolved as AUTH_PASSWD. With the prefix cortex, the the environment variable is resolved as CORTEX_AUTH_PASSWD.

Returns

Returns a dictionary of values which were set from enviroment variables.

Return type

dict

setConfFromFile(path, force=False)

Set the opts for a conf object from YAML file path.

Parameters

• path (str) – Path to the yaml load. If it exists, it must represent a dictionary.

• force (bool) – Force the update instead of using setdefault() behavior.

Returns

None

setConfFromOpts(opts=None)

Set the opts for a conf object from a namespace object.

Parameters

• opts (argparse.Namespace) – A Namespace object made from parsing args with an ArgumentParser

• getArgumentParser. (made with) –

Returns

Returns None.

Return type

None

synapse.lib.config.getJsSchema(confbase, confdefs)

Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.

Parameters

• confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.

• confdefs (dict) – A JSON Schema dictionary of properties for the object.

Notes

This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify

Returns

A complete JSON schema.

Return type

dict

synapse.lib.config.getJsValidator(schema, use_default=True)

Get a fastjsonschema callable.

Parameters

• schema (dict) – A JSON Schema object.

• use_default (bool) – Whether to insert “default” key arguments into the validated data structure.

Returns

A callable function that can be used to validate data against the json schema.

Return type

callable

synapse.lib.config.make_envar_name(key, prefix=None)

Convert a colon delimited string into an uppercase, underscore delimited string.

Parameters

• key (str) – Config key to convert.

• prefix (str) – Optional string prefix to prepend the the config key.

Returns

The string to lookup against a envar.

Return type

str

synapse.lib.const module

synapse.lib.coro module

Async/Coroutine related utilities.

class synapse.lib.coro.Event(*, loop=<object object>)

Bases: asyncio.locks.Event

async timewait(timeout=None)

class synapse.lib.coro.GenrHelp(genr)

Bases: object

async list()

async spin()

async synapse.lib.coro.agen(item)

Wrap an async_generator or generator in an async_generator.

Notes

Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.

async synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None)

Wait on an an asyncio event with an optional timeout

Returns

true if the event got set, False if timed out

synapse.lib.coro.executor(func, *args, **kwargs)

Execute a non-coroutine function in the ioloop executor pool.

Parameters

• func – Function to execute.

• *args – Args for the function.

• **kwargs – Kwargs for the function.

Examples

Execute a blocking API call in the executor pool:

import requests

def block(url, params=None):
    return requests.get(url, params=params).json()

fut = s_coro.executor(block, 'http://some.tld/thign')
resp = await fut

Returns

An asyncio future.

Return type

asyncio.Future

async synapse.lib.coro.forked(func, *args, **kwargs)

Execute a target function in the forked process pool.

Parameters

• func – The target function.

• *args – Function positional arguments.

• **kwargs – Function keyword arguments.

Returns

The target function return.

Raises

• The function may raise from the target function, or raise a s_exc.FatalErr in the event of a broken forked –

• process pool. The fatalerr represents a unrecoverable application state. –

synapse.lib.coro.genrhelp(f)

synapse.lib.coro.iscoro(item)

async synapse.lib.coro.ornot(func, *args, **kwargs)

Calls func and awaits it if a returns a coroutine.

Note

This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.

This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage:

ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)

synapse.lib.coro.set_pool_logging(logger_, logconf)

async synapse.lib.coro.spawn(todo, timeout=None, ctx=None, log_conf=None)

Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.

Parameters

• todo (tuple) – A tuple of function, *args, and **kwargs.

• timeout (int) – The timeout to wait for the todo function to finish.

• ctx (multiprocess.Context) – A optional multiprocessing context object.

• log_conf (dict) – An optional logging configuration for the spawned process.

Notes

The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.

Returns

The return value of executing the todo function.

async synapse.lib.coro.waittask(task, timeout=None)

Await a task without cancelling it when you time out.

Returns

True if the task completed before the timeout.

Return type

boolean

synapse.lib.datfile module

Utilities for handling data files embedded within python packages.

synapse.lib.datfile.openDatFile(datpath)

Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.dyndeps module

synapse.lib.dyndeps.getDynLocal(name)

Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()

synapse.lib.dyndeps.getDynMeth(name)

Retrieve and return an unbound method by python path.

synapse.lib.dyndeps.getDynMod(name)

Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’)

synapse.lib.dyndeps.runDynTask(task)

Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)

Dynamically import a module and call a function or raise an exception.

synapse.lib.dyndeps.tryDynLocal(name)

Dynamically import a module and return a module local or raise an exception.

synapse.lib.dyndeps.tryDynMod(name)

Dynamically import a python module or exception.

synapse.lib.encoding module

synapse.lib.encoding.addFormat(name, fn, opts)

Add an additional ingest file format

synapse.lib.encoding.decode(name, byts, **opts)

Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with +

to encode for that name/layer.

synapse.lib.encoding.encode(name, item, **opts)

synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)

Iterate through the data provided by a file like object.

Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd:
    for row in iterdata(fd, format='csv', encoding='utf8'):
        dostuff(row)

Parameters

• fd (file) – File like object to iterate over.

• close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.

• **opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.

Yields

An item to process. The type of the item is dependent on the format parameters.

synapse.lib.gis module

synapse.lib.gis.bbox(lat, lon, dist)

Calculate a min/max bounding box for the circle defined by lalo/dist.

Parameters

• lat (float) – The latitude in degrees

• lon (float) – The longitude in degrees

• dist (int) – A distance in geo:dist base units (mm)

Returns

(latmin, latmax, lonmin, lonmax)

Return type

(float,float,float,float)

synapse.lib.gis.dms2dec(degs, mins, secs)

Convert degrees, minutes, seconds lat/long form to degrees float.

Parameters

• degs (int) – Degrees

• mins (int) – Minutes

• secs (int) – Seconds

Returns

Degrees

Return type

(float)

synapse.lib.gis.haversine(px, py, r=6371008800.0)

Calculate the haversine distance between two points defined by (lat,lon) tuples.

Parameters

• px ((float,float)) – lat/long position 1

• py ((float,float)) – lat/long position 2

• r (float) – Radius of sphere

Returns

Distance in mm.

Return type

(int)

synapse.lib.gis.latlong(text)

Chop a latlong string and return (float,float). Does not perform validation on the coordinates.

Parameters

text (str) – A longitude,latitude string.

Returns

A longitude, latitude float tuple.

Return type

(float,float)

synapse.lib.gis.near(point, dist, points)

Determine if the given point is within dist of any of points.

Parameters

• point ((float,float)) – A latitude, longitude float tuple.

• dist (int) – A distance in mm ( base units )

• points (list) – A list of latitude, longitude float tuples to compare against.

synapse.lib.grammar module

synapse.lib.grammar.chop_float(text, off)

synapse.lib.grammar.isBasePropNoPivprop(name)

synapse.lib.grammar.isCmdName(name)

synapse.lib.grammar.isFormName(name)

synapse.lib.grammar.isPropName(name)

synapse.lib.grammar.isUnivName(name)

synapse.lib.grammar.meh(txt, off, cset)

synapse.lib.grammar.nom(txt, off, cset, trim=True)

Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’)

name,off = nom(text,0,chars)

Note:

This really shouldn’t be used for new code

synapse.lib.grammar.parse_float(text, off)

synapse.lib.hashitem module

synapse.lib.hashitem.hashitem(item)

Generate a uniq hash for the JSON compatible primitive data structure.

synapse.lib.hashitem.normdict(item)

synapse.lib.hashitem.normitem(item)

synapse.lib.hashitem.normiter(item)

synapse.lib.hashset module

class synapse.lib.hashset.HashSet

Bases: object

digests()

Get a list of (name, bytes) tuples for the hashes in the hashset.

eatfd(fd)

Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd)

guid()

Use elements from this hash set to create a unique (re)identifier.

update(byts)

Update all the hashes in the set with the given bytes.

synapse.lib.health module

class synapse.lib.health.HealthCheck(iden)

Bases: object

getStatus()

pack()

setStatus(valu)

update(name, status, mesg='', data=None)

Append a new component to the Healcheck object.

Parameters

• name (str) – Name of the reported component.

• status (str) – nomdinal/degraded/failed status code.

• mesg (str) – Optional message about the component status.

• data (dict) – Optional arbitrary dictionary of additional metadata about the component.

Returns

None

synapse.lib.hive module

class synapse.lib.hive.Hive

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.

async add(full, valu)

Atomically increments a node’s value.

async dict(full, nexs=False)

Open a HiveDict at the given full path.

Parameters

full (tuple) – A full path tuple.

Returns

A HiveDict for the full path.

Return type

HiveDict

dir(full)

List subnodes of the given Hive path.

Parameters

full (tuple) – A full path tuple.

Notes

This returns None if there is not a node at the path.

Returns

A list of tuples. Each tuple contains the name, node value, and the number of children nodes.

Return type

list

async exists(full)

Returns whether the Hive path has already been created.

async get(full, defv=None)

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async getHiveAuth()

Retrieve a HiveAuth for hive standalone or non-cell uses.

Note

This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell

async getTeleApi(link, mesg, path)

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async loadHiveTree(tree, path=(), trim=False)

async open(full)

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(full, nexs=False)

Remove and return the value for the given node.

async rename(oldpath, newpath)

Moves a node at oldpath and all its descendant nodes to newpath. newpath must not exist

async saveHiveTree(path=())

async set(full, valu, nexs=False)

A set operation at the hive level (full path).

async storNodeDele(path)

async storNodeValu(full, valu)

class synapse.lib.hive.HiveApi

Bases: synapse.lib.base.Base

async addAndSync(path, valu, iden)

async edits()

async get(full)

async loadHiveTree(tree, path=(), trim=False)

async popAndSync(path, iden, nexs=False)

async saveHiveTree(path=())

async setAndSync(path, valu, iden, nexs=False)

async treeAndSync(path, iden)

class synapse.lib.hive.HiveDict

Bases: synapse.lib.base.Base

get(name, default=None)

items()

pack()

async pop(name, default=None)

async set(name, valu, nexs=None)

setdefault(name, valu)

values()

class synapse.lib.hive.Node

Bases: synapse.lib.base.Base

A single node within the Hive tree.

async add(valu)

Increments existing node valu

async dict(nexs=False)

Get a HiveDict for this Node.

Returns

A HiveDict for this Node.

Return type

HiveDict

dir()

get(name)

name()

async open(path)

Open a child Node of the this Node.

Parameters

path (tuple) – A child path of the current node.

Returns

A Node at the child path.

Return type

Node

parent()

async pop(path=())

async set(valu)

class synapse.lib.hive.SlabHive

Bases: synapse.lib.hive.Hive

async storNodeDele(full)

async storNodeValu(full, valu)

class synapse.lib.hive.TeleHive

Bases: synapse.lib.hive.Hive

A Hive that acts as a consistent read cache for a telepath proxy Hive

async add(path, valu)

Atomically increments a node’s value.

async get(path)

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async open(path)

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(path, nexs=False)

Remove and return the value for the given node.

async set(path, valu, nexs=False)

A set operation at the hive level (full path).

synapse.lib.hive.iterpath(path)

async synapse.lib.hive.opendir(dirn, conf=None)

async synapse.lib.hive.openurl(url, **opts)

synapse.lib.hiveauth module

class synapse.lib.hiveauth.Auth

Bases: synapse.lib.nexus.Pusher

Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.

Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).

Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.

Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.

Node layout:

Auth root (passed into constructor)
├ roles
│   ├ <role iden 1>
│   ├ ...
│   └ last role
├ users
│   ├ <user iden 1>
│   ├ ...
│   └ last user
└ authgates
    ├ <iden 1>
    │   ├ roles
    │   │   ├ <role iden 1>
    │   │   ├ ...
    │   │   └ last role
    │   └ users
    │       ├ <user iden 1>
    │       ├ ...
    │       └ last user
    ├ <iden 2>
    │   ├ ...
    └ ... last authgate

async addAuthGate(iden, authgatetype)

Retrieve AuthGate by iden. Create if not present.

Note

Not change distributed

Returns

(HiveAuthGate)

async addRole(name, iden=None)

async addUser(name, passwd=None, email=None, iden=None)

Add a User to the Hive.

Parameters

• name (str) – The name of the User.

• passwd (str) – A optional password for the user.

• email (str) – A optional email for the user.

• iden (str) – A optional iden to use as the user iden.

Returns

A Hive User.

Return type

HiveUser

async delAuthGate(iden)

Delete AuthGate by iden.

Note

Not change distributed

async delRole(iden)

async delUser(iden)

async feedBeholder(evnt, info, gateiden=None, logged=True)

getAuthGate(iden)

getAuthGates()

async getRoleByName(name)

async getUserByName(name)

Get a user by their username.

Parameters

name (str) – Name of the user to get.

Returns

A Hive User. May return None if there is no user by the requested name.

Return type

HiveUser

async getUserIdenByName(name)

reqAuthGate(iden)

async reqRole(iden)

async reqRoleByName(name)

async reqUser(iden)

async reqUserByName(name)

async reqUserByNameOrIden(name)

role(iden)

roles()

async setRoleInfo(iden, name, valu, gateiden=None, logged=True, mesg=None)

async setRoleName(iden, name)

async setUserInfo(iden, name, valu, gateiden=None, logged=True, mesg=None)

async setUserName(iden, name)

user(iden)

users()

class synapse.lib.hiveauth.AuthGate

Bases: synapse.lib.base.Base

The storage object for object specific rules for users/roles.

async delete()

async genRoleInfo(iden)

async genUserInfo(iden)

pack()

class synapse.lib.hiveauth.HiveRole

Bases: synapse.lib.hiveauth.HiveRuler

A role within the Hive authorization subsystem.

A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.

allowed(perm, default=None, gateiden=None)

clearAuthCache()

async genGateInfo(gateiden)

pack()

async setName(name)

class synapse.lib.hiveauth.HiveRuler

Bases: synapse.lib.base.Base

A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those

async addRule(rule, indx=None, gateiden=None, nexs=True)

async delRule(rule, gateiden=None)

getRules(gateiden=None)

async setRules(rules, gateiden=None, nexs=True, mesg=None)

class synapse.lib.hiveauth.HiveUser

Bases: synapse.lib.hiveauth.HiveRuler

A user (could be human or computer) of the system within HiveAuth.

Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.

async allow(perm)

allowed(perm, default=None, gateiden=None)

clearAuthCache()

confirm(perm, default=None, gateiden=None)

async genGateInfo(gateiden)

getRoles()

async grant(roleiden, indx=None)

hasRole(iden)

isAdmin(gateiden=None)

isLocked()

pack(packroles=False)

raisePermDeny(perm, gateiden=None)

async revoke(iden, nexs=True)

async setAdmin(admin, gateiden=None, logged=True)

async setArchived(archived)

async setLocked(locked, logged=True)

async setName(name)

async setPasswd(passwd, nexs=True)

async setRoles(roleidens)

Replace all the roles for a given user with a new list of roles.

Parameters

roleidens (list) – A list of roleidens.

Notes

The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.

Returns

None

async tryPasswd(passwd, nexs=True)

synapse.lib.hiveauth.getShadow(passwd)

This API is deprecated.

synapse.lib.httpapi module

class synapse.lib.httpapi.ActiveV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.AuthAddRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthAddUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthDelRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthGrantV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()

async post()

class synapse.lib.httpapi.AuthRevokeV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()

async post()

class synapse.lib.httpapi.AuthRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get(iden)

async post(iden)

class synapse.lib.httpapi.AuthRolesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.AuthUserPasswdV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post(iden)

class synapse.lib.httpapi.AuthUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get(iden)

async post(iden)

class synapse.lib.httpapi.AuthUsersV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.BeholdSockV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.WebSocket

async onInitMessage(byts)

async on_message(byts)

Handle incoming messages on the WebSocket

This method must be overridden.

Changed in version 4.5: on_message can be a coroutine.

class synapse.lib.httpapi.CoreInfoV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/core/info

async get()

class synapse.lib.httpapi.FeedV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/feed

Examples

Example data:

{
    'name': 'syn.nodes',
    'view': null,
    'items': [...],
}

async post()

class synapse.lib.httpapi.Handler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

on_connection_close()

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

prepare()

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

class synapse.lib.httpapi.HandlerBase

Bases: object

async allowed(perm, gateiden=None)

Check if the authenticated user has the given permission.

Parameters

• perm (tuple) – The permission tuple to check.

• gateiden (str) – The gateiden to check the permission against.

Notes

This API sets up HTTP response values if it returns False.

Returns

True if the user has the requested permission.

Return type

bool

async authenticated()

Check if the request has an authenticated user or not.

Returns

True if the request has an authenticated user, false otherwise.

Return type

bool

check_origin(origin)

getAuthCell()

Return a reference to the cell used for auth operations.

getCustomHeaders()

getJsonBody(validator=None)

async getUserBody()

Helper function to confirm that there is an auth user and a valid JSON body in the request.

Deprecated, use the getUseridenBody() function instead.

async getUseridenBody(validator=None)

Helper function to confirm that there is an auth user and a valid JSON body in the request.

Parameters

validator – Validator function run on the deserialized JSON body.

Returns

The user definition and body of the request as deserialized JSON, or a tuple of s_common.novalu objects if there was no user or json body.

Return type

(str, object)

async handleBasicAuth()

Handle basic authentication in the handler.

Notes

Implementors may override this to disable or implement their own basic auth schemes. This is expected to set web_useriden and web_username upon successful authentication.

Returns

The user iden of the logged in user.

Return type

str

initialize(cell)

isOrigHost(origin)

async isUserAdmin()

Check if the current authenticated user is an admin or not.

Returns

True if the user is an admin, false otherwise.

Return type

bool

loadJsonMesg(byts, validator=None)

options()

async reqAuthAdmin()

Require the current authenticated user to be an admin.

Notes

If this returns False, an error message has already been sent and no additional processing for the request should be done.

Returns

True if the user is an admin, false otherwise.

Return type

bool

async reqAuthUser()

sendAuthRequired()

sendRestErr(code, mesg)

sendRestExc(e)

sendRestRetn(valu)

async sess(gen=True)

Get the heavy Session object for the request.

Parameters

gen (bool) – If set to True, generate a new session if there is no sess cookie.

Notes

This stores the identifier in the sess cookie for with a 14 day expiration, stored in the Cell. Valid requests with that sess cookie will resolve to the same Session object.

Returns

A heavy session object. If the sess cookie is invalid or gen is false, this returns None.

Return type

Sess

set_default_headers()

async user()

Get the heavy User object from the Cell for the authenticated user.

Deprecated, use the useriden() function instead.

async useriden()

Get the user iden of the current session user.

Note

This function will pull the iden from the current session, or attempt to resolve the useriden with basic authentication.

Returns

The iden of the current session user.

Return type

str

class synapse.lib.httpapi.HealthCheckV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.LoginV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.ModelNormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

async post()

class synapse.lib.httpapi.ModelV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.OnePassIssueV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/onepass/issue

async post()

class synapse.lib.httpapi.ReqValidStormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.RobotHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

async get()

class synapse.lib.httpapi.Sess

Bases: synapse.lib.base.Base

addWebSock(sock)

delWebSock(sock)

async login(user)

async logout()

async set(name, valu)

class synapse.lib.httpapi.StormCallV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormExportV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

getCore()

class synapse.lib.httpapi.StormNodesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest,