synapse.lib package

Subpackages

Submodules

synapse.lib.agenda module

class synapse.lib.agenda.Agenda[source]

Bases: synapse.lib.base.Base

Organize and execute all the scheduled storm queries in a cortex.

async add(cdef)[source]

Persistently adds an appointment

Parameters

cdef (dict) –

May contain the following keys: creator (str): iden of the creating user iden (str): Iden of the appointment storm (str): storm query to run reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…]):

one or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.

incunit (Union[None, TimeUnit]):

the unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.

incvals (Union[None, int, Iterable[int]): count of units of incunit or explicit day of week or day of

month. Not allowed for incunit == None, required for others (1 would be a typical value)

Notes

For values in reqs that are lists and incvals if a list, all combinations of all values (the product) are used

Returns

Packed appointment definition

async delete(iden)[source]

Delete an appointment

async disable(iden)[source]
async enable(iden)[source]
async get(iden)[source]
list()[source]
async mod(iden, query)[source]

Change the query of an appointment

async move(croniden, viewiden)[source]

Move a cronjob from one view to another

async start()[source]

Enable cron jobs to start running, start the scheduler loop

Go through all the appointments, making sure the query is valid, and remove the ones that aren’t. (We can’t evaluate queries until enabled because not all the modules are loaded yet.)

async stop()[source]

Cancel the scheduler loop, and set self.enabled to False.

class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)[source]

Bases: object

Represents a single element of a single combination of an appointment

nexttime(lastts)[source]

Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

pack()[source]

Make ApptRec json/msgpack-friendly

classmethod unpack(val)[source]

Convert from json/msgpack-friendly

class synapse.lib.agenda.TimeUnit(value)[source]

Bases: enum.IntEnum

Unit of time that recurring and required parts of appointments are made of

DAY = 5
DAYOFMONTH = 3
DAYOFWEEK = 4
HOUR = 6
MINUTE = 7
MONTH = 2
NOW = 8
YEAR = 1
classmethod fromString(s)[source]

synapse.lib.aha module

class synapse.lib.aha.AhaApi[source]

Bases: synapse.lib.cell.CellApi

async addAhaSvc(name, info, network=None)[source]

Register a service with the AHA discovery server.

NOTE: In order for the service to remain marked “up” a caller

must maintain the telepath link.

async delAhaSvc(name, network=None)[source]

Remove an AHA service entry.

async genCaCert(network)[source]
async getAhaSvc(name)[source]

Return an AHA service description dictionary for a fully qualified service name.

async getAhaSvcs(network=None)[source]

Yield AHA svcinfo dictionaries.

Parameters

network (str) – Optionally specify a network to filter on.

async getAhaUrls()[source]
async getCaCert(network)[source]
async signHostCsr(csrtext, signas=None, sans=None)[source]
async signUserCsr(csrtext, signas=None)[source]
class synapse.lib.aha.AhaCell[source]

Bases: synapse.lib.cell.Cell

async addAhaSvc(name, info, network=None)[source]
cellapi

alias of synapse.lib.aha.AhaApi

confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}}
async delAhaSvc(name, network=None)[source]
async genCaCert(network)[source]
async getAhaSvc(name)[source]
async getAhaSvcs(network=None)[source]
async getCaCert(network)[source]
async initServiceRuntime()[source]
async initServiceStorage()[source]
async saveCaCert(name, cakey, cacert)[source]
async setAhaSvcDown(name, linkiden, network=None)[source]
async signHostCsr(csrtext, signas=None, sans=None)[source]
async signUserCsr(csrtext, signas=None)[source]

synapse.lib.ast module

class synapse.lib.ast.AbsProp(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.AbsPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AndCond(kids=())[source]

Bases: synapse.lib.ast.Cond

<cond> and <cond>

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]
class synapse.lib.ast.ArgvQuery(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
isRuntSafe(runt)[source]
validate(runt)[source]
class synapse.lib.ast.ArrayCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AstNode(kids=())[source]

Bases: object

Base class for all nodes in the Storm abstract syntax tree.

addKid(astn)[source]
format(depth=0)[source]
getRuntVars(runt)[source]
hasAstClass(clss)[source]
init(core)[source]
isRuntSafe(runt)[source]
iterright()[source]

Yield “rightward” siblings until None.

optimize()[source]
prepare()[source]
repr()[source]
sibling(offs=1)[source]

Return sibling node by relative offset from self.

validate(runt)[source]
class synapse.lib.ast.Bool(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.BreakOper(kids=())[source]

Bases: synapse.lib.ast.AstNode

async run(runt, genr)[source]
class synapse.lib.ast.CallArgs(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.CallKwarg(kids=())[source]

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CallKwargs(kids=())[source]

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CaseEntry(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.CatchBlock(kids=())[source]

Bases: synapse.lib.ast.AstNode

async catches(name, runt, path=None)[source]
errvar()[source]
getRuntVars(runt)[source]
async run(runt, genr)[source]
class synapse.lib.ast.CmdOper(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.Cmpr(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.Cond(kids=())[source]

Bases: synapse.lib.ast.Value

A condition that is evaluated to filter nodes.

class synapse.lib.ast.Const(valu, kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
isRuntSafe(runt)[source]
repr()[source]
value()[source]
class synapse.lib.ast.ContinueOper(kids=())[source]

Bases: synapse.lib.ast.AstNode

async run(runt, genr)[source]
class synapse.lib.ast.DollarExpr(kids=())[source]

Bases: synapse.lib.ast.Value

Top level node for $(…) expressions

async compute(runt, path)[source]
class synapse.lib.ast.Edit(kids=())[source]

Bases: synapse.lib.ast.Oper

class synapse.lib.ast.EditEdgeAdd(kids=(), n2=False)[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditEdgeDel(kids=(), n2=False)[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditNodeAdd(kids=())[source]

Bases: synapse.lib.ast.Edit

async addFromPath(form, runt, path)[source]

Add a node using the context from path.

NOTE: CALLER MUST CHECK PERMS

prepare()[source]
async run(runt, genr)[source]
class synapse.lib.ast.EditParens(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditPropDel(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditPropSet(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditTagAdd(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditTagDel(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EditTagPropDel(kids=())[source]

Bases: synapse.lib.ast.Edit

[ -#foo.bar:baz ]

async run(runt, genr)[source]
class synapse.lib.ast.EditTagPropSet(kids=())[source]

Bases: synapse.lib.ast.Edit

[ #foo.bar:baz=10 ]

async run(runt, genr)[source]
class synapse.lib.ast.EditUnivDel(kids=())[source]

Bases: synapse.lib.ast.Edit

async run(runt, genr)[source]
class synapse.lib.ast.EmbedQuery(valu, kids=())[source]

Bases: synapse.lib.ast.Const

async compute(runt, path)[source]
getRuntVars(runt)[source]
validate(runt)[source]
class synapse.lib.ast.Emit(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.ExprAndNode(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.ExprNode(kids=())[source]

Bases: synapse.lib.ast.Value

A binary (i.e. two argument) expression node

async compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.ExprOrNode(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.FiltByArray(kids=())[source]

Bases: synapse.lib.ast.FiltOper

+:foo*[^=visi]

class synapse.lib.ast.FiltOper(kids=())[source]

Bases: synapse.lib.ast.Oper

async getLiftHints(runt, path)[source]
async run(runt, genr)[source]
class synapse.lib.ast.FiniBlock(kids=())[source]

Bases: synapse.lib.ast.AstNode

An AST node that runs only once after all nodes have been consumed.

Example

Using a fini block:

fini {
   // stuff here runs *once* after the last node yield (even if there are no nodes)
}

Notes

A fini block must be runtsafe.

async run(runt, genr)[source]
class synapse.lib.ast.ForLoop(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
async run(runt, genr)[source]
class synapse.lib.ast.FormName(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.FormPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> foo:bar

async run(runt, genr)[source]
class synapse.lib.ast.FormTagProp(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.FuncArgs(kids=())[source]

Bases: synapse.lib.ast.AstNode

Represents the function arguments in a function definition

async compute(runt, path)[source]
class synapse.lib.ast.FuncCall(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.Function(kids=())[source]

Bases: synapse.lib.ast.AstNode

( name, args, body )

// use args/kwargs syntax function bar(x, v=$(30)) { }

# we auto-detect the behavior of the target function

# return a value function bar(x, y) { return ($(x + y)) }

# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }

$foo = $bar(10, v=20)

async callfunc(runt, argdefs, args, kwargs)[source]

Execute a function call using the given runtime.

This function may return a value / generator / async generator

getRuntVars(runt)[source]
isRuntSafe(runt)[source]
prepare()[source]
async run(runt, genr)[source]
validate(runt)[source]
class synapse.lib.ast.HasAbsPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.HasRelPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]
async hasProp(node, runt, name)[source]
class synapse.lib.ast.HasTagPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.IfClause(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.IfStmt(kids=())[source]

Bases: synapse.lib.ast.Oper

prepare()[source]
async run(runt, genr)[source]
class synapse.lib.ast.InitBlock(kids=())[source]

Bases: synapse.lib.ast.AstNode

An AST node that runs only once before yielding nodes.

Example

Using a init block:

init {
    // stuff here runs *once* before the first node yield (even if there are no nodes)
}
async run(runt, genr)[source]
class synapse.lib.ast.LiftByArray(kids=())[source]

Bases: synapse.lib.ast.LiftOper

:prop*[range=(200, 400)]

async lift(runt, path)[source]
class synapse.lib.ast.LiftFormTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)[source]
class synapse.lib.ast.LiftFormTagProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

hehe:haha#foo.bar:baz [ = x ]

async lift(runt, path)[source]
class synapse.lib.ast.LiftOper(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.LiftProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

async getRightHints(runt, path)[source]
async lift(runt, path)[source]
class synapse.lib.ast.LiftPropBy(kids=())[source]

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)[source]
class synapse.lib.ast.LiftTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)[source]
class synapse.lib.ast.LiftTagProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

#foo.bar:baz [ = x ]

async lift(runt, path)[source]
class synapse.lib.ast.LiftTagTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

##foo.bar

async lift(runt, path)[source]
class synapse.lib.ast.List(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
repr()[source]
class synapse.lib.ast.Lookup(kids, autoadd=False)[source]

Bases: synapse.lib.ast.Query

When storm input mode is “lookup”

async run(runt, genr)[source]
class synapse.lib.ast.N1Walk(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
async walkNodeEdges(runt, node, verb=None)[source]
class synapse.lib.ast.N1WalkNPivo(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOut

async run(runt, genr)[source]
class synapse.lib.ast.N2Walk(kids=())[source]

Bases: synapse.lib.ast.N1Walk

async walkNodeEdges(runt, node, verb=None)[source]
class synapse.lib.ast.N2WalkNPivo(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotIn

async run(runt, genr)[source]
class synapse.lib.ast.NotCond(kids=())[source]

Bases: synapse.lib.ast.Cond

not <cond>

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.Oper(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.OrCond(kids=())[source]

Bases: synapse.lib.ast.Cond

<cond> or <cond>

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.PivotIn(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

<- *

async getPivsIn(runt, node, path)[source]
async run(runt, genr)[source]
class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

<- foo:edge

async run(runt, genr)[source]
class synapse.lib.ast.PivotOper(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.Oper

repr()[source]
class synapse.lib.ast.PivotOut(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> *

async getPivsOut(runt, node, path)[source]
async run(runt, genr)[source]
class synapse.lib.ast.PivotToTags(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present

async run(runt, genr)[source]
class synapse.lib.ast.PropName(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.PropPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

:foo -> bar:foo

async run(runt, genr)[source]
class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

:prop -> *

async run(runt, genr)[source]
class synapse.lib.ast.PropValue(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
async getPropAndValu(runt, path)[source]
isRuntSafe(valu)[source]
prepare()[source]
class synapse.lib.ast.Query(kids=())[source]

Bases: synapse.lib.ast.AstNode

async iterNodePaths(runt, genr=None)[source]
async run(runt, genr)[source]
class synapse.lib.ast.RawPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> { <varsfrompath> }

async run(runt, genr)[source]
class synapse.lib.ast.RelProp(kids=())[source]

Bases: synapse.lib.ast.PropName

class synapse.lib.ast.RelPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

(:foo:bar or .univ) <cmpr> <value>

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]
class synapse.lib.ast.RelPropValue(kids=())[source]

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Return(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.SetItemOper(kids=())[source]

Bases: synapse.lib.ast.Oper

$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz

async run(runt, genr)[source]
class synapse.lib.ast.SetVarOper(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
async run(runt, genr)[source]
class synapse.lib.ast.Stop(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.SubGraph(rules)[source]

Bases: object

An Oper like object which generates a subgraph.

Notes

The rules format for the subgraph is shaped like the following:

rules = {

    'degrees': 1,

    'edges': True,
    'filterinput': True,
    'yieldfiltered': False,

    'filters': [
        '-(#foo or #bar)',
        '-(foo:bar or baz:faz)',
    ],

    'pivots': [
        '-> * | limit 100',
        '<- * | limit 100',
    ]

    'forms': {

        'inet:fqdn':{
            'filters': [],
            'pivots': [],
        }

        '*': {
            'filters': [],
            'pivots': [],
        },
    },
}

Nodes which were original seeds have path.meta(‘graph:seed’).

All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.

async omit(runt, node)[source]
async pivots(runt, node, path)[source]
async run(runt, genr)[source]
class synapse.lib.ast.SubQuery(kids=())[source]

Bases: synapse.lib.ast.Oper

async compute(runt, path)[source]

Use subquery as a value. It is error if the subquery used in this way doesn’t yield exactly one node or has a return statement.

Its value is the primary property of the node yielded, or the returned value.

async compute_array(runt, path)[source]

Use subquery as an array.

async inline(runt, genr)[source]

Operate subquery as if it were inlined

async run(runt, genr)[source]
class synapse.lib.ast.SubqCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.SwitchCase(kids=())[source]

Bases: synapse.lib.ast.Oper

prepare()[source]
async run(runt, genr)[source]
class synapse.lib.ast.TagCond(kids=())[source]

Bases: synapse.lib.ast.Cond

#foo.bar

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]
class synapse.lib.ast.TagMatch(kids=())[source]

Bases: synapse.lib.ast.TagName

Like TagName, but can have asterisks

hasglob()[source]
class synapse.lib.ast.TagName(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.TagProp(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.TagPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagPropValue(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.TagValuCond(kids=())[source]

Bases: synapse.lib.ast.Cond

async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagValue(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
isRuntSafe(runt)[source]
class synapse.lib.ast.TryCatch(kids=())[source]

Bases: synapse.lib.ast.AstNode

async getCatchBlock(name, runt, path=None)[source]
async getErrValu(e)[source]
async run(runt, genr)[source]
class synapse.lib.ast.UnaryExprNode(kids=())[source]

Bases: synapse.lib.ast.Value

A unary (i.e. single-argument) expression node

async compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.UnivProp(kids=())[source]

Bases: synapse.lib.ast.RelProp

async compute(runt, path)[source]
class synapse.lib.ast.UnivPropValue(kids=())[source]

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Value(kids=())[source]

Bases: synapse.lib.ast.AstNode

The base class for all values and value expressions.

async compute(runt, path)[source]
async getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)[source]
isRuntSafe(runt)[source]
class synapse.lib.ast.VarDeref(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
class synapse.lib.ast.VarEvalOper(kids=())[source]

Bases: synapse.lib.ast.Oper

Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)

async run(runt, genr)[source]
class synapse.lib.ast.VarList(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.VarListSetOper(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
async run(runt, genr)[source]
class synapse.lib.ast.VarValue(kids=())[source]

Bases: synapse.lib.ast.Value

async compute(runt, path)[source]
isRuntSafe(runt)[source]
prepare()[source]
validate(runt)[source]
class synapse.lib.ast.WhileLoop(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
class synapse.lib.ast.YieldValu(kids=())[source]

Bases: synapse.lib.ast.Oper

async run(runt, genr)[source]
async yieldFromValu(runt, valu)[source]
async synapse.lib.ast.expr_add(x, y)[source]
async synapse.lib.ast.expr_div(x, y)[source]
async synapse.lib.ast.expr_eq(x, y)[source]
async synapse.lib.ast.expr_ge(x, y)[source]
async synapse.lib.ast.expr_gt(x, y)[source]
async synapse.lib.ast.expr_le(x, y)[source]
async synapse.lib.ast.expr_lt(x, y)[source]
async synapse.lib.ast.expr_mul(x, y)[source]
async synapse.lib.ast.expr_ne(x, y)[source]
async synapse.lib.ast.expr_not(x)[source]
async synapse.lib.ast.expr_prefix(x, y)[source]
async synapse.lib.ast.expr_re(x, y)[source]
async synapse.lib.ast.expr_sub(x, y)[source]
synapse.lib.ast.parseNumber(x)[source]
async synapse.lib.ast.pullone(genr)[source]

synapse.lib.autodoc module

class synapse.lib.autodoc.RstHelp[source]

Bases: object

addHead(name, lvl=0, link=None)[source]
addLines(*lines)[source]
getRstText()[source]
synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)[source]

Process a list of StormTypes doc information to add them to a a RstHelp object.

Notes

This will create internal hyperlink link targets for each header item. The link prefix string must be given with the linkprefix argument.

Parameters
  • page (RstHelp) – The RST page to add .

  • docinfo (dict) – A Stormtypes Doc.

  • linkprefix (str) – The RST link prefix string to use.

  • islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.

  • lvl (int) – The base header level to use when adding headers to the page.

Returns

None

synapse.lib.autodoc.genCallsig(rtype)[source]
synapse.lib.autodoc.getArgLines(rtype)[source]
synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None)[source]
synapse.lib.autodoc.getRtypeStr(rtype, known_types, types_prefix, suffix)[source]
synapse.lib.autodoc.ljuster(ilines)[source]

Helper to lstrip lines of whitespace an appropriate amount.

synapse.lib.autodoc.prepareRstLines(doc)[source]

Prepare a desc string for RST lines.

synapse.lib.autodoc.scrubLines(lines)[source]

Remove any empty lines until we encounter non-empty linee

synapse.lib.base module

class synapse.lib.base.Base[source]

Bases: object

Base class for Synapse objects.

Acts as an observable, enables async init and fini.

Example

class Foo(Base):

async def __anit__(self, x, y):

await Base.__anit__(self)

await stuff(x, y)

foo = await Foo.anit(10)

Note

One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

async addSignalHandlers()[source]

Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

async classmethod anit(*args, **kwargs)[source]
async dist(mesg)[source]

Distribute an existing event tuple.

Parameters

mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) )

async enter_context(item)[source]

Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.

Returns

The result of item’s own __aenter__ or __enter__() method.

async fini()[source]

Shut down the object and notify any onfini() coroutines.

Returns

Remaining ref count

async fire(evtname, **info)[source]

Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’):

print(‘got: %r’ % (ret,))

incref()[source]

Increment the reference count for this base. This API may be optionally used to control fini().

Add a callback function to receive all events.

Example

base1 = Base() base2 = Base()

base1.link( base2.dist )

# all events on base1 are also propagated on base2

async main()[source]

Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )

NOTE: This API may only be used when the ioloop is also the main thread.

off(evnt, func)[source]

Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc )

on(evnt, func, base=None)[source]

Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.

Parameters
  • evnt (str) – An event name

  • func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it:

async def baz(event):

x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y

d.on(‘foo’, baz)

# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)

Return type

None

onWith(evnt, func)[source]

A context manager which can be used to add a callback and remove it when using a with statement.

Parameters
  • evnt (str) – An event name

  • func (function) – A callback function to receive event tufo

onfini(func)[source]

Add a function/coroutine/Base to be called on fini().

async postAnit()[source]

Method called after self.__anit__() has completed, but before anit() returns the object to the caller.

schedCallSafe(func, *args, **kwargs)[source]

Schedule a function to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on the function completion.

Parameters
  • func

  • *args

  • **kwargs

Notes

This method may be called from outside of the event loop on a different thread.

Returns

A Future representing the eventual function execution.

Return type

concurrent.futures.Future

schedCoro(coro)[source]

Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.

Precondition:

This function is not threadsafe and must be run on the Base’s event loop

Returns

An asyncio.Task object.

Return type

asyncio.Task

schedCoroSafe(coro)[source]

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on coroutine completion.

Notes

This method may be run outside the event loop on a different thread.

Returns

A Future representing the eventual coroutine execution.

Return type

concurrent.futures.Future

schedCoroSafePend(coro)[source]

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note

This method may not be run inside an event loop

Remove a callback function previously added with link()

Example

base.unlink( callback )

waiter(count, *names)[source]

Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events…

waiter = base.waiter(10,’foo:bar’)

# .. fire thread that will cause foo:bar events

events = await waiter.wait(timeout=3)

if events == None:

# handle the timeout case…

for event in events:

# parse the events if you need…

NOTE: use with caution… it’s easy to accidentally construct

race conditions with this mechanism ;)

async waitfini(timeout=None)[source]

Wait for the base to fini()

Returns

None if timed out, True if fini happened

Example

base.waitfini(timeout=30)

class synapse.lib.base.BaseRef[source]

Bases: synapse.lib.base.Base

An object for managing multiple Base instances by name.

async gen(name)[source]

Atomically get/gen a Base and incref. (requires ctor during BaseRef init)

Parameters

name (str) – The name/iden of the Base instance.

get(name)[source]

Retrieve a Base instance by name.

Parameters

name (str) – The name/iden of the Base

Returns

The Base instance (or None)

Return type

(Base)

items()[source]
pop(name)[source]

Remove and return a Base from the BaseRef.

Parameters

name (str) – The name/iden of the Base instance

Returns

The named base ( or None )

Return type

(Base)

put(name, base)[source]

Add a Base (or sub-class) to the BaseRef by name.

Parameters
  • name (str) – The name/iden of the Base

  • base (Base) – The Base instance

Returns

(None)

vals()[source]
class synapse.lib.base.Waiter(base, count, *names)[source]

Bases: object

A helper to wait for a given number of events on a Base.

fini()[source]
async wait(timeout=None)[source]

Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30)

if evnts == None:

handleTimedOut() return

for evnt in evnts:

doStuff(evnt)

async synapse.lib.base.main(coro)[source]
async synapse.lib.base.schedGenr(genr, maxsize=100)[source]

Schedule a generator to run on a separate task and yield results to this task (pipelined generator).

synapse.lib.boss module

class synapse.lib.boss.Boss[source]

Bases: synapse.lib.base.Base

An object to track “promoted” async tasks.

async execute(coro, name, user, info=None)[source]

Create a synapse task from the given coroutine.

get(iden)[source]
async promote(name, user, info=None)[source]

Promote the currently running task.

Parameters
  • name (str) – The name of the task.

  • user – The User who owns the task.

  • info – An optional information dictionary containing information about the task.

Returns

The Synapse Task object.

Return type

s_task.Task

ps()[source]

synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully.

class synapse.lib.cache.FixedCache(callback, size=10000)[source]

Bases: object

async aget(key)[source]
clear()[source]
get(key)[source]
pop(key)[source]
put(key, val)[source]
class synapse.lib.cache.LruDict(size=10000)[source]

Bases: collections.abc.MutableMapping

Maintains the last n accessed keys

get(key, default=None)[source]

Note: we override default impl from parent to avoid costly KeyError

items() a set-like object providing a view on D's items[source]
values() an object providing a view on D's values[source]
class synapse.lib.cache.TagGlobs[source]

Bases: object

An object that manages multiple tag globs and values for caching.

add(name, valu, base=None)[source]
get(name)[source]
rem(name, valu)[source]
synapse.lib.cache.getTagGlobRegx(name)[source]
synapse.lib.cache.memoize(size=16384)[source]
synapse.lib.cache.memoizemethod(size=16384)[source]

A version of memoize that doesn’t cause GC cycles when applied to a method.

synapse.lib.cache.regexizeTagGlob(tag)[source]
Returns

a regular expression string with ** and * interpreted as tag globs

Precondition:

tag is a valid tagmatch

Notes

A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.

The returned string does not contain a starting ‘^’ or trailing ‘$’.

synapse.lib.cell module

class synapse.lib.cell.Cell[source]

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

A Cell() implements a synapse micro-service.

A Cell has 5 phases of startup:
  1. Universal cell data structures

  2. Service specific storage/data (pre-nexs)

  3. Nexus subsystem initialization

  4. Service specific startup (with nexus)

  5. Networking and mirror services

BACKUP_SPAWN_TIMEOUT = 60.0
COMMIT = ''
VERSION = (2, 78, 0)
VERSTRING = '2.78.0'
addActiveCoro(func, iden=None, base=None)[source]

Add a function callback to be run as a coroutine when the Cell is active.

Parameters
  • func (coroutine function) – The function run as a coroutine.

  • iden (str) – The iden to use for the coroutine.

  • base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d

Returns

A GUID string that identifies the coroutine for delActiveCoro()

Return type

str

Note

This will re-fire the coroutine if it exits and the Cell is still active.

addHealthFunc(func)[source]

Register a callback function to get a HealthCheck object.

addHttpApi(path, ctor, info)[source]
async addHttpsPort(port, host='0.0.0.0', sslctx=None)[source]
async addRole(name)[source]
async addRoleRule(iden, rule, indx=None, gateiden=None)[source]
async addUser(name, passwd=None, email=None, iden=None)[source]
async addUserRole(useriden, roleiden)[source]
async addUserRule(iden, rule, indx=None, gateiden=None)[source]
cellapi

alias of synapse.lib.cell.CellApi

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class.  Used by stemcell.', 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the cell.', 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'type': 'string'}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to the cell.  Required for mirroring (on both sides).', 'type': 'boolean'}}
confdefs = {}
async cullNexsLog(offs)[source]
async delActiveCoro(iden)[source]

Remove an Active coroutine previously added with addActiveCoro().

Parameters

iden (str) – The iden returned by addActiveCoro()

async delBackup(name)[source]
async delRole(iden)[source]
async delRoleRule(iden, rule, gateiden=None)[source]
async delUser(iden)[source]
async delUserRole(useriden, roleiden)[source]
async delUserRule(iden, rule, gateiden=None)[source]
async dyncall(iden, todo, gatekeys=())[source]
async dyniter(iden, todo, gatekeys=())[source]
async classmethod execmain(argv, outp=None)[source]

The main entry point for running the Cell as an application.

Parameters
  • argv (list) – A list of command line arguments to launch the Cell with.

  • outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.

Returns

None.

async genHttpSess(iden)[source]
classmethod getArgParser(conf=None)[source]

Get an argparse.ArgumentParser for the Cell.

Parameters

conf (s_config.Config) – Optional, a Config object which

Notes

Boot time configuration data is placed in the argument group called config. This adds default dirn, --telepath, --https and --name arguements to the argparser instance. Configuration values which have the hideconf or hidecmdl value set to True are not added to the argparser instance.

Returns

A ArgumentParser for the Cell.

Return type

argparse.ArgumentParser

async getAuthGate(iden)[source]
async getAuthGates()[source]
async getAuthRoles()[source]
async getAuthUsers(archived=False)[source]
async getBackupInfo()[source]

Gets information about recent backup activity

async getBackups()[source]
async getCellApi(link, user, path)[source]
getCellIden()[source]
async getCellInfo()[source]

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns

A Dictionary of metadata.

Return type

Dict

classmethod getCellType()[source]
async getConfOpt(name)[source]
async getDmonSessions()[source]
classmethod getEnvPrefix()[source]
async getHealthCheck()[source]
async getHiveKey(path)[source]

Get the value of a key in the cell default hive

async getHiveKeys(path)[source]

Return a list of (name, value) tuples for nodes under the path.

getLocalProxy(share='*', user='root')[source]
getLocalUrl(share='*', user='root')[source]
async getLogExtra(**kwargs)[source]

Get an extra dictionary for structured logging which can be used as a extra argument for loggers.

Parameters

**kwargs – Additional key/value items to add to the log.

Returns

A dictionary

Return type

Dict

async getNexsIndx()[source]
async getNexusChanges(offs)[source]
async getRoleDef(iden)[source]
async getRoleDefByName(name)[source]
async getRoleDefs()[source]
async getSystemInfo()[source]

Get info about the system in which the cell is running

Returns

  • volsize - Volume where cell is running total space

  • volfree - Volume where cell is running free space

  • backupvolsize - Backup directory volume total space

  • backupvolfree - Backup directory volume free space

  • cellstarttime - Cell start time in epoch milliseconds

  • celluptime - Cell uptime in milliseconds

  • cellrealdisk - Cell’s use of disk, equivalent to du

  • cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

  • osversion - OS version/architecture

  • pyversion - Python version

  • totalmem - Total memory in the system

  • availmem - Available memory in the system

  • cpucount - Number of CPUs on system

Return type

A dictionary with the following keys. All size values are in bytes

async getTeleApi(link, mesg, path)[source]

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async getUserDef(iden)[source]
async getUserDefByName(name)[source]
async getUserDefs()[source]
getUserName(iden, defv='<unknown>')[source]

Translate the user iden to a user name.

async getUserProfInfo(iden, name)[source]
async getUserProfile(iden)[source]
classmethod initCellConf()[source]

Create a Config object for the Cell.

Notes

The Config object has a envar_prefix set according to the results of cls.getEnvPrefix().

Returns

A Config helper object.

Return type

s_config.Config

async classmethod initFromArgv(argv, outp=None)[source]

Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.

Parameters
  • argv (list) – A list of command line arguments to launch the Cell with.

  • outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This does the following items:
  • Create a Config object from the Cell class.

  • Creates an Argument Parser from the Cell class and Config object.

  • Parses the provided arguments.

  • Loads configuration data from the parsed options and environment variables.

  • Sets logging for the process.

  • Creates the Cell from the Cell Ctor.

  • Adds a Telepath listener, HTTPs port listeners and Telepath share names.

  • Returns the Cell.

Returns

This returns an instance of the Cell.

Return type

Cell

async initNexusSubsystem()[source]
async initServiceActive()[source]
async initServiceNetwork()[source]
async initServicePassive()[source]
async initServiceRuntime()[source]
async initServiceStorage()[source]
initSslCtx(certpath, keypath)[source]
async isCellActive()[source]
async isUserAllowed(iden, perm, gateiden=None)[source]
async iterBackupArchive(name, user)[source]
async iterNewBackupArchive(user, name=None, remove=False)[source]
async kill(user, iden)[source]
async listHiveKey(path=None)[source]
async loadHiveTree(tree, path=(), trim=False)[source]

Note: this is for expert emergency use only.

async popHiveKey(path)[source]

Remove and return the value of a key in the cell default hive.

Note: this is for expert emergency use only.

async promote()[source]

Transform this cell from a passive follower to an active cell that writes changes locally.

async ps(user)[source]
async reqGateKeys(gatekeys)[source]
async rotateNexsLog()[source]
async runBackup(name=None, wait=True)[source]
async saveHiveTree(path=())[source]
async setCellActive(active)[source]
async setHiveKey(path, valu)[source]

Set or change the value of a key in the cell default hive

async setNexsIndx(indx)[source]
async setRoleName(iden, name)[source]
async setRoleRules(iden, rules, gateiden=None)[source]
async setUserAdmin(iden, admin, gateiden=None)[source]
async setUserArchived(iden, archived)[source]
async setUserEmail(useriden, email)[source]
async setUserLocked(iden, locked)[source]
async setUserName(useriden, name)[source]
async setUserPasswd(iden, passwd)[source]
async setUserProfInfo(iden, name, valu)[source]
async setUserRoles(useriden, roleidens)[source]
async setUserRules(iden, rules, gateiden=None)[source]
async sync()[source]

no-op mutable for testing purposes. If I am follower, when this returns, I have received and applied all the writes that occurred on the leader before this call.

async trimNexsLog(consumers=None, timeout=30)[source]
async tryUserPasswd(name, passwd)[source]
async waitNexsOffs(offs, timeout=None)[source]
class synapse.lib.cell.CellApi[source]

Bases: synapse.lib.base.Base

addAuthRole(name)[source]
addAuthRule(name, rule, indx=None, gateiden=None)[source]
addRole(name)[source]
addRoleRule(iden, rule, indx=None, gateiden=None)[source]
addUser(name, passwd=None, email=None, iden=None)[source]
addUserRole(useriden, roleiden)[source]
addUserRule(iden, rule, indx=None, gateiden=None)[source]
async allowed(perm, default=None)[source]

Check if the user has the requested permission.

Parameters
  • perm – permission path components to check

  • default – Value returned if no value stored

Examples

Form a path and check the permission from a remote proxy:

perm = ('node', 'add', 'inet:ipv4')
allowed = await prox.allowed(perm)
if allowed:
    dostuff()
Returns

True if the user has permission, False if explicitly denied, None if no entry

Return type

Optional[bool]

cullNexsLog(offs)[source]

Remove Nexus log entries up to (and including) the given offset.

Note

If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.

Only rotated logs where the last index is less than the provided offset will be removed from disk.

Parameters

offs (int) – The offset to remove entries up to.

Returns

Whether the cull was executed

Return type

bool

delAuthRole(name)[source]
delAuthRule(name, rule, gateiden=None)[source]
delAuthUser(name)[source]
delBackup(name)[source]

Delete a backup by name.

Parameters

name (str) – The name of the backup to delete.

delRole(iden)[source]
delRoleRule(iden, rule, gateiden=None)[source]
delUser(iden)[source]
delUserRole(useriden, roleiden)[source]
delUserRule(iden, rule, gateiden=None)[source]
dyncall(iden, todo, gatekeys=())[source]
dyniter(iden, todo, gatekeys=())[source]
getAuthGate(iden)[source]
getAuthGates()[source]
getAuthInfo(name)[source]
getAuthRoles()[source]
getAuthUsers(archived=False)[source]
Parameters

archived (bool) – If true, list all users, else list non-archived users

getBackupInfo()[source]

Get information about recent backup activity.

Returns

  • currduration - If backup currently running, time in ms since backup started, otherwise None

  • laststart - Last time (in epoch milliseconds) a backup started

  • lastend - Last time (in epoch milliseconds) a backup ended

  • lastduration - How long last backup took in ms

  • lastsize - Disk usage of last backup completed

  • lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive

  • lastexception - Tuple of exception information if last backup failed, otherwise None

Return type

(dict) It has the following keys

Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.

getBackups()[source]

Retrieve a list of backups.

Returns

A list of backup names.

Return type

list[str]

getCellIden()[source]
async getCellInfo()[source]
getCellType()[source]
getCellUser()[source]
getDiagInfo()[source]
getDmonSessions()[source]
async getHealthCheck()[source]
getHiveKey(path)[source]
getHiveKeys(path)[source]
getNexsIndx()[source]
getNexusChanges(offs)[source]
getRoleDef(iden)[source]
getRoleDefByName(name)[source]
getRoleDefs()[source]
async getRoleInfo(name)[source]
getSystemInfo()[source]

Get info about the system in which the cell is running

Returns

  • volsize - Volume where cell is running total space

  • volfree - Volume where cell is running free space

  • backupvolsize - Backup directory volume total space

  • backupvolfree - Backup directory volume free space

  • celluptime - Cell uptime in milliseconds

  • cellrealdisk - Cell’s use of disk, equivalent to du

  • cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

  • osversion - OS version/architecture

  • pyversion - Python version

  • totalmem - Total memory in the system

  • availmem - Available memory in the system

Return type

A dictionary with the following keys. All size values are in bytes

getUserDef(iden)[source]
getUserDefByName(name)[source]
getUserDefs()[source]
async getUserInfo(name)[source]
getUserProfInfo(iden, name)[source]
getUserProfile(iden)[source]
async initCellApi()[source]
async isCellActive()[source]

Returns True if the cell is an active/leader cell.

isUserAllowed(iden, perm, gateiden=None)[source]
issue(nexsiden: str, event: str, args, kwargs, meta=None)[source]
iterBackupArchive(name)[source]

Retrieve a backup by name as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters

name (str) – The name of the backup to retrieve.

iterNewBackupArchive(name=None, remove=False)[source]

Run a new backup and return it as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters
  • name (str) – The name of the backup to retrieve.

  • remove (bool) – Delete the backup after streaming.

async kill(iden)[source]
listHiveKey(path=None)[source]
popHiveKey(path)[source]
promote()[source]
async ps()[source]
rotateNexsLog()[source]

Rotate the Nexus log at the current offset.

Returns

The starting index of the active Nexus log

Return type

int

runBackup(name=None, wait=True)[source]

Run a new backup.

Parameters
  • name (str) – The optional name of the backup.

  • wait (bool) – On True, wait for backup to complete before returning.

Returns

The name of the newly created backup.

Return type

str

saveHiveTree(path=())[source]
setAuthAdmin(name, isadmin)[source]
setCellUser(iden)[source]

Switch to another user (admin only).

This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.

setHiveKey(path, valu)[source]
setRoleRules(iden, rules, gateiden=None)[source]
setUserAdmin(iden, admin, gateiden=None)[source]
setUserArchived(useriden, archived)[source]
setUserEmail(useriden, email)[source]
setUserLocked(useriden, locked)[source]
async setUserPasswd(iden, passwd)[source]
setUserProfInfo(iden, name, valu)[source]
setUserRoles(useriden, roleidens)[source]
setUserRules(iden, rules, gateiden=None)[source]
trimNexsLog(consumers=None, timeout=60)[source]

Rotate and cull the Nexus log (and those of any consumers) at the current offset.

Note

If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.

Parameters
  • consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.

  • timeout (int) – Time in seconds to wait for downstream consumers to be caught up.

Returns

The offset that the Nexus log was culled up to and including.

Return type

int

tryUserPasswd(name, passwd)[source]
waitNexsOffs(offs, timeout=None)[source]

Wait for the Nexus log to write an offset.

Parameters
  • offs (int) – The offset to wait for.

  • timeout (int or None) – An optional timeout in seconds.

Returns

True if the offset was written, False if it timed out.

Return type

bool

synapse.lib.cell.SLAB_MAP_SIZE = 134217728

Base classes for the synapse “cell” microservice architecture.

synapse.lib.cell.adminapi(log=False)[source]

Decorator for CellApi (and subclasses) for requiring a method to be called only by an admin user.

Parameters

log (bool) – If set to True, log the user, function and arguments.

synapse.lib.certdir module

class synapse.lib.certdir.CertDir(path=None)[source]

Bases: object

Certificate loading/generation/signing utilities.

Features:
  • Locates and load certificates, keys, and certificate signing requests (CSRs).

  • Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.

  • Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).

  • Signs certificate signing requests (CSRs).

  • Generates PKCS#12 archives for use in browser.

Parameters

path (str) – Optional path which can override the default path directory.

Notes

  • All certificates will be loaded from and written to ~/.syn/certs by default. Set the envvar SYN_CERT_DIR to override.

  • All certificate generation methods create 4096 bit RSA keypairs.

  • All certificate signing methods use sha256 as the signature algorithm.

  • CertDir does not currently support signing CA CSRs.

addCertPath(*path)[source]
delCertPath(*path)[source]
genCaCert(name, signas=None, outp=None, save=True)[source]

Generates a CA keypair.

Parameters
  • name (str) – The name of the CA keypair.

  • signas (str) – The CA keypair to sign the new CA with.

  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make a CA named “myca”:

mycakey, mycacert = cdir.genCaCert(‘myca’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genClientCert(name, outp=None)[source]

Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material.

Parameters
  • name (str) – The name of the user keypair.

  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”:

myuserpkcs12 = cdir.genClientCert(‘myuser’)

Returns

The PKCS #12 archive.

Return type

OpenSSL.crypto.PKCS12

genHostCert(name, signas=None, outp=None, csr=None, sans=None)[source]

Generates a host keypair.

Parameters
  • name (str) – The name of the host keypair.

  • signas (str) – The CA keypair to sign the new host keypair with.

  • outp (synapse.lib.output.Output) – The output buffer.

  • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

  • sans (list) – List of subject alternative names.

Examples

Make a host keypair named “myhost”:

myhostkey, myhostcert = cdir.genHostCert(‘myhost’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genHostCsr(name, outp=None)[source]

Generates a host certificate signing request.

Parameters
  • name (str) – The name of the host CSR.

  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the host key named “myhost”:

cdir.genHostCsr(‘myhost’)

Returns

The bytes of the CSR.

Return type

bytes

genUserCert(name, signas=None, outp=None, csr=None)[source]

Generates a user keypair.

Parameters
  • name (str) – The name of the user keypair.

  • signas (str) – The CA keypair to sign the new user keypair with.

  • outp (synapse.lib.output.Output) – The output buffer.

  • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”:

myuserkey, myusercert = cdir.genUserCert(‘myuser’)

Returns

Tuple containing the key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genUserCsr(name, outp=None)[source]

Generates a user certificate signing request.

Parameters
  • name (str) – The name of the user CSR.

  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the user “myuser”:

cdir.genUserCsr(‘myuser’)

Returns

The bytes of the CSR.

Return type

bytes

getCaCert(name)[source]

Loads the X509 object for a given CA.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the certificate for the CA “myca”

mycacert = cdir.getCaCert(‘myca’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getCaCertPath(name)[source]

Gets the path to a CA certificate.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the CA certificate for the CA “myca”:

mypath = cdir.getCACertPath(‘myca’)

Returns

The path if exists.

Return type

str

getCaCerts()[source]

Return a list of CA certs from the CertDir.

Returns

List of CA certificates.

Return type

[OpenSSL.crypto.X509]

getCaKey(name)[source]

Loads the PKey object for a given CA keypair.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the private key for the CA “myca”:

mycakey = cdir.getCaKey(‘myca’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getCaKeyPath(name)[source]

Gets the path to a CA key.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”:

mypath = cdir.getCAKeyPath(‘myca’)

Returns

The path if exists.

Return type

str

getClientCert(name)[source]

Loads the PKCS12 archive object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”:

mypkcs12 = cdir.getClientCert(‘myuser’)

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool

Returns

The PKCS12 archive, if exists.

Return type

OpenSSL.crypto.PKCS12

getClientCertPath(name)[source]

Gets the path to a client certificate.

Parameters

name (str) – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”:

mypath = cdir.getClientCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getClientSSLContext(certname=None)[source]

Returns an ssl.SSLContext appropriate for initiating a TLS session

Parameters

certname – If specified, use the user certificate with the matching name to authenticate to the remote service.

getHostCaPath(name)[source]

Gets the path to the CA certificate that issued a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”:

mypath = cdir.getHostCaPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostCert(name)[source]

Loads the X509 object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”:

myhostcert = cdir.getHostCert(‘myhost’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getHostCertPath(name)[source]

Gets the path to a host certificate.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”:

mypath = cdir.getHostCertPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostKey(name)[source]

Loads the PKey object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the private key object for the host “myhost”:

myhostkey = cdir.getHostKey(‘myhost’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getHostKeyPath(name)[source]

Gets the path to a host key.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”:

mypath = cdir.getHostKeyPath(‘myhost’)

Returns

The path if exists.

Return type

str

getServerSSLContext(hostname=None, caname=None)[source]

Returns an ssl.SSLContext appropriate to listen on a socket

Parameters
  • hostname – if None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory

  • caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.

getUserCaPath(name)[source]

Gets the path to the CA certificate that issued a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the CA cert which issue the cert for “myuser”:

mypath = cdir.getUserCaPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserCert(name)[source]

Loads the X509 object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”:

myusercert = cdir.getUserCert(‘myuser’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getUserCertPath(name)[source]

Gets the path to a user certificate.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”:

mypath = cdir.getUserCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserForHost(user, host)[source]

Gets the name of the first existing user cert for a given user and host.

Parameters
  • user (str) – The name of the user.

  • host (str) – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”:

usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)

Returns

The cert name, if exists.

Return type

str

getUserKey(name)[source]

Loads the PKey object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”:

myuserkey = cdir.getUserKey(‘myuser’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getUserKeyPath(name)[source]

Gets the path to a user key.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the user key for “myuser”:

mypath = cdir.getUserKeyPath(‘myuser’)

Returns

The path if exists.

Return type

str

importFile(path, mode, outp=None)[source]

Imports certs and keys into the Synapse cert directory

Parameters
  • path (str) – The path of the file to be imported.

  • mode (str) – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory.

certdir.importFile(‘mycoolca.crt’, ‘cas’)

Notes

importFile does not perform any validation on the files it imports.

Returns

None

isCaCert(name)[source]

Checks if a CA certificate exists.

Parameters

name (str) – The name of the CA keypair.

Examples

Check if the CA certificate for “myca” exists:

exists = cdir.isCaCert(‘myca’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isClientCert(name)[source]

Checks if a user client certificate (PKCS12) exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists:

exists = cdir.isClientCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isHostCert(name)[source]

Checks if a host certificate exists.

Parameters

name (str) – The name of the host keypair.

Examples

Check if the host cert “myhost” exists:

exists = cdir.isUserCert(‘myhost’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isUserCert(name)[source]

Checks if a user certificate exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the user cert “myuser” exists:

exists = cdir.isUserCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

saveCertPem(cert, path)[source]

Save a certificate in PEM format to a file outside the certdir.

savePkeyPem(pkey, path)[source]

Save a private key in PEM format to a file outside the certdir.

selfSignCert(cert, pkey)[source]

Self-sign a certificate.

Parameters
  • cert (OpenSSL.crypto.X509) – The certificate to sign.

  • pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key:

cdir.selfSignCert(mycert, myotherprivatekey)

Returns

None

signCertAs(cert, signas)[source]

Signs a certificate with a CA keypair.

Parameters
  • cert (OpenSSL.crypto.X509) – The certificate to sign.

  • signas (str) – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”:

cdir.signCertAs(mycert, ‘myca’)

Returns

None

signHostCsr(xcsr, signas, outp=None, sans=None)[source]

Signs a host CSR with a CA keypair.

Parameters
  • cert (OpenSSL.crypto.X509Req) – The certificate signing request.

  • signas (str) – The CA keypair name to sign the CSR with.

  • outp (synapse.lib.output.Output) – The output buffer.

  • sans (list) – List of subject alternative names.

Examples

Sign a host key with the CA “myca”:

cdir.signHostCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

signUserCsr(xcsr, signas, outp=None)[source]

Signs a user CSR with a CA keypair.

Parameters
  • cert (OpenSSL.crypto.X509Req) – The certificate signing request.

  • signas (str) – The CA keypair name to sign the CSR with.

  • outp (synapse.lib.output.Output) – The output buffer.

Examples

cdir.signUserCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

valUserCert(byts, cacerts=None)[source]

Validate the PEM encoded x509 user certificate bytes and return it.

Parameters
  • byts (bytes) – The bytes for the User Certificate.

  • cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates.

Raises

OpenSSL.crypto.X509StoreContextError – If the certificate is not valid.

Returns

The certificate, if it is valid.

Return type

OpenSSL.crypto.X509

synapse.lib.certdir.addCertPath(path)[source]
synapse.lib.certdir.delCertPath(path)[source]
synapse.lib.certdir.getCertDir()[source]
synapse.lib.certdir.iterFqdnUp(fqdn)[source]

synapse.lib.chop module

synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)

Shared primitive routines for chopping up strings and values.

synapse.lib.chop.digits(text)[source]
synapse.lib.chop.hexstr(text)[source]

Ensure a string is valid hex.

Parameters

text (str) – String to normalize.

Examples

Norm a few strings:

hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.

Returns

Normalized hex string.

Return type

str

synapse.lib.chop.intstr(text)[source]
synapse.lib.chop.onespace(text)[source]
synapse.lib.chop.printables(text)[source]
synapse.lib.chop.stormstring(s)[source]

Make a string storm safe by escaping backslashes and double quotes.

Parameters

s (str) – String to make storm safe.

Notes

This does not encapsulate a string in double quotes.

Returns

A string which can be embedded directly into a storm query.

Return type

str

synapse.lib.chop.tag(text)[source]
synapse.lib.chop.tagpath(text)[source]
synapse.lib.chop.tags(norm)[source]

Divide a normalized tag string into hierarchical layers.

synapse.lib.chop.validateTagMatch(tag)[source]

Raises an exception if tag is not a valid tagmatch (i.e. a tag that might have globs)

synapse.lib.cli module

class synapse.lib.cli.Cli[source]

Bases: synapse.lib.base.Base

A modular / event-driven CLI base object.

addCmdClass(ctor, **opts)[source]

Add a Cmd subclass to this cli.

async addSignalHandlers()[source]

Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.

get(name, defval=None)[source]
getCmdByName(name)[source]

Return a Cmd instance by name.

getCmdNames()[source]

Return a list of all the known command names for the CLI.

getCmdPrompt()[source]

Get the command prompt.

Returns

Configured command prompt

Return type

str

histfile = 'cmdr_history'
initCmdClasses()[source]
printf(mesg, addnl=True, color=None)[source]
async prompt(text=None)[source]

Prompt for user input from stdin.

async runCmdLine(line)[source]

Run a single command line.

Parameters

line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch:

await cli.runCmdLine(‘woot –help’)

Returns

Arbitrary data from the cmd class.

Return type

object

async runCmdLoop()[source]

Run commands from a user in an interactive fashion until fini() or EOFError is raised.

set(name, valu)[source]
class synapse.lib.cli.Cmd(cli, **opts)[source]

Bases: object

Base class for modular commands in the synapse CLI.

getCmdBrief()[source]

Return the single-line description for this command.

getCmdDoc()[source]

Return the help/doc output for this command.

getCmdItem()[source]

Get a reference to the object we are commanding.

getCmdName()[source]
getCmdOpts(text)[source]

Use the _cmd_syntax def to split/parse/normalize the cmd line.

Parameters

text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type

Returns

An opts dictionary.

Return type

dict

printf(mesg, addnl=True, color=None)[source]
async runCmdLine(line)[source]

Run a line of command input for this command.

Parameters

line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdHelp(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

List commands and display help output.

Example

help foocmd

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdLocals(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

List the current locals for a given CLI object.

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdQuit(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

Quit the current command line interpreter.

Example

quit

async runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

synapse.lib.cmd module

class synapse.lib.cmd.Parser(prog=None, outp=<synapse.lib.output.OutPut object>, **kwargs)[source]

Bases: argparse.ArgumentParser

exit(status=0, message=None)[source]

Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead.

synapse.lib.cmdr module

async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)[source]

Construct and return a cmdr for the given remote cell.

Parameters
  • cell – Cell proxy being commanded.

  • outp – Output helper object.

  • color (bool) – If true, enable colorized output.

  • **opts – Additional options pushed into the Cmdr locs.

Examples

Get the cmdr for a proxy:

cmdr = await getItemCmdr(foo)
Returns

A Cli instance with Cmds loaeded into it.

Return type

s_cli.Cli

async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)[source]

Create a cmdr for the given item and run the cmd loop.

Parameters
  • item – Cell proxy being commanded.

  • outp – Output helper object.

  • color (bool) – If true, enable colorized output.

  • **opts – Additional options pushed into the Cmdr locs.

Notes

This function does not return while the command loop is run.

Examples

Run the Cmdr for a proxy:

await runItemCmdr(foo)
Returns

This function returns None.

Return type

None

synapse.lib.config module

class synapse.lib.config.Config(schema, conf=None, envar_prefix=None)[source]

Bases: collections.abc.MutableMapping

Synapse configuration helper based on JSON Schema.

Parameters
  • schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.

  • conf (dict) – Optional, a set of configuration data to preload.

  • envar_prefix (str) – Optional, a prefix used when collecting configuration data from environment variables.

Notes

This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.

The default values provided in the schema must be able to be recreated from the repr() of their Python value.

Default values are not loaded into the configuration data until the reqConfValid() method is called.

asDict()[source]

Get a copy of configuration data.

Returns

A copy of the configuration data.

Return type

dict

getArgParseArgs()[source]
getCmdlineMapping()[source]
classmethod getConfFromCell(cell, conf=None, envar_prefix=None)[source]

Get a Config object from a Cell directly (either the ctor or the instance thereof).

Returns

A Config object.

Return type

Config

getEnvarMapping()[source]

Get a mapping of config values to envars.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

reqConfValid()[source]

Validate that the loaded configuration data is valid according to the schema.

Notes

The validation set does set any default values which are not currently set for configuration options.

Returns

This returns nothing.

Return type

None

reqConfValu(key)[source]

Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.

Parameters

key (str) – The key to require.

Returns

The requested value.

setConfFromEnvs()[source]

Set configuration options from environment variables.

Notes

Environment variables are resolved from configuration options after doing the following transform:

  • Replace : characters with _.

  • Add a config provided prefix, if set.

  • Uppercase the string.

  • Resolve the environment variable

  • If the environment variable is set, set the config value to the results of yaml.yaml_safeload() on the value.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

Examples

For the configuration value auth:passwd, the environment variable is resolved as AUTH_PASSWD. With the prefix cortex, the the environment variable is resolved as CORTEX_AUTH_PASSWD.

Returns

Returns None.

Return type

None

setConfFromOpts(opts)[source]

Set the opts for a conf object from a namespace object.

Parameters
  • opts (argparse.Namespace) – A Namespace object made from parsing args with an ArgumentParser

  • getArgumentParser. (made with) –

Returns

Returns None.

Return type

None

synapse.lib.config.getJsSchema(confbase, confdefs)[source]

Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.

Parameters
  • confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.

  • confdefs (dict) – A JSON Schema dictionary of properties for the object.

Notes

This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify

Returns

A complete JSON schema.

Return type

dict

synapse.lib.config.getJsValidator(schema)[source]

Get a fastjsonschema callable.

Parameters

schema (dict) – A JSON Schema object.

Returns

A callable function that can be used to validate data against the json schema.

Return type

callable

synapse.lib.config.make_envar_name(key, prefix=None)[source]

Convert a colon delimited string into an uppercase, underscore delimited string.

Parameters
  • key (str) – Config key to convert.

  • prefix (str) – Optional string prefix to prepend the the config key.

Returns

The string to lookup against a envar.

Return type

str

synapse.lib.const module

synapse.lib.coro module

Async/Coroutine related utilities.

class synapse.lib.coro.Event(*, loop=None)[source]

Bases: asyncio.locks.Event

async timewait(timeout=None)[source]
class synapse.lib.coro.GenrHelp(genr)[source]

Bases: object

async list()[source]
async spin()[source]
async synapse.lib.coro.agen(item)[source]

Wrap an async_generator or generator in an async_generator.

Notes

Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.

async synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None)[source]

Wait on an an asyncio event with an optional timeout

Returns

true if the event got set, False if timed out

synapse.lib.coro.executor(func, *args, **kwargs)[source]

Execute a non-coroutine function in the ioloop executor pool.

Parameters
  • func – Function to execute.

  • *args – Args for the function.

  • **kwargs – Kwargs for the function.

Examples

Execute a blocking API call in the executor pool:

import requests

def block(url, params=None):
    return requests.get(url, params=params).json()

fut = s_coro.executor(block, 'http://some.tld/thign')
resp = await fut
Returns

An asyncio future.

Return type

asyncio.Future

async synapse.lib.coro.forked(func, *args, **kwargs)[source]

Execute a target function in the forked process pool.

Parameters
  • func – The target function.

  • *args – Function positional arguments.

  • **kwargs – Function keyword arguments.

Returns

The target function return.

Raises
  • The function may raise from the target function, or raise a s_exc.FatalErr in the event of a broken forked

  • process pool. The fatalerr represents a unrecoverable application state.

synapse.lib.coro.genrhelp(f)[source]
synapse.lib.coro.iscoro(item)[source]
async synapse.lib.coro.ornot(func, *args, **kwargs)[source]

Calls func and awaits it if a returns a coroutine.

Note

This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.

This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage:

ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)

synapse.lib.coro.set_pool_logging(logger_, logconf)[source]
async synapse.lib.coro.spawn(todo, timeout=None, ctx=None)[source]

Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.

Parameters
  • todo (tuple) – A tuple of function, *args, and **kwargs.

  • timeout (int) – The timeout to wait for the todo function to finish.

  • ctx (multiprocess.Context) – A optional multiprocessing context object.

Notes

The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.

Returns

The return value of executing the todo function.

async synapse.lib.coro.waittask(task, timeout=None)[source]

Await a task without cancelling it when you time out.

Returns

True if the task completed before the timeout.

Return type

boolean

synapse.lib.datfile module

Utilities for handling data files embedded within python packages.

synapse.lib.datfile.openDatFile(datpath)[source]

Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.dyndeps module

synapse.lib.dyndeps.getDynLocal(name)[source]

Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()

synapse.lib.dyndeps.getDynMeth(name)[source]

Retrieve and return an unbound method by python path.

synapse.lib.dyndeps.getDynMod(name)[source]

Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’)

synapse.lib.dyndeps.runDynTask(task)[source]

Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)[source]

Dynamically import a module and call a function or raise an exception.

synapse.lib.dyndeps.tryDynLocal(name)[source]

Dynamically import a module and return a module local or raise an exception.

synapse.lib.dyndeps.tryDynMod(name)[source]

Dynamically import a python module or exception.

synapse.lib.encoding module

synapse.lib.encoding.addFormat(name, fn, opts)[source]

Add an additional ingest file format

synapse.lib.encoding.decode(name, byts, **opts)[source]

Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with +

to encode for that name/layer.

synapse.lib.encoding.encode(name, item, **opts)[source]
synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)[source]

Iterate through the data provided by a file like object.

Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd:
    for row in iterdata(fd, format='csv', encoding='utf8'):
        dostuff(row)
Parameters
  • fd (file) – File like object to iterate over.

  • close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.

  • **opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.

Yields

An item to process. The type of the item is dependent on the format parameters.

synapse.lib.gis module

synapse.lib.gis.bbox(lat, lon, dist)[source]

Calculate a min/max bounding box for the circle defined by lalo/dist.

Parameters
  • lat (float) – The latitude in degrees

  • lon (float) – The longitude in degrees

  • dist (int) – A distance in geo:dist base units (mm)

Returns

(latmin, latmax, lonmin, lonmax)

Return type

(float,float,float,float)

synapse.lib.gis.dms2dec(degs, mins, secs)[source]

Convert degrees, minutes, seconds lat/long form to degrees float.

Parameters
  • degs (int) – Degrees

  • mins (int) – Minutes

  • secs (int) – Seconds

Returns

Degrees

Return type

(float)

synapse.lib.gis.haversine(px, py, r=6371008800.0)[source]

Calculate the haversine distance between two points defined by (lat,lon) tuples.

Parameters
  • px ((float,float)) – lat/long position 1

  • py ((float,float)) – lat/long position 2

  • r (float) – Radius of sphere

Returns

Distance in mm.

Return type

(int)

synapse.lib.gis.latlong(text)[source]

Chop a latlong string and return (float,float). Does not perform validation on the coordinates.

Parameters

text (str) – A longitude,latitude string.

Returns

A longitude, latitude float tuple.

Return type

(float,float)

synapse.lib.gis.near(point, dist, points)[source]

Determine if the given point is within dist of any of points.

Parameters
  • point ((float,float)) – A latitude, longitude float tuple.

  • dist (int) – A distance in mm ( base units )

  • points (list) – A list of latitude, longitude float tuples to compare against.

synapse.lib.grammar module

synapse.lib.grammar.isBasePropNoPivprop(name)[source]
synapse.lib.grammar.isCmdName(name)[source]
synapse.lib.grammar.isFormName(name)[source]
synapse.lib.grammar.isPropName(name)[source]
synapse.lib.grammar.isUnivName(name)[source]
synapse.lib.grammar.meh(txt, off, cset)[source]
synapse.lib.grammar.nom(txt, off, cset, trim=True)[source]

Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’)

name,off = nom(text,0,chars)

Note:

This really shouldn’t be used for new code

synapse.lib.grammar.parse_float(text, off)[source]

synapse.lib.hashitem module

synapse.lib.hashitem.hashitem(item)[source]

Generate a uniq hash for the JSON compatible primitive data structure.

synapse.lib.hashitem.normdict(item)[source]
synapse.lib.hashitem.normitem(item)[source]
synapse.lib.hashitem.normiter(item)[source]

synapse.lib.hashset module

class synapse.lib.hashset.HashSet[source]

Bases: object

digests()[source]

Get a list of (name, bytes) tuples for the hashes in the hashset.

eatfd(fd)[source]

Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd)

guid()[source]

Use elements from this hash set to create a unique (re)identifier.

update(byts)[source]

Update all the hashes in the set with the given bytes.

synapse.lib.health module

class synapse.lib.health.HealthCheck(iden)[source]

Bases: object

getStatus()[source]
pack()[source]
setStatus(valu)[source]
update(name, status, mesg='', data=None)[source]

Append a new component to the Healcheck object.

Parameters
  • name (str) – Name of the reported component.

  • status (str) – nomdinal/degraded/failed status code.

  • mesg (str) – Optional message about the component status.

  • data (dict) – Optional arbitrary dictionary of additional metadata about the component.

Returns

None

synapse.lib.hive module

class synapse.lib.hive.Hive[source]

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.

async add(full, valu)[source]

Atomically increments a node’s value.

async dict(full, nexs=False)[source]

Open a HiveDict at the given full path.

Parameters

full (tuple) – A full path tuple.

Returns

A HiveDict for the full path.

Return type

HiveDict

dir(full)[source]

List subnodes of the given Hive path.

Parameters

full (tuple) – A full path tuple.

Notes

This returns None if there is not a node at the path.

Returns

A list of tuples. Each tuple contains the name, node value, and the number of children nodes.

Return type

list

async exists(full)[source]

Returns whether the Hive path has already been created.

async get(full, defv=None)[source]

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async getHiveAuth()[source]

Retrieve a HiveAuth for hive standalone or non-cell uses.

Note

This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell

async getTeleApi(link, mesg, path)[source]

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async loadHiveTree(tree, path=(), trim=False)[source]
async open(full)[source]

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(full, nexs=False)[source]

Remove and return the value for the given node.

async rename(oldpath, newpath)[source]

Moves a node at oldpath and all its descendant nodes to newpath. newpath must not exist

async saveHiveTree(path=())[source]
async set(full, valu, nexs=False)[source]

A set operation at the hive level (full path).

async storNodeDele(path)[source]
async storNodeValu(full, valu)[source]
class synapse.lib.hive.HiveApi[source]

Bases: synapse.lib.base.Base

async addAndSync(path, valu, iden)[source]
async edits()[source]
async get(full)[source]
async loadHiveTree(tree, path=(), trim=False)[source]
async popAndSync(path, iden, nexs=False)[source]
async saveHiveTree(path=())[source]
async setAndSync(path, valu, iden, nexs=False)[source]
async treeAndSync(path, iden)[source]
class synapse.lib.hive.HiveDict[source]

Bases: synapse.lib.base.Base

get(name, default=None)[source]
items()[source]
pack()[source]
async pop(name, default=None)[source]
async set(name, valu)[source]
setdefault(name, valu)[source]
values()[source]
class synapse.lib.hive.Node[source]

Bases: synapse.lib.base.Base

A single node within the Hive tree.

async add(valu)[source]

Increments existing node valu

async dict(nexs=False)[source]

Get a HiveDict for this Node.

Returns

A HiveDict for this Node.

Return type

HiveDict

dir()[source]
get(name)[source]
name()[source]
async open(path)[source]

Open a child Node of the this Node.

Parameters

path (tuple) – A child path of the current node.

Returns

A Node at the child path.

Return type

Node

parent()[source]
async pop(path=())[source]
async set(valu)[source]
class synapse.lib.hive.SlabHive[source]

Bases: synapse.lib.hive.Hive

async storNodeDele(full)[source]
async storNodeValu(full, valu)[source]
class synapse.lib.hive.TeleHive[source]

Bases: synapse.lib.hive.Hive

A Hive that acts as a consistent read cache for a telepath proxy Hive

async add(path, valu)[source]

Atomically increments a node’s value.

async get(path)[source]

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async open(path)[source]

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(path, nexs=False)[source]

Remove and return the value for the given node.

async set(path, valu, nexs=False)[source]

A set operation at the hive level (full path).

synapse.lib.hive.iterpath(path)[source]
async synapse.lib.hive.opendir(dirn, conf=None)[source]
async synapse.lib.hive.openurl(url, **opts)[source]

synapse.lib.hiveauth module

class synapse.lib.hiveauth.Auth[source]

Bases: synapse.lib.nexus.Pusher

Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.

Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).

Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.

Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.

Node layout:

Auth root (passed into constructor)
├ roles
│   ├ <role iden 1>
│   ├ ...
│   └ last role
├ users
│   ├ <user iden 1>
│   ├ ...
│   └ last user
└ authgates
    ├ <iden 1>
    │   ├ roles
    │   │   ├ <role iden 1>
    │   │   ├ ...
    │   │   └ last role
    │   └ users
    │       ├ <user iden 1>
    │       ├ ...
    │       └ last user
    ├ <iden 2>
    │   ├ ...
    └ ... last authgate
async addAuthGate(iden, authgatetype)[source]

Retrieve AuthGate by iden. Create if not present.

Note

Not change distributed

Returns

(HiveAuthGate)

async addRole(name)[source]
async addUser(name, passwd=None, email=None, iden=None)[source]

Add a User to the Hive.

Parameters
  • name (str) – The name of the User.

  • passwd (str) – A optional password for the user.

  • email (str) – A optional email for the user.

  • iden (str) – A optional iden to use as the user iden.

Returns

A Hive User.

Return type

HiveUser

async delAuthGate(iden)[source]

Delete AuthGate by iden.

Note

Not change distributed

async delRole(iden)[source]
async delUser(iden)[source]
getAuthGate(iden)[source]
getAuthGates()[source]
async getRoleByName(name)[source]
async getUserByName(name)[source]

Get a user by their username.

Parameters

name (str) – Name of the user to get.

Returns

A Hive User. May return None if there is no user by the requested name.

Return type

HiveUser

async getUserIdenByName(name)[source]
reqAuthGate(iden)[source]
async reqRole(iden)[source]
async reqRoleByName(name)[source]
async reqUser(iden)[source]
async reqUserByName(name)[source]
async reqUserByNameOrIden(name)[source]
role(iden)[source]
roles()[source]
async setRoleInfo(iden, name, valu, gateiden=None)[source]
async setRoleName(iden, name)[source]
async setUserInfo(iden, name, valu, gateiden=None)[source]
async setUserName(iden, name)[source]
user(iden)[source]
users()[source]
class synapse.lib.hiveauth.AuthGate[source]

Bases: synapse.lib.base.Base

The storage object for object specific rules for users/roles.

async delete()[source]
async genRoleInfo(iden)[source]
async genUserInfo(iden)[source]
pack()[source]
class synapse.lib.hiveauth.HiveRole[source]

Bases: synapse.lib.hiveauth.HiveRuler

A role within the Hive authorization subsystem.

A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.

clearAuthCache()[source]
async genGateInfo(gateiden)[source]
pack()[source]
async setName(name)[source]
class synapse.lib.hiveauth.HiveRuler[source]

Bases: synapse.lib.base.Base

A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those

async addRule(rule, indx=None, gateiden=None, nexs=True)[source]
async delRule(rule, gateiden=None)[source]
getRules(gateiden=None)[source]
async setRules(rules, gateiden=None, nexs=True)[source]
class synapse.lib.hiveauth.HiveUser[source]

Bases: synapse.lib.hiveauth.HiveRuler

A user (could be human or computer) of the system within HiveAuth.

Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.

allowed(perm, default=None, gateiden=None)[source]
clearAuthCache()[source]
confirm(perm, default=None, gateiden=None)[source]
async genGateInfo(gateiden)[source]
getRoles()[source]
async grant(roleiden, indx=None)[source]
hasRole(iden)[source]
isAdmin(gateiden=None)[source]
isLocked()[source]
pack(packroles=False)[source]
raisePermDeny(perm, gateiden=None)[source]
async revoke(iden, nexs=True)[source]
async setAdmin(admin, gateiden=None, logged=True)[source]
async setArchived(archived)[source]
async setLocked(locked, logged=True)[source]
async setName(name)[source]
async setPasswd(passwd, nexs=True)[source]
async setRoles(roleidens)[source]

Replace all the roles for a given user with a new list of roles.

Parameters

roleidens (list) – A list of roleidens.

Notes

The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.

Returns

None

async tryPasswd(passwd)[source]
synapse.lib.hiveauth.getShadow(passwd)[source]

synapse.lib.httpapi module

class synapse.lib.httpapi.ActiveV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.AuthAddRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.AuthAddUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.AuthDelRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.AuthGrantV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()[source]
async post()[source]
class synapse.lib.httpapi.AuthRevokeV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()[source]
async post()[source]
class synapse.lib.httpapi.AuthRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get(iden)[source]
async post(iden)[source]
class synapse.lib.httpapi.AuthRolesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.AuthUserPasswdV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post(iden)[source]
class synapse.lib.httpapi.AuthUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get(iden)[source]
async post(iden)[source]
class synapse.lib.httpapi.AuthUsersV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.CoreInfoV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/core/info

async get()[source]
class synapse.lib.httpapi.FeedV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/feed

{

‘name’: ‘syn.nodes’, ‘view’: null, ‘items’: […],

}

async post()[source]
class synapse.lib.httpapi.Handler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

on_connection_close()[source]

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

prepare()[source]

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

class synapse.lib.httpapi.HandlerBase[source]

Bases: object

async allowed(perm, gateiden=None)[source]

Return true if there is a logged in user with the given permissions.

NOTE: This API sets up HTTP response values if it returns False.

NOTE: This API uses the Handler.getAuthCell() abstraction and is safe for use

in split-auth cells.

async authenticated()[source]
check_origin(origin)[source]
getAuthCell()[source]

Return a reference to the cell used for auth operations.

getCustomHeaders()[source]
getJsonBody(validator=None)[source]
async getUserBody()[source]

Helper function to confirm that there is a auth user and a valid JSON body in the request.

Returns

The user and body of the request as deserialized JSON, or a tuple of s_common.novalu

objects if there was no user or json body.

Return type

(User, object)

initialize(cell)[source]
isOrigHost(origin)[source]
loadJsonMesg(byts, validator=None)[source]
options()[source]
async reqAuthAdmin()[source]
async reqAuthUser()[source]
sendAuthRequired()[source]
sendRestErr(code, mesg)[source]
sendRestExc(e)[source]
sendRestRetn(valu)[source]
async sess(gen=True)[source]
set_default_headers()[source]
async user()[source]
async useriden()[source]

Return the user iden of the current session user.

NOTE: APIs should migrate toward using this rather than the heavy

Handler.user() API to facilitate reuse of handler objects with telepath references.

class synapse.lib.httpapi.HealthCheckV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.LoginV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.ModelNormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
async post()[source]
class synapse.lib.httpapi.ModelV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.OnePassIssueV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/onepass/issue

async post()[source]
class synapse.lib.httpapi.ReqValidStormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.StormHandler

async get()[source]
async post()[source]
class synapse.lib.httpapi.RobotHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

async get()[source]
class synapse.lib.httpapi.Sess[source]

Bases: synapse.lib.base.Base

addWebSock(sock)[source]
delWebSock(sock)[source]
async login(user)[source]
async logout()[source]
class synapse.lib.httpapi.StormCallV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.StormHandler

async get()[source]
async post()[source]
class synapse.lib.httpapi.StormExportV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.StormHandler

async get()[source]
async post()[source]
class synapse.lib.httpapi.StormHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

getCore()[source]
class synapse.lib.httpapi.StormNodesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.StormHandler

async get()[source]
async post()[source]
class synapse.lib.httpapi.StormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.StormHandler

async get()[source]
async post()[source]
class synapse.lib.httpapi.StormVarsGetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async get()[source]
class synapse.lib.httpapi.StormVarsPopV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.StormVarsSetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

async post()[source]
class synapse.lib.httpapi.StreamHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.Handler

Subclass for Tornado streaming uploads.

Notes

  • Async method prepare() is called after headers are read but before body processing.

  • Sync method on_finish() can be used to cleanup after a request.

  • Sync method on_connection_close() can be used to cleanup after a client disconnect.

  • Async methods post(), put(), etc are called after the streaming has completed.

async data_received(chunk)[source]

Implement this method to handle streamed request data.

Requires the .stream_request_body decorator.

May be a coroutine for flow control.

class synapse.lib.httpapi.WatchSockV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.WebSocket

A web-socket based API endpoint for distributing cortex events.

async onWatchMesg(byts)[source]
async on_message(byts)[source]

Handle incoming messages on the WebSocket

This method must be overridden.

Changed in version 4.5: on_message can be a coroutine.

class synapse.lib.httpapi.WebSocket(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.websocket.WebSocketHandler

async xmit(name, **info)[source]

synapse.lib.ingest module

synapse.lib.interval module

A few utilities for dealing with intervals.

synapse.lib.interval.fold(*vals)[source]

Initialize a new (min,max) tuple interval from values.

Parameters

*vals ([int,...]) – A list of values (or Nones)

Returns

A (min,max) interval tuple or None

Return type

((int,int))

synapse.lib.interval.overlap(ival0, ival1)[source]

Determine if two interval tuples have overlap.

Parameters
  • iv0 ((int,int)) – An interval tuple

  • iv1 ((int,int)) –

Returns

True if the intervals overlap, otherwise False

Return type

(bool)

synapse.lib.interval.parsetime(text)[source]

Parse an interval time string and return a (min,max) tuple.

Parameters

text (str) – A time interval string

Returns

A epoch millis epoch time string

Return type

((int,int))

synapse.lib.jsonstor module

class synapse.lib.jsonstor.JsonStor[source]

Bases: synapse.lib.base.Base

A filesystem like storage mechanism that allows hirarchical lookup of reference counted “objects” that have individually editable properties.

#TODO json validation by path glob matches? (persists?) #TODO GUID ACCESS with index generation by type #TODO registered types jsonschema with optional write-back validation

async cmpDelPathObjProp(path, prop, valu)[source]
async copyPathObj(oldp, newp)[source]
async copyPathObjs(paths)[source]
async delPathObj(path)[source]

Remove a path and decref the object it references.

async delPathObjProp(path, prop)[source]
async getPathList(path)[source]
async getPathObj(path)[source]
async getPathObjProp(path, prop)[source]
async getPathObjs(path)[source]
async hasPathObj(path)[source]
async popPathObjProp(path, prop, defv=None)[source]

Add a link from the given srcpath to the dstpath. NOTE: This causes the item at dstpath to be incref’d

async setPathObj(path, item)[source]

Set (and/or reinitialize) the object at the given path.

NOTE: This will break any links by creating a new object.

async setPathObjProp(path, prop, valu)[source]
class synapse.lib.jsonstor.JsonStorApi[source]

Bases: synapse.lib.cell.CellApi

async addQueue(name, info)[source]
async cmpDelPathObjProp(path, name, valu)[source]
async copyPathObj(oldp, newp)[source]
async copyPathObjs(paths)[source]
async cullQueue(name, offs)[source]
async delPathObj(path)[source]
async delPathObjProp(path, name)[source]
async delQueue(name)[source]
async getPathList(path)[source]
async getPathObj(path)[source]
async getPathObjProp(path, prop)[source]
async getPathObjs(path)[source]
async getsQueue(name, offs, size=None, cull=True, wait=True)[source]
async hasPathObj(path)[source]
async popPathObjProp(path, prop)[source]
async putsQueue(name, items)[source]
async setPathObj(path, item)[source]
async setPathObjProp(path, prop, valu)[source]
class synapse.lib.jsonstor.JsonStorCell[source]

Bases: synapse.lib.cell.Cell

async addQueue(name, info)[source]
cellapi

alias of synapse.lib.jsonstor.JsonStorApi

async cmpDelPathObjProp(path, name, valu)[source]
async copyPathObj(oldp, newp)[source]
async copyPathObjs(paths)[source]
async cullQueue(name, offs)[source]
async delPathObj(path)[source]
async delPathObjProp(path, name)[source]
async delQueue(name)[source]
async getPathList(path)[source]
async getPathObj(path)[source]
async getPathObjProp(path, prop)[source]
async getPathObjs(path)[source]
async getsQueue(name, offs, size=None, cull=True, wait=True)[source]
async hasPathObj(path)[source]
async initServiceStorage()[source]
async popPathObjProp(path, prop)[source]
async putsQueue(name, items, reqid=None)[source]
async setPathObj(path, item)[source]
async setPathObjProp(path, prop, valu)[source]

synapse.lib.jupyter module

class synapse.lib.jupyter.CmdrCore[source]

Bases: synapse.lib.base.Base

A helper for jupyter/storm CLI interaction

async addFeedData(name, items, *, viewiden=None)[source]

Add feed data to the cortex.

async eval(text, opts=None, num=None, cmdr=False)[source]

A helper for executing a storm command and getting a list of packed nodes.

Parameters
  • text (str) – Storm command to execute.

  • opts (dict) – Opt to pass to the cortex during execution.

  • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.

  • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True.

Returns

A list of packed nodes.

Return type

list

async runCmdLine(text)[source]

Run a line of text directly via cmdr.

async storm(text, opts=None, num=None, cmdr=False, suppress_logging=False)[source]

A helper for executing a storm command and getting a list of storm messages.

Parameters
  • text (str) – Storm command to execute.

  • opts (dict) – Opt to pass to the cortex during execution.

  • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.

  • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

  • suppress_logging (bool) – If True, suppresses some logging related to Storm runtime exceptions.

Notes

The opts dictionary will not be used if cmdr=True.

Returns

A list of storm messages.

Return type

list

suppress_logging(suppress)[source]

Context manager to suppress specific loggers.

synapse.lib.jupyter.genTempCoreProxy(mods=None)[source]

Get a temporary cortex proxy.

synapse.lib.jupyter.genTempStormsvcProxy(cmdrcore, svcname, svcctor, conf=None)[source]
synapse.lib.jupyter.getDocData(fp, root=None)[source]
Parameters
  • fn (str) – Name of the file to retrieve the data of.

  • root (str) – Optional root path to look for a docdata directory in.

Notes

Will detect json/jsonl/yaml/mpk extensions and automatically decode that data if found; otherwise it returns bytes.

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns

May be deserialized data or bytes.

Return type

data

Raises

ValueError if the file does not exist or directory traversal attempted..

synapse.lib.jupyter.getDocPath(fn, root=None)[source]

Helper for getting a documentation data file paths.

Parameters
  • fn (str) – Name of the file to retrieve the full path for.

  • root (str) – Optional root path to look for a docdata in.

Notes

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns

A file path.

Return type

str

Raises

ValueError if the file does not exist or directory traversal attempted..

async synapse.lib.jupyter.getItemCmdr(prox, outp=None, locs=None)[source]

Get a Cmdr instance with prepopulated locs

async synapse.lib.jupyter.getTempCoreCmdr(mods=None, outp=None)[source]

Get a CmdrCore instance which is backed by a temporary Cortex.

Parameters
  • mods (list) – A list of additional CoreModules to load in the Cortex.

  • outp – A output helper. Will be used for the Cmdr instance.

Notes

The CmdrCore returned by this should be fini()’d to tear down the temporary Cortex.

Returns

A CmdrCore instance.

Return type

CmdrCore

async synapse.lib.jupyter.getTempCoreCmdrStormsvc(svcname, svcctor, svcconf=None, outp=None)[source]

Get a proxy to a Storm service and a CmdrCore instance backed by a temporary Cortex with the service added.

Parameters
  • svcname (str) – Storm service name

  • svcctor – Storm service constructor (e.g. Example.anit)

  • svcconf – Optional conf for the Storm service

  • outp – A output helper for the Cmdr instance

Notes

Both the CmdrCore and Storm service proxy should be fini()’d for proper teardown

Returns

A CmdrCore instance and proxy to the Storm service

Return type

(CmdrCore, Proxy)

async synapse.lib.jupyter.getTempCoreProx(mods=None)[source]

Get a Telepath Proxt to a Cortex instance which is backed by a temporary Cortex.

Parameters

mods (list) – A list of additional CoreModules to load in the Cortex.

Notes

The Proxy returned by this should be fini()’d to tear down the temporary Cortex.

Returns

s_telepath.Proxy

synapse.lib.layer module

The Layer 2.0 archtecture introduces several optimized node/message serialization formats used by the layers to optimize returning primitives and facilitate efficient node construction:

Note

This interface is subject to change between minor revisions.

Storage Types (<stortype>)

In Layers 2.0, each node property from the model has an associated “storage type”. Each storage type determines how the data is indexed and represented within the Layer. This formalizes the separation of “data model” from “storage model”. Each data model type has a “stortype” property which coresponds to one of the STOR_TYPE_XXX values. The knowledge of the mapping of data model types to storage types is the responsibility of the data model, making the Layer implementation fully decoupled from the data model.

Node Edits / Edits

A node edit consists of a (<buid>, <form>, [edits]) tuple. An edit is Tuple of (<type>, <info>, List[NodeEdits]) where the first element is an int that matches to an EDIT_* constant below, the info is a tuple that varies depending on the first element, and the third element is a list of dependent NodeEdits that will only be applied if the edit actually makes a change.

Storage Node (<sode>)

A storage node is a layer/storage optimized node representation which is similar to a “packed node”. A storage node may be partial ( as it is produced by a given layer ) and are joined by the view/snap into “full” storage nodes which are used to construct Node() instances.

Sode format:

(<buid>, {

    'ndef': (<formname>, <formvalu>),

    'props': {
        <propname>: <propvalu>,
    }

    'tags': {
        <tagname>: <tagvalu>,
    }

    'tagprops: {
        <tagname>: {
            <propname>: <propvalu>,
        },
    }

    # changes that were *just* made.
    'edits': [
        <edit>
    ]

}),
class synapse.lib.layer.IndxBy(layr, abrv, db)[source]

Bases: object

IndxBy sub-classes encapsulate access methods and encoding details for various types of properties within the layer to be lifted/compared by storage types.

buidsByDups(indx)[source]
buidsByPref(indx=b'')[source]
buidsByRange(minindx, maxindx)[source]
buidsByRangeBack(minindx, maxindx)[source]
getNodeValu(buid)[source]
hasIndxBuid(indx, buid)[source]
keyBuidsByDups(indx)[source]
keyBuidsByPref(indx=b'')[source]
keyBuidsByRange(minindx, maxindx)[source]
keyBuidsByRangeBack(minindx, maxindx)[source]

Yields backwards from maxindx to minindx

scanByDups(indx)[source]
scanByPref(indx=b'')[source]
scanByRange(minindx, maxindx)[source]
class synapse.lib.layer.IndxByForm(layr, form)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByProp(layr, form, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByPropArray(layr, form, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByTag(layr, form, tag)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValuForm(buid)[source]
class synapse.lib.layer.IndxByTagProp(layr, form, tag, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.Layer[source]

Bases: synapse.lib.nexus.Pusher

The base class for a cortex layer.

async clone(newdirn)[source]

Copy the contents of this layer to a new layer

async delete()[source]

Delete the underlying storage

getAbrvProp(abrv)[source]
async getEdgeVerbs()[source]
async getEdges(verb=None)[source]
async getEditIndx()[source]

Returns what will be the next (i.e. 1 past the last) nodeedit log index.

async getEditOffs()[source]

Return the offset of the last recorded log entry. Returns -1 if nodeedit log is disabled or empty.

async getEditSize()[source]
async getFormCounts()[source]
getFormProps()[source]
getIdenFutu(iden=None)[source]
async getLayerSize()[source]

Get the total storage size for the layer.

async getMirrorStatus()[source]
async getModelVers()[source]
async getNodeData(buid, name)[source]

Return a single element of a buid’s node data

getNodeEditWindow()[source]
async getNodeForm(buid)[source]
async getNodeTag(buid, tag)[source]
async getNodeValu(buid, prop=None)[source]

Retrieve either the form valu or a prop valu for the given node by buid.

getPropAbrv(form, prop)[source]
async getPropCount(formname, propname=None, maxsize=None)[source]

Return the number of property rows in the layer for the given form/prop.

getStorIndx(stortype, valu)[source]
async getStorNode(buid)[source]
async getStorNodes()[source]

Yield (buid, sode) tuples for all the nodes with props/tags/tagprops stored in this layer.

async getTagCount(tagname, formname=None)[source]

Return the number of tag rows in the layer for the given tag/form.

getTagPropAbrv(*args)[source]
async getUnivPropCount(propname, maxsize=None)[source]

Return the number of universal property rows in the layer for the given prop.

async hasNodeData(buid, name)[source]
async hasTagProp(name)[source]
async initLayerActive()[source]
async initLayerPassive()[source]
async initUpstreamSync(url)[source]
async iterFormRows(form, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes of a single form, optionally (re)starting at startvalu.

Parameters
  • form (str) – A form name.

  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async iterLayerNodeEdits()[source]

Scan the full layer and yield artificial sets of nodeedits.

async iterNodeData(buid)[source]

Return a generator of all a buid’s node data

async iterNodeEdgesN1(buid, verb=None)[source]
async iterNodeEdgesN2(buid, verb=None)[source]
async iterNodeEditLog(offs=0)[source]

Iterate the node edit log and yield (offs, edits, meta) tuples.

async iterNodeEditLogBack(offs=0)[source]

Iterate the node edit log and yield (offs, edits, meta) tuples in reverse.

async iterPropRows(form, prop, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes with a particular secondary property, optionally (re)starting at startvalu.

Parameters
  • form (str) – A form name.

  • prop (str) – A universal property name.

  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async iterTagPropRows(tag, prop, form=None, stortype=None, startvalu=None)[source]

Yields (buid, valu) that match a tag:prop, optionally (re)starting at startvalu.

Parameters
  • tag (str) – tag name

  • prop (str) – prop name

  • form (Optional[str]) – optional form name

  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

  • startvalu (Any) – The value to start at. May on