synapse.lib package

Submodules

synapse.lib.agenda module

class synapse.lib.agenda.Agenda[source]

Bases: synapse.lib.base.Base

Organize and execute all the scheduled storm queries in a cortex.

add(cdef)[source]

Persistently adds an appointment

Parameters:cdef (dict) –

May contain the following keys: creator (str): iden of the creating user iden (str): Iden of the appointment storm (str): storm query to run reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…]):

one or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.
incunit (Union[None, TimeUnit]):
the unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.
incvals (Union[None, int, Iterable[int]): count of units of incunit or explicit day of week or day of
month. Not allowed for incunit == None, required for others (1 would be a typical value)

Notes

For values in reqs that are lists and incvals if a list, all combinations of all values (the product) are used

Returns:Packed appointment definition
delete(iden)[source]

Delete an appointment

disable(iden)[source]
enable(iden)[source]
get(iden)[source]
list()[source]
mod(iden, query)[source]

Change the query of an appointment

move(croniden, viewiden)[source]

Move a cronjob from one view to another

start()[source]

Enable cron jobs to start running, start the scheduler loop

Go through all the appointments, making sure the query is valid, and remove the ones that aren’t. (We can’t evaluate queries until enabled because not all the modules are loaded yet.)

stop()[source]

Cancel the scheduler loop, and set self.enabled to False.

class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)[source]

Bases: object

Represents a single element of a single combination of an appointment

nexttime(lastts)[source]

Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

pack()[source]

Make ApptRec json/msgpack-friendly

classmethod unpack(val)[source]

Convert from json/msgpack-friendly

class synapse.lib.agenda.TimeUnit[source]

Bases: enum.IntEnum

Unit of time that recurring and required parts of appointments are made of

DAY = 5
DAYOFMONTH = 3
DAYOFWEEK = 4
HOUR = 6
MINUTE = 7
MONTH = 2
NOW = 8
YEAR = 1
fromString = <bound method TimeUnit.fromString of <enum 'TimeUnit'>>[source]

synapse.lib.aha module

class synapse.lib.aha.AhaApi[source]

Bases: synapse.lib.cell.CellApi

addAhaSvc(name, info, network=None)[source]

Register a service with the AHA discovery server.

NOTE: In order for the service to remain marked “up” a caller
must maintain the telepath link.
delAhaSvc(name, network=None)[source]

Remove an AHA service entry.

genCaCert(network)[source]
getAhaSvc(name)[source]

Return an AHA service description dictionary for a fully qualified service name.

getAhaSvcs(network=None)[source]

Yield AHA svcinfo dictionaries.

Parameters:network (str) – Optionally specify a network to filter on.
getAhaUrls()[source]
getCaCert(network)[source]
signHostCsr(csrtext, signas=None, sans=None)[source]
signUserCsr(csrtext, signas=None)[source]
class synapse.lib.aha.AhaCell[source]

Bases: synapse.lib.cell.Cell

addAhaSvc(name, info, network=None)[source]
cellapi

alias of AhaApi

confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}}
delAhaSvc(name, network=None)[source]
genCaCert(network)[source]
getAhaSvc(name)[source]
getAhaSvcs(network=None)[source]
getCaCert(network)[source]
initServiceRuntime()[source]
initServiceStorage()[source]
saveCaCert(name, cakey, cacert)[source]
setAhaSvcDown(name, linkiden, network=None)[source]
signHostCsr(csrtext, signas=None, sans=None)[source]
signUserCsr(csrtext, signas=None)[source]

synapse.lib.ast module

class synapse.lib.ast.AbsProp(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.AbsPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AndCond(kids=())[source]

Bases: synapse.lib.ast.Cond

<cond> and <cond>

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

getLiftHints(runt, path)[source]
class synapse.lib.ast.ArgvQuery(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
isRuntSafe(runt)[source]
validate(runt)[source]
class synapse.lib.ast.ArrayCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AstNode(kids=())[source]

Bases: object

Base class for all nodes in the Storm abstract syntax tree.

addKid(astn)[source]
format(depth=0)[source]
getRuntVars(runt)[source]
hasAstClass(clss)[source]
init(core)[source]
isRuntSafe(runt)[source]
iterright()[source]

Yield “rightward” siblings until None.

optimize()[source]
prepare()[source]
repr()[source]
sibling(offs=1)[source]

Return sibling node by relative offset from self.

validate(runt)[source]
class synapse.lib.ast.Bool(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.BreakOper(kids=())[source]

Bases: synapse.lib.ast.AstNode

run(runt, genr)[source]
class synapse.lib.ast.CallArgs(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.CallKwarg(kids=())[source]

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CallKwargs(kids=())[source]

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CaseEntry(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.CmdOper(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
class synapse.lib.ast.Cmpr(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.Cond(kids=())[source]

Bases: synapse.lib.ast.Value

A condition that is evaluated to filter nodes.

class synapse.lib.ast.Const(valu, kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
isRuntSafe(runt)[source]
repr()[source]
value()[source]
class synapse.lib.ast.ContinueOper(kids=())[source]

Bases: synapse.lib.ast.AstNode

run(runt, genr)[source]
class synapse.lib.ast.DollarExpr(kids=())[source]

Bases: synapse.lib.ast.Value

Top level node for $(…) expressions

compute(runt, path)[source]
class synapse.lib.ast.Edit(kids=())[source]

Bases: synapse.lib.ast.Oper

class synapse.lib.ast.EditEdgeAdd(kids=(), n2=False)[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditEdgeDel(kids=(), n2=False)[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditNodeAdd(kids=())[source]

Bases: synapse.lib.ast.Edit

addFromPath(form, runt, path)[source]

Add a node using the context from path.

NOTE: CALLER MUST CHECK PERMS

prepare()[source]
run(runt, genr)[source]
class synapse.lib.ast.EditParens(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditPropDel(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditPropSet(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditTagAdd(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditTagDel(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EditTagPropDel(kids=())[source]

Bases: synapse.lib.ast.Edit

[ -#foo.bar:baz ]

run(runt, genr)[source]
class synapse.lib.ast.EditTagPropSet(kids=())[source]

Bases: synapse.lib.ast.Edit

[ #foo.bar:baz=10 ]

run(runt, genr)[source]
class synapse.lib.ast.EditUnivDel(kids=())[source]

Bases: synapse.lib.ast.Edit

run(runt, genr)[source]
class synapse.lib.ast.EmbedQuery(valu, kids=())[source]

Bases: synapse.lib.ast.Const

compute(runt, path)[source]
getRuntVars(runt)[source]
validate(runt)[source]
class synapse.lib.ast.ExprAndNode(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.ExprNode(kids=())[source]

Bases: synapse.lib.ast.Value

A binary (i.e. two argument) expression node

compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.ExprOrNode(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.FiltByArray(kids=())[source]

Bases: synapse.lib.ast.FiltOper

+:foo*[^=visi]

class synapse.lib.ast.FiltOper(kids=())[source]

Bases: synapse.lib.ast.Oper

getLiftHints(runt, path)[source]
run(runt, genr)[source]
class synapse.lib.ast.FiniBlock(kids=())[source]

Bases: synapse.lib.ast.AstNode

An AST node that runs only once after all nodes have been consumed.

Example

Using a fini block:

fini {
   // stuff here runs *once* after the last node yield (even if there are no nodes)
}

Notes

A fini block must be runtsafe.

run(runt, genr)[source]
class synapse.lib.ast.ForLoop(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
run(runt, genr)[source]
class synapse.lib.ast.FormName(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.FormPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> foo:bar

run(runt, genr)[source]
class synapse.lib.ast.FormTagProp(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.FuncArgs(kids=())[source]

Bases: synapse.lib.ast.AstNode

Represents the function arguments in a function definition

compute(runt, path)[source]
class synapse.lib.ast.FuncCall(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.Function(kids=())[source]

Bases: synapse.lib.ast.AstNode

( name, args, body )

// use args/kwargs syntax function bar(x, v=$(30)) { }

# we auto-detect the behavior of the target function

# return a value function bar(x, y) { return ($(x + y)) }

# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }

$foo = $bar(10, v=20)

callfunc(runt, argdefs, args, kwargs)[source]

Execute a function call using the given runtime.

This function may return a value / generator / async generator

getRuntVars(runt)[source]
isRuntSafe(runt)[source]
prepare()[source]
run(runt, genr)[source]
validate(runt)[source]
class synapse.lib.ast.HasAbsPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.HasRelPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

getLiftHints(runt, path)[source]
hasProp(node, runt, name)[source]
class synapse.lib.ast.HasTagPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.IfClause(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.IfStmt(kids=())[source]

Bases: synapse.lib.ast.Oper

prepare()[source]
run(runt, genr)[source]
class synapse.lib.ast.InitBlock(kids=())[source]

Bases: synapse.lib.ast.AstNode

An AST node that runs only once before yielding nodes.

Example

Using a init block:

init {
    // stuff here runs *once* before the first node yield (even if there are no nodes)
}
run(runt, genr)[source]
class synapse.lib.ast.LiftByArray(kids=())[source]

Bases: synapse.lib.ast.LiftOper

:prop*[range=(200, 400)]

lift(runt, path)[source]
class synapse.lib.ast.LiftFormTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

lift(runt, path)[source]
class synapse.lib.ast.LiftFormTagProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

hehe:haha#foo.bar:baz [ = x ]

lift(runt, path)[source]
class synapse.lib.ast.LiftOper(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
class synapse.lib.ast.LiftProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

getRightHints(runt, path)[source]
lift(runt, path)[source]
class synapse.lib.ast.LiftPropBy(kids=())[source]

Bases: synapse.lib.ast.LiftOper

lift(runt, path)[source]
class synapse.lib.ast.LiftTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

lift(runt, path)[source]
class synapse.lib.ast.LiftTagProp(kids=())[source]

Bases: synapse.lib.ast.LiftOper

#foo.bar:baz [ = x ]

lift(runt, path)[source]
class synapse.lib.ast.LiftTagTag(kids=())[source]

Bases: synapse.lib.ast.LiftOper

##foo.bar

lift(runt, path)[source]
class synapse.lib.ast.List(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
repr()[source]
class synapse.lib.ast.Lookup(kids, autoadd=False)[source]

Bases: synapse.lib.ast.Query

When storm input mode is “lookup”

run(runt, genr)[source]
class synapse.lib.ast.N1Walk(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
walkNodeEdges(runt, node, verb=None)[source]
class synapse.lib.ast.N1WalkNPivo(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOut

run(runt, genr)[source]
class synapse.lib.ast.N2Walk(kids=())[source]

Bases: synapse.lib.ast.N1Walk

walkNodeEdges(runt, node, verb=None)[source]
class synapse.lib.ast.N2WalkNPivo(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotIn

run(runt, genr)[source]
class synapse.lib.ast.NotCond(kids=())[source]

Bases: synapse.lib.ast.Cond

not <cond>

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.Oper(kids=())[source]

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.OrCond(kids=())[source]

Bases: synapse.lib.ast.Cond

<cond> or <cond>

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.PivotIn(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

<- *

getPivsIn(runt, node, path)[source]
run(runt, genr)[source]
class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

<- foo:edge

run(runt, genr)[source]
class synapse.lib.ast.PivotOper(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.Oper

repr()[source]
class synapse.lib.ast.PivotOut(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> *

getPivsOut(runt, node, path)[source]
run(runt, genr)[source]
class synapse.lib.ast.PivotToTags(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present

run(runt, genr)[source]
class synapse.lib.ast.PropName(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.PropPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

:foo -> bar:foo

run(runt, genr)[source]
class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

:prop -> *

run(runt, genr)[source]
class synapse.lib.ast.PropValue(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
getPropAndValu(runt, path)[source]
isRuntSafe(valu)[source]
prepare()[source]
class synapse.lib.ast.Query(kids=())[source]

Bases: synapse.lib.ast.AstNode

iterNodePaths(runt, genr=None)[source]
run(runt, genr)[source]
class synapse.lib.ast.RawPivot(kids=(), isjoin=False)[source]

Bases: synapse.lib.ast.PivotOper

-> { <varsfrompath> }

run(runt, genr)[source]
class synapse.lib.ast.RelProp(kids=())[source]

Bases: synapse.lib.ast.PropName

class synapse.lib.ast.RelPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

(:foo:bar or .univ) <cmpr> <value>

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

getLiftHints(runt, path)[source]
class synapse.lib.ast.RelPropValue(kids=())[source]

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Return(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
class synapse.lib.ast.SetItemOper(kids=())[source]

Bases: synapse.lib.ast.Oper

$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz

run(runt, genr)[source]
class synapse.lib.ast.SetVarOper(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
run(runt, genr)[source]
class synapse.lib.ast.SubGraph(rules)[source]

Bases: object

An Oper like object which generates a subgraph.

Notes

The rules format for the subgraph is shaped like the following:

rules = {

    'degrees': 1,

    'edges': True,
    'filterinput': True,
    'yieldfiltered': False,

    'filters': [
        '-(#foo or #bar)',
        '-(foo:bar or baz:faz)',
    ],

    'pivots': [
        '-> * | limit 100',
        '<- * | limit 100',
    ]

    'forms': {

        'inet:fqdn':{
            'filters': [],
            'pivots': [],
        }

        '*': {
            'filters': [],
            'pivots': [],
        },
    },
}

Nodes which were original seeds have path.meta(‘graph:seed’).

All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.

omit(runt, node)[source]
pivots(runt, node, path)[source]
run(runt, genr)[source]
class synapse.lib.ast.SubQuery(kids=())[source]

Bases: synapse.lib.ast.Oper

compute(runt, path)[source]

Use subquery as a value. It is error if the subquery used in this way doesn’t yield exactly one node or has a return statement.

Its value is the primary property of the node yielded, or the returned value.

compute_array(runt, path)[source]

Use subquery as an array.

inline(runt, genr)[source]

Operate subquery as if it were inlined

run(runt, genr)[source]
class synapse.lib.ast.SubqCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.SwitchCase(kids=())[source]

Bases: synapse.lib.ast.Oper

prepare()[source]
run(runt, genr)[source]
class synapse.lib.ast.TagCond(kids=())[source]

Bases: synapse.lib.ast.Cond

#foo.bar

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

getLiftHints(runt, path)[source]
class synapse.lib.ast.TagMatch(kids=())[source]

Bases: synapse.lib.ast.TagName

Like TagName, but can have asterisks

hasglob()[source]
class synapse.lib.ast.TagName(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.TagProp(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.TagPropCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagPropValue(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.TagValuCond(kids=())[source]

Bases: synapse.lib.ast.Cond

getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagValue(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
isRuntSafe(runt)[source]
class synapse.lib.ast.UnaryExprNode(kids=())[source]

Bases: synapse.lib.ast.Value

A unary (i.e. single-argument) expression node

compute(runt, path)[source]
prepare()[source]
class synapse.lib.ast.UnivProp(kids=())[source]

Bases: synapse.lib.ast.RelProp

compute(runt, path)[source]
class synapse.lib.ast.UnivPropValue(kids=())[source]

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Value(kids=())[source]

Bases: synapse.lib.ast.AstNode

The base class for all values and value expressions.

compute(runt, path)[source]
getCondEval(runt)[source]

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

getLiftHints(runt, path)[source]
isRuntSafe(runt)[source]
class synapse.lib.ast.VarDeref(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
class synapse.lib.ast.VarEvalOper(kids=())[source]

Bases: synapse.lib.ast.Oper

Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)

run(runt, genr)[source]
class synapse.lib.ast.VarList(valu, kids=())[source]

Bases: synapse.lib.ast.Const

class synapse.lib.ast.VarListSetOper(kids=())[source]

Bases: synapse.lib.ast.Oper

getRuntVars(runt)[source]
run(runt, genr)[source]
class synapse.lib.ast.VarValue(kids=())[source]

Bases: synapse.lib.ast.Value

compute(runt, path)[source]
isRuntSafe(runt)[source]
prepare()[source]
validate(runt)[source]
class synapse.lib.ast.WhileLoop(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
class synapse.lib.ast.YieldValu(kids=())[source]

Bases: synapse.lib.ast.Oper

run(runt, genr)[source]
yieldFromValu(runt, valu)[source]
synapse.lib.ast.expr_add(x, y)[source]
synapse.lib.ast.expr_div(x, y)[source]
synapse.lib.ast.expr_eq(x, y)[source]
synapse.lib.ast.expr_ge(x, y)[source]
synapse.lib.ast.expr_gt(x, y)[source]
synapse.lib.ast.expr_le(x, y)[source]
synapse.lib.ast.expr_lt(x, y)[source]
synapse.lib.ast.expr_mul(x, y)[source]
synapse.lib.ast.expr_ne(x, y)[source]
synapse.lib.ast.expr_not(x)[source]
synapse.lib.ast.expr_prefix(x, y)[source]
synapse.lib.ast.expr_re(x, y)[source]
synapse.lib.ast.expr_sub(x, y)[source]
synapse.lib.ast.parseNumber(x)[source]
synapse.lib.ast.pullone(genr)[source]

synapse.lib.autodoc module

class synapse.lib.autodoc.RstHelp[source]

Bases: object

addHead(name, lvl=0, link=None)[source]
addLines(*lines)[source]
getRstText()[source]
synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)[source]

Process a list of StormTypes doc information to add them to a a RstHelp object.

Notes
This will create internal hyperlink link targets for each header item. The link prefix string must be given with the linkprefix argument.
Parameters:
  • page (RstHelp) – The RST page to add .
  • docinfo (dict) – A Stormtypes Doc.
  • linkprefix (str) – The RST link prefix string to use.
  • islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.
  • lvl (int) – The base header level to use when adding headers to the page.
Returns:

None

synapse.lib.autodoc.genCallsig(rtype)[source]
synapse.lib.autodoc.getArgLines(rtype)[source]
synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None)[source]
synapse.lib.autodoc.getRtypeStr(rtype, known_types, types_prefix, suffix)[source]
synapse.lib.autodoc.ljuster(ilines)[source]

Helper to lstrip lines of whitespace an appropriate amount.

synapse.lib.autodoc.prepareRstLines(doc)[source]

Prepare a desc string for RST lines.

synapse.lib.autodoc.scrubLines(lines)[source]

Remove any empty lines until we encounter non-empty linee

synapse.lib.base module

class synapse.lib.base.Base[source]

Bases: object

Base class for Synapse objects.

Acts as an observable, enables async init and fini.

Example

class Foo(Base):

async def __anit__(self, x, y):

await Base.__anit__(self)

await stuff(x, y)

foo = await Foo.anit(10)

Note

One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

addSignalHandlers()[source]

Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

classmethod anit(*args, **kwargs)[source]
dist(mesg)[source]

Distribute an existing event tuple.

Parameters:mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) )

enter_context(item)[source]

Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.

Returns:The result of item’s own __aenter__ or __enter__() method.
fini()[source]

Shut down the object and notify any onfini() coroutines.

Returns:Remaining ref count
fire(evtname, **info)[source]

Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’):
print(‘got: %r’ % (ret,))
incref()[source]

Increment the reference count for this base. This API may be optionally used to control fini().

Add a callback function to receive all events.

Example

base1 = Base() base2 = Base()

base1.link( base2.dist )

# all events on base1 are also propagated on base2

main()[source]

Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )

NOTE: This API may only be used when the ioloop is also the main thread.

off(evnt, func)[source]

Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc )

on(evnt, func, base=None)[source]

Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.

Parameters:
  • evnt (str) – An event name
  • func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it:

async def baz(event):
x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y

d.on(‘foo’, baz)

# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)

Returns:
Return type:None
onWith(evnt, func)[source]

A context manager which can be used to add a callback and remove it when using a with statement.

Parameters:
  • evnt (str) – An event name
  • func (function) – A callback function to receive event tufo
onfini(func)[source]

Add a function/coroutine/Base to be called on fini().

postAnit()[source]

Method called after self.__anit__() has completed, but before anit() returns the object to the caller.

schedCallSafe(func, *args, **kwargs)[source]

Schedule a function to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on the function completion.

Parameters:
  • func
  • *args
  • **kwargs

Notes

This method may be called from outside of the event loop on a different thread.

Returns:A Future representing the eventual function execution.
Return type:concurrent.futures.Future
schedCoro(coro)[source]

Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.

Precondition:
This function is not threadsafe and must be run on the Base’s event loop
Returns:An asyncio.Task object.
Return type:asyncio.Task
schedCoroSafe(coro)[source]

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on coroutine completion.

Notes

This method may be run outside the event loop on a different thread.

Returns:A Future representing the eventual coroutine execution.
Return type:concurrent.futures.Future
schedCoroSafePend(coro)[source]

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note

This method may not be run inside an event loop

Remove a callback function previously added with link()

Example

base.unlink( callback )

waiter(count, *names)[source]

Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events…

waiter = base.waiter(10,’foo:bar’)

# .. fire thread that will cause foo:bar events

events = await waiter.wait(timeout=3)

if events == None:
# handle the timeout case…
for event in events:
# parse the events if you need…
NOTE: use with caution… it’s easy to accidentally construct
race conditions with this mechanism ;)
waitfini(timeout=None)[source]

Wait for the base to fini()

Returns:None if timed out, True if fini happened

Example

base.waitfini(timeout=30)

class synapse.lib.base.BaseRef[source]

Bases: synapse.lib.base.Base

An object for managing multiple Base instances by name.

gen(name)[source]

Atomically get/gen a Base and incref. (requires ctor during BaseRef init)

Parameters:name (str) – The name/iden of the Base instance.
get(name)[source]

Retrieve a Base instance by name.

Parameters:name (str) – The name/iden of the Base
Returns:The Base instance (or None)
Return type:(Base)
items()[source]
pop(name)[source]

Remove and return a Base from the BaseRef.

Parameters:name (str) – The name/iden of the Base instance
Returns:The named base ( or None )
Return type:(Base)
put(name, base)[source]

Add a Base (or sub-class) to the BaseRef by name.

Parameters:
  • name (str) – The name/iden of the Base
  • base (Base) – The Base instance
Returns:

(None)

vals()[source]
class synapse.lib.base.Waiter(base, count, *names)[source]

Bases: object

A helper to wait for a given number of events on a Base.

fini()[source]
wait(timeout=None)[source]

Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30)

if evnts == None:
handleTimedOut() return
for evnt in evnts:
doStuff(evnt)
synapse.lib.base.main(coro)[source]
synapse.lib.base.schedGenr(genr, maxsize=100)[source]

Schedule a generator to run on a separate task and yield results to this task (pipelined generator).

synapse.lib.boss module

class synapse.lib.boss.Boss[source]

Bases: synapse.lib.base.Base

An object to track “promoted” async tasks.

execute(coro, name, user, info=None)[source]

Create a synapse task from the given coroutine.

get(iden)[source]
promote(name, user, info=None)[source]

Promote the currently running task.

Parameters:
  • name (str) – The name of the task.
  • user – The User who owns the task.
  • info – An optional information dictionary containing information about the task.
Returns:

The Synapse Task object.

Return type:

s_task.Task

ps()[source]

synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully.

class synapse.lib.cache.FixedCache(callback, size=10000)[source]

Bases: object

aget(key)[source]
clear()[source]
get(key)[source]
pop(key)[source]
put(key, val)[source]
class synapse.lib.cache.LruDict(size=10000)[source]

Bases: collections.abc.MutableMapping

Maintains the last n accessed keys

get(key, default=None)[source]

Note: we override default impl from parent to avoid costly KeyError

items() → a set-like object providing a view on D's items[source]
values() → an object providing a view on D's values[source]
class synapse.lib.cache.TagGlobs[source]

Bases: object

An object that manages multiple tag globs and values for caching.

add(name, valu, base=None)[source]
get(name)[source]
rem(name, valu)[source]
synapse.lib.cache.getTagGlobRegx[source]
synapse.lib.cache.memoize(size=16384)[source]
synapse.lib.cache.memoizemethod(size=16384)[source]

A version of memoize that doesn’t cause GC cycles when applied to a method.

synapse.lib.cache.regexizeTagGlob(tag)[source]
Returns:a regular expression string with ** and * interpreted as tag globs
Precondition:
tag is a valid tagmatch

Notes

A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.

The returned string does not contain a starting ‘^’ or trailing ‘$’.

synapse.lib.cell module

class synapse.lib.cell.Cell[source]

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

A Cell() implements a synapse micro-service.

A Cell has 5 phases of startup:
  1. Universal cell data structures
  2. Service specific storage/data (pre-nexs)
  3. Nexus subsystem initialization
  4. Service specific startup (with nexus)
  5. Networking and mirror services
BACKUP_SPAWN_TIMEOUT = 60.0
COMMIT = ''
VERSION = (2, 64, 1)
VERSTRING = '2.64.1'
addActiveCoro(func, iden=None, base=None)[source]

Add a function callback to be run as a coroutine when the Cell is active.

Parameters:
  • func (coroutine function) – The function run as a coroutine.
  • iden (str) – The iden to use for the coroutine.
  • base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d
Returns:

A GUID string that identifies the coroutine for delActiveCoro()

Return type:

str

Note

This will re-fire the coroutine if it exits and the Cell is still active.

addHealthFunc(func)[source]

Register a callback function to get a HealthCheck object.

addHttpApi(path, ctor, info)[source]
addHttpsPort(port, host='0.0.0.0', sslctx=None)[source]
addRole(name)[source]
addRoleRule(iden, rule, indx=None, gateiden=None)[source]
addUser(name, passwd=None, email=None, iden=None)[source]
addUserRole(useriden, roleiden)[source]
addUserRule(iden, rule, indx=None, gateiden=None)[source]
cellapi

alias of CellApi

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the cell.', 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'type': 'string'}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to the cell. Required for mirroring (on both sides).', 'type': 'boolean'}}
confdefs = {}
cullNexsLog(offs)[source]
delActiveCoro(iden)[source]

Remove an Active coroutine previously added with addActiveCoro().

Parameters:iden (str) – The iden returned by addActiveCoro()
delBackup(name)[source]
delRole(iden)[source]
delRoleRule(iden, rule, gateiden=None)[source]
delUser(iden)[source]
delUserRole(useriden, roleiden)[source]
delUserRule(iden, rule, gateiden=None)[source]
dyncall(iden, todo, gatekeys=())[source]
dyniter(iden, todo, gatekeys=())[source]
classmethod execmain(argv, outp=None)[source]

The main entry point for running the Cell as an application.

Parameters:
  • argv (list) – A list of command line arguments to launch the Cell with.
  • outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.

Returns:None.
genHttpSess(iden)[source]
classmethod getArgParser(conf=None)[source]

Get an argparse.ArgumentParser for the Cell.

Parameters:conf (s_config.Config) – Optional, a Config object which

Notes

Boot time configuration data is placed in the argument group called config. This adds default dirn, --telepath, --https and --name arguements to the argparser instance. Configuration values which have the hideconf or hidecmdl value set to True are not added to the argparser instance.

Returns:A ArgumentParser for the Cell.
Return type:argparse.ArgumentParser
getAuthGate(iden)[source]
getAuthGates()[source]
getAuthRoles()[source]
getAuthUsers(archived=False)[source]
getBackupInfo()[source]

Gets information about recent backup activity

getBackups()[source]
getCellApi(link, user, path)[source]
getCellIden()[source]
getCellInfo()[source]

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns:A Dictionary of metadata.
Return type:Dict
classmethod getCellType()[source]
getConfOpt(name)[source]
getDmonSessions()[source]
classmethod getEnvPrefix()[source]
getHealthCheck()[source]
getHiveKey(path)[source]

Get the value of a key in the cell default hive

getHiveKeys(path)[source]

Return a list of (name, value) tuples for nodes under the path.

getLocalProxy(share='*', user='root')[source]
getLocalUrl(share='*', user='root')[source]
getLogExtra(**kwargs)[source]

Get an extra dictionary for structured logging which can be used as a extra argument for loggers.

Parameters:**kwargs – Additional key/value items to add to the log.
Returns:A dictionary
Return type:Dict
getNexsIndx()[source]
getNexusChanges(offs)[source]
getRoleDef(iden)[source]
getRoleDefByName(name)[source]
getRoleDefs()[source]
getSystemInfo()[source]

Get info about the system in which the cell is running

Returns:
  • volsize - Volume where cell is running total space
  • volfree - Volume where cell is running free space
  • backupvolsize - Backup directory volume total space
  • backupvolfree - Backup directory volume free space
  • cellstarttime - Cell start time in epoch milliseconds
  • celluptime - Cell uptime in milliseconds
  • cellrealdisk - Cell’s use of disk, equivalent to du
  • cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
  • osversion - OS version/architecture
  • pyversion - Python version
  • totalmem - Total memory in the system
  • availmem - Available memory in the system
  • cpucount - Number of CPUs on system
Return type:A dictionary with the following keys. All size values are in bytes
getTeleApi(link, mesg, path)[source]

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

getUserDef(iden)[source]
getUserDefByName(name)[source]
getUserDefs()[source]
getUserName(iden, defv='<unknown>')[source]

Translate the user iden to a user name.

getUserProfInfo(iden, name)[source]
getUserProfile(iden)[source]
classmethod initCellConf()[source]

Create a Config object for the Cell.

Notes

The Config object has a envar_prefix set according to the results of cls.getEnvPrefix().

Returns:A Config helper object.
Return type:s_config.Config
classmethod initFromArgv(argv, outp=None)[source]

Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.

Parameters:
  • argv (list) – A list of command line arguments to launch the Cell with.
  • outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This does the following items:
  • Create a Config object from the Cell class.
  • Creates an Argument Parser from the Cell class and Config object.
  • Parses the provided arguments.
  • Loads configuration data from the parsed options and environment variables.
  • Sets logging for the process.
  • Creates the Cell from the Cell Ctor.
  • Adds a Telepath listener, HTTPs port listeners and Telepath share names.
  • Returns the Cell.
Returns:This returns an instance of the Cell.
Return type:Cell
initNexusSubsystem()[source]
initServiceActive()[source]
initServiceNetwork()[source]
initServicePassive()[source]
initServiceRuntime()[source]
initServiceStorage()[source]
initSslCtx(certpath, keypath)[source]
isCellActive()[source]
isUserAllowed(iden, perm, gateiden=None)[source]
iterBackupArchive(name, user)[source]
iterNewBackupArchive(user, name=None, remove=False)[source]
kill(user, iden)[source]
listHiveKey(path=None)[source]
loadHiveTree(tree, path=(), trim=False)[source]

Note: this is for expert emergency use only.

popHiveKey(path)[source]

Remove and return the value of a key in the cell default hive.

Note: this is for expert emergency use only.

promote()[source]

Transform this cell from a passive follower to an active cell that writes changes locally.

ps(user)[source]
rotateNexsLog()[source]
runBackup(name=None, wait=True)[source]
saveHiveTree(path=())[source]
setCellActive(active)[source]
setHiveKey(path, valu)[source]

Set or change the value of a key in the cell default hive

setNexsIndx(indx)[source]
setRoleName(iden, name)[source]
setRoleRules(iden, rules, gateiden=None)[source]
setUserAdmin(iden, admin, gateiden=None)[source]
setUserArchived(iden, archived)[source]
setUserEmail(useriden, email)[source]
setUserLocked(iden, locked)[source]
setUserName(useriden, name)[source]
setUserPasswd(iden, passwd)[source]
setUserProfInfo(iden, name, valu)[source]
setUserRoles(useriden, roleidens)[source]
setUserRules(iden, rules, gateiden=None)[source]
sync()[source]

no-op mutable for testing purposes. If I am follower, when this returns, I have received and applied all the writes that occurred on the leader before this call.

trimNexsLog(consumers=None, timeout=30)[source]
tryUserPasswd(name, passwd)[source]
waitNexsOffs(offs, timeout=None)[source]
class synapse.lib.cell.CellApi[source]

Bases: synapse.lib.base.Base

addAuthRole(name)[source]
addAuthRule(name, rule, indx=None, gateiden=None)[source]
addRole(name)[source]
addRoleRule(iden, rule, indx=None, gateiden=None)[source]
addUser(name, passwd=None, email=None, iden=None)[source]
addUserRole(useriden, roleiden)[source]
addUserRule(iden, rule, indx=None, gateiden=None)[source]
allowed(perm, default=None)[source]

Check if the user has the requested permission.

Parameters:
  • perm – permission path components to check
  • default – Value returned if no value stored

Examples

Form a path and check the permission from a remote proxy:

perm = ('node', 'add', 'inet:ipv4')
allowed = await prox.allowed(perm)
if allowed:
    dostuff()
Returns:True if the user has permission, False if explicitly denied, None if no entry
Return type:Optional[bool]
cullNexsLog(offs)[source]

Remove Nexus log entries up to (and including) the given offset.

Note

If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.

Only rotated logs where the last index is less than the provided offset will be removed from disk.

Parameters:offs (int) – The offset to remove entries up to.
Returns:Whether the cull was executed
Return type:bool
delAuthRole(name)[source]
delAuthRule(name, rule, gateiden=None)[source]
delAuthUser(name)[source]
delBackup(name)[source]

Delete a backup by name.

Parameters:name (str) – The name of the backup to delete.
delRole(iden)[source]
delRoleRule(iden, rule, gateiden=None)[source]
delUser(iden)[source]
delUserRole(useriden, roleiden)[source]
delUserRule(iden, rule, gateiden=None)[source]
dyncall(iden, todo, gatekeys=())[source]
dyniter(iden, todo, gatekeys=())[source]
getAuthGate(iden)[source]
getAuthGates()[source]
getAuthInfo(name)[source]
getAuthRoles()[source]
getAuthUsers(archived=False)[source]
Parameters:archived (bool) – If true, list all users, else list non-archived users
getBackupInfo()[source]

Get information about recent backup activity.

Returns:
  • currduration - If backup currently running, time in ms since backup started, otherwise None
  • laststart - Last time (in epoch milliseconds) a backup started
  • lastend - Last time (in epoch milliseconds) a backup ended
  • lastduration - How long last backup took in ms
  • lastsize - Disk usage of last backup completed
  • lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive
  • lastexception - Tuple of exception information if last backup failed, otherwise None
Return type:(dict) It has the following keys

Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.

getBackups()[source]

Retrieve a list of backups.

Returns:A list of backup names.
Return type:list[str]
getCellIden()[source]
getCellInfo()[source]
getCellType()[source]
getCellUser()[source]
getDiagInfo()[source]
getDmonSessions()[source]
getHealthCheck()[source]
getHiveKey(path)[source]
getHiveKeys(path)[source]
getNexsIndx()[source]
getNexusChanges(offs)[source]
getRoleDef(iden)[source]
getRoleDefByName(name)[source]
getRoleDefs()[source]
getRoleInfo(name)[source]
getSystemInfo()[source]

Get info about the system in which the cell is running

Returns:
  • volsize - Volume where cell is running total space
  • volfree - Volume where cell is running free space
  • backupvolsize - Backup directory volume total space
  • backupvolfree - Backup directory volume free space
  • celluptime - Cell uptime in milliseconds
  • cellrealdisk - Cell’s use of disk, equivalent to du
  • cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
  • osversion - OS version/architecture
  • pyversion - Python version
  • totalmem - Total memory in the system
  • availmem - Available memory in the system
Return type:A dictionary with the following keys. All size values are in bytes
getUserDef(iden)[source]
getUserDefByName(name)[source]
getUserDefs()[source]
getUserInfo(name)[source]
getUserProfInfo(iden, name)[source]
getUserProfile(iden)[source]
initCellApi()[source]
isCellActive()[source]

Returns True if the cell is an active/leader cell.

isUserAllowed(iden, perm, gateiden=None)[source]
issue(nexsiden: str, event: str, args, kwargs, meta=None)[source]

Note: this swallows exceptions and return values. It is expected that the nexus _followerLoop would be the return path

iterBackupArchive(name)[source]

Retrieve a backup by name as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters:name (str) – The name of the backup to retrieve.
iterNewBackupArchive(name=None, remove=False)[source]

Run a new backup and return it as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters:
  • name (str) – The name of the backup to retrieve.
  • remove (bool) – Delete the backup after streaming.
kill(iden)[source]
listHiveKey(path=None)[source]
popHiveKey(path)[source]
promote()[source]
ps()[source]
rotateNexsLog()[source]

Rotate the Nexus log at the current offset.

Returns:The starting index of the active Nexus log
Return type:int
runBackup(name=None, wait=True)[source]

Run a new backup.

Parameters:
  • name (str) – The optional name of the backup.
  • wait (bool) – On True, wait for backup to complete before returning.
Returns:

The name of the newly created backup.

Return type:

str

saveHiveTree(path=())[source]
setAuthAdmin(name, isadmin)[source]
setCellUser(iden)[source]

Switch to another user (admin only).

This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.

setHiveKey(path, valu)[source]
setRoleRules(iden, rules, gateiden=None)[source]
setUserAdmin(iden, admin, gateiden=None)[source]
setUserArchived(useriden, archived)[source]
setUserEmail(useriden, email)[source]
setUserLocked(useriden, locked)[source]
setUserPasswd(iden, passwd)[source]
setUserProfInfo(iden, name, valu)[source]
setUserRoles(useriden, roleidens)[source]
setUserRules(iden, rules, gateiden=None)[source]
trimNexsLog(consumers=None, timeout=60)[source]

Rotate and cull the Nexus log (and those of any consumers) at the current offset.

Note

If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.

Parameters:
  • consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.
  • timeout (int) – Time in seconds to wait for downstream consumers to be caught up.
Returns:

The offset that the Nexus log was culled up to and including.

Return type:

int

tryUserPasswd(name, passwd)[source]
waitNexsOffs(offs, timeout=None)[source]

Wait for the Nexus log to write an offset.

Parameters:
  • offs (int) – The offset to wait for.
  • timeout (int or None) – An optional timeout in seconds.
Returns:

True if the offset was written, False if it timed out.

Return type:

bool

synapse.lib.cell.SLAB_MAP_SIZE = 134217728

Base classes for the synapse “cell” microservice architecture.

synapse.lib.cell.adminapi(log=False)[source]

Decorator for CellApi (and subclasses) for requiring a method to be called only by an admin user.

Parameters:log (bool) – If set to True, log the user, function and arguments.

synapse.lib.certdir module

class synapse.lib.certdir.CertDir(path=None)[source]

Bases: object

Certificate loading/generation/signing utilities.

Features:
  • Locates and load certificates, keys, and certificate signing requests (CSRs).
  • Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.
  • Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).
  • Signs certificate signing requests (CSRs).
  • Generates PKCS#12 archives for use in browser.
Parameters:path (str) – Optional path which can override the default path directory.

Notes

  • All certificates will be loaded from and written to ~/.syn/certs by default. Set the envvar SYN_CERT_DIR to override.
  • All certificate generation methods create 4096 bit RSA keypairs.
  • All certificate signing methods use sha256 as the signature algorithm.
  • CertDir does not currently support signing CA CSRs.
addCertPath(*path)[source]
delCertPath(*path)[source]
genCaCert(name, signas=None, outp=None, save=True)[source]

Generates a CA keypair.

Parameters:
  • name (str) – The name of the CA keypair.
  • signas (str) – The CA keypair to sign the new CA with.
  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make a CA named “myca”:

mycakey, mycacert = cdir.genCaCert(‘myca’)
Returns:Tuple containing the private key and certificate objects.
Return type:((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
genClientCert(name, outp=None)[source]

Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material.

Parameters:
  • name (str) – The name of the user keypair.
  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”:

myuserpkcs12 = cdir.genClientCert(‘myuser’)
Returns:The PKCS #12 archive.
Return type:OpenSSL.crypto.PKCS12
genHostCert(name, signas=None, outp=None, csr=None, sans=None)[source]

Generates a host keypair.

Parameters:
  • name (str) – The name of the host keypair.
  • signas (str) – The CA keypair to sign the new host keypair with.
  • outp (synapse.lib.output.Output) – The output buffer.
  • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.
  • sans (list) – List of subject alternative names.

Examples

Make a host keypair named “myhost”:

myhostkey, myhostcert = cdir.genHostCert(‘myhost’)
Returns:Tuple containing the private key and certificate objects.
Return type:((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
genHostCsr(name, outp=None)[source]

Generates a host certificate signing request.

Parameters:
  • name (str) – The name of the host CSR.
  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the host key named “myhost”:

cdir.genHostCsr(‘myhost’)
Returns:The bytes of the CSR.
Return type:bytes
genUserCert(name, signas=None, outp=None, csr=None)[source]

Generates a user keypair.

Parameters:
  • name (str) – The name of the user keypair.
  • signas (str) – The CA keypair to sign the new user keypair with.
  • outp (synapse.lib.output.Output) – The output buffer.
  • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”:

myuserkey, myusercert = cdir.genUserCert(‘myuser’)
Returns:Tuple containing the key and certificate objects.
Return type:((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
genUserCsr(name, outp=None)[source]

Generates a user certificate signing request.

Parameters:
  • name (str) – The name of the user CSR.
  • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the user “myuser”:

cdir.genUserCsr(‘myuser’)
Returns:The bytes of the CSR.
Return type:bytes
getCaCert(name)[source]

Loads the X509 object for a given CA.

Parameters:name (str) – The name of the CA keypair.

Examples

Get the certificate for the CA “myca”

mycacert = cdir.getCaCert(‘myca’)
Returns:The certificate, if exists.
Return type:OpenSSL.crypto.X509
getCaCertPath(name)[source]

Gets the path to a CA certificate.

Parameters:name (str) – The name of the CA keypair.

Examples

Get the path to the CA certificate for the CA “myca”:

mypath = cdir.getCACertPath(‘myca’)
Returns:The path if exists.
Return type:str
getCaCerts()[source]

Return a list of CA certs from the CertDir.

Returns:List of CA certificates.
Return type:[OpenSSL.crypto.X509]
getCaKey(name)[source]

Loads the PKey object for a given CA keypair.

Parameters:name (str) – The name of the CA keypair.

Examples

Get the private key for the CA “myca”:

mycakey = cdir.getCaKey(‘myca’)
Returns:The private key, if exists.
Return type:OpenSSL.crypto.PKey
getCaKeyPath(name)[source]

Gets the path to a CA key.

Parameters:name (str) – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”:

mypath = cdir.getCAKeyPath(‘myca’)
Returns:The path if exists.
Return type:str
getClientCert(name)[source]

Loads the PKCS12 archive object for a given user keypair.

Parameters:name (str) – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”:

mypkcs12 = cdir.getClientCert(‘myuser’)

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool

Returns:The PKCS12 archive, if exists.
Return type:OpenSSL.crypto.PKCS12
getClientCertPath(name)[source]

Gets the path to a client certificate.

Parameters:name (str) – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”:

mypath = cdir.getClientCertPath(‘myuser’)
Returns:The path if exists.
Return type:str
getClientSSLContext(certname=None)[source]

Returns an ssl.SSLContext appropriate for initiating a TLS session

Parameters:certname – If specified, use the user certificate with the matching name to authenticate to the remote service.
getHostCaPath(name)[source]

Gets the path to the CA certificate that issued a given host keypair.

Parameters:name (str) – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”:

mypath = cdir.getHostCaPath(‘myhost’)
Returns:The path if exists.
Return type:str
getHostCert(name)[source]

Loads the X509 object for a given host keypair.

Parameters:name (str) – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”:

myhostcert = cdir.getHostCert(‘myhost’)
Returns:The certificate, if exists.
Return type:OpenSSL.crypto.X509
getHostCertPath(name)[source]

Gets the path to a host certificate.

Parameters:name (str) – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”:

mypath = cdir.getHostCertPath(‘myhost’)
Returns:The path if exists.
Return type:str
getHostKey(name)[source]

Loads the PKey object for a given host keypair.

Parameters:name (str) – The name of the host keypair.

Examples

Get the private key object for the host “myhost”:

myhostkey = cdir.getHostKey(‘myhost’)
Returns:The private key, if exists.
Return type:OpenSSL.crypto.PKey
getHostKeyPath(name)[source]

Gets the path to a host key.

Parameters:name (str) – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”:

mypath = cdir.getHostKeyPath(‘myhost’)
Returns:The path if exists.
Return type:str
getServerSSLContext(hostname=None, caname=None)[source]

Returns an ssl.SSLContext appropriate to listen on a socket

Parameters:
  • hostname – if None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory
  • caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.
getUserCaPath(name)[source]

Gets the path to the CA certificate that issued a given user keypair.

Parameters:name (str) – The name of the user keypair.

Examples

Get the path to the CA cert which issue the cert for “myuser”:

mypath = cdir.getUserCaPath(‘myuser’)
Returns:The path if exists.
Return type:str
getUserCert(name)[source]

Loads the X509 object for a given user keypair.

Parameters:name (str) – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”:

myusercert = cdir.getUserCert(‘myuser’)
Returns:The certificate, if exists.
Return type:OpenSSL.crypto.X509
getUserCertPath(name)[source]

Gets the path to a user certificate.

Parameters:name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”:

mypath = cdir.getUserCertPath(‘myuser’)
Returns:The path if exists.
Return type:str
getUserForHost(user, host)[source]

Gets the name of the first existing user cert for a given user and host.

Parameters:
  • user (str) – The name of the user.
  • host (str) – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”:

usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)
Returns:The cert name, if exists.
Return type:str
getUserKey(name)[source]

Loads the PKey object for a given user keypair.

Parameters:name (str) – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”:

myuserkey = cdir.getUserKey(‘myuser’)
Returns:The private key, if exists.
Return type:OpenSSL.crypto.PKey
getUserKeyPath(name)[source]

Gets the path to a user key.

Parameters:name (str) – The name of the user keypair.

Examples

Get the path to the user key for “myuser”:

mypath = cdir.getUserKeyPath(‘myuser’)
Returns:The path if exists.
Return type:str
importFile(path, mode, outp=None)[source]

Imports certs and keys into the Synapse cert directory

Parameters:
  • path (str) – The path of the file to be imported.
  • mode (str) – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory.

certdir.importFile(‘mycoolca.crt’, ‘cas’)

Notes

importFile does not perform any validation on the files it imports.

Returns:None
isCaCert(name)[source]

Checks if a CA certificate exists.

Parameters:name (str) – The name of the CA keypair.

Examples

Check if the CA certificate for “myca” exists:

exists = cdir.isCaCert(‘myca’)
Returns:True if the certificate is present, False otherwise.
Return type:bool
isClientCert(name)[source]

Checks if a user client certificate (PKCS12) exists.

Parameters:name (str) – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists:

exists = cdir.isClientCert(‘myuser’)
Returns:True if the certificate is present, False otherwise.
Return type:bool
isHostCert(name)[source]

Checks if a host certificate exists.

Parameters:name (str) – The name of the host keypair.

Examples

Check if the host cert “myhost” exists:

exists = cdir.isUserCert(‘myhost’)
Returns:True if the certificate is present, False otherwise.
Return type:bool
isUserCert(name)[source]

Checks if a user certificate exists.

Parameters:name (str) – The name of the user keypair.

Examples

Check if the user cert “myuser” exists:

exists = cdir.isUserCert(‘myuser’)
Returns:True if the certificate is present, False otherwise.
Return type:bool
saveCertPem(cert, path)[source]

Save a certificate in PEM format to a file outside the certdir.

savePkeyPem(pkey, path)[source]

Save a private key in PEM format to a file outside the certdir.

selfSignCert(cert, pkey)[source]

Self-sign a certificate.

Parameters:
  • cert (OpenSSL.crypto.X509) – The certificate to sign.
  • pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key:

cdir.selfSignCert(mycert, myotherprivatekey)
Returns:None
signCertAs(cert, signas)[source]

Signs a certificate with a CA keypair.

Parameters:
  • cert (OpenSSL.crypto.X509) – The certificate to sign.
  • signas (str) – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”:

cdir.signCertAs(mycert, ‘myca’)
Returns:None
signHostCsr(xcsr, signas, outp=None, sans=None)[source]

Signs a host CSR with a CA keypair.

Parameters:
  • cert (OpenSSL.crypto.X509Req) – The certificate signing request.
  • signas (str) – The CA keypair name to sign the CSR with.
  • outp (synapse.lib.output.Output) – The output buffer.
  • sans (list) – List of subject alternative names.

Examples

Sign a host key with the CA “myca”:

cdir.signHostCsr(mycsr, ‘myca’)
Returns:Tuple containing the public key and certificate objects.
Return type:((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
signUserCsr(xcsr, signas, outp=None)[source]

Signs a user CSR with a CA keypair.

Parameters:
  • cert (OpenSSL.crypto.X509Req) – The certificate signing request.
  • signas (str) – The CA keypair name to sign the CSR with.
  • outp (synapse.lib.output.Output) – The output buffer.

Examples

cdir.signUserCsr(mycsr, ‘myca’)

Returns:Tuple containing the public key and certificate objects.
Return type:((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
valUserCert(byts, cacerts=None)[source]

Validate the PEM encoded x509 user certificate bytes and return it.

Parameters:
  • byts (bytes) – The bytes for the User Certificate.
  • cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates.
Raises:

OpenSSL.crypto.X509StoreContextError – If the certificate is not valid.

Returns:

The certificate, if it is valid.

Return type:

OpenSSL.crypto.X509

synapse.lib.certdir.addCertPath(path)[source]
synapse.lib.certdir.delCertPath(path)[source]
synapse.lib.certdir.getCertDir()[source]
synapse.lib.certdir.iterFqdnUp(fqdn)[source]

synapse.lib.chop module

synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)

Shared primitive routines for chopping up strings and values.

synapse.lib.chop.digits(text)[source]
synapse.lib.chop.hexstr(text)[source]

Ensure a string is valid hex.

Parameters:text (str) – String to normalize.

Examples

Norm a few strings:

hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.

Returns:Normalized hex string.
Return type:str
synapse.lib.chop.intstr(text)[source]
synapse.lib.chop.onespace(text)[source]
synapse.lib.chop.printables(text)[source]
synapse.lib.chop.stormstring[source]

Make a string storm safe by escaping backslashes and double quotes.

Parameters:s (str) – String to make storm safe.

Notes

This does not encapsulate a string in double quotes.

Returns:A string which can be embedded directly into a storm query.
Return type:str
synapse.lib.chop.tag[source]
synapse.lib.chop.tagpath[source]
synapse.lib.chop.tags[source]

Divide a normalized tag string into hierarchical layers.

synapse.lib.chop.validateTagMatch(tag)[source]

Raises an exception if tag is not a valid tagmatch (i.e. a tag that might have globs)

synapse.lib.cli module

class synapse.lib.cli.Cli[source]

Bases: synapse.lib.base.Base

A modular / event-driven CLI base object.

addCmdClass(ctor, **opts)[source]

Add a Cmd subclass to this cli.

addSignalHandlers()[source]

Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.

get(name, defval=None)[source]
getCmdByName(name)[source]

Return a Cmd instance by name.

getCmdNames()[source]

Return a list of all the known command names for the CLI.

getCmdPrompt()[source]

Get the command prompt.

Returns:Configured command prompt
Return type:str
histfile = 'cmdr_history'
initCmdClasses()[source]
printf(mesg, addnl=True, color=None)[source]
prompt(text=None)[source]

Prompt for user input from stdin.

runCmdLine(line)[source]

Run a single command line.

Parameters:line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch:

await cli.runCmdLine(‘woot –help’)
Returns:Arbitrary data from the cmd class.
Return type:object
runCmdLoop()[source]

Run commands from a user in an interactive fashion until fini() or EOFError is raised.

set(name, valu)[source]
class synapse.lib.cli.Cmd(cli, **opts)[source]

Bases: object

Base class for modular commands in the synapse CLI.

getCmdBrief()[source]

Return the single-line description for this command.

getCmdDoc()[source]

Return the help/doc output for this command.

getCmdItem()[source]

Get a reference to the object we are commanding.

getCmdName()[source]
getCmdOpts(text)[source]

Use the _cmd_syntax def to split/parse/normalize the cmd line.

Parameters:text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type

Returns:An opts dictionary.
Return type:dict
printf(mesg, addnl=True, color=None)[source]
runCmdLine(line)[source]

Run a line of command input for this command.

Parameters:line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)
runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:opts (dict) – Options dictionary.
class synapse.lib.cli.CmdHelp(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

List commands and display help output.

Example

help foocmd

runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:opts (dict) – Options dictionary.
class synapse.lib.cli.CmdLocals(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

List the current locals for a given CLI object.

runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:opts (dict) – Options dictionary.
class synapse.lib.cli.CmdQuit(cli, **opts)[source]

Bases: synapse.lib.cli.Cmd

Quit the current command line interpreter.

Example

quit

runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:opts (dict) – Options dictionary.

synapse.lib.cmd module

class synapse.lib.cmd.Parser(prog=None, outp=<synapse.lib.output.OutPut object>, **kwargs)[source]

Bases: argparse.ArgumentParser

exit(status=0, message=None)[source]

Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead.

synapse.lib.cmdr module

synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)[source]

Construct and return a cmdr for the given remote cell.

Parameters:
  • cell – Cell proxy being commanded.
  • outp – Output helper object.
  • color (bool) – If true, enable colorized output.
  • **opts – Additional options pushed into the Cmdr locs.

Examples

Get the cmdr for a proxy:

cmdr = await getItemCmdr(foo)
Returns:A Cli instance with Cmds loaeded into it.
Return type:s_cli.Cli
synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)[source]

Create a cmdr for the given item and run the cmd loop.

Parameters:
  • item – Cell proxy being commanded.
  • outp – Output helper object.
  • color (bool) – If true, enable colorized output.
  • **opts – Additional options pushed into the Cmdr locs.

Notes

This function does not return while the command loop is run.

Examples

Run the Cmdr for a proxy:

await runItemCmdr(foo)
Returns:This function returns None.
Return type:None

synapse.lib.config module

class synapse.lib.config.Config(schema, conf=None, envar_prefix=None)[source]

Bases: collections.abc.MutableMapping

Synapse configuration helper based on JSON Schema.

Parameters:
  • schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.
  • conf (dict) – Optional, a set of configuration data to preload.
  • envar_prefix (str) – Optional, a prefix used when collecting configuration data from environment variables.

Notes

This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.

The default values provided in the schema must be able to be recreated from the repr() of their Python value.

Default values are not loaded into the configuration data until the reqConfValid() method is called.

asDict()[source]

Get a copy of configuration data.

Returns:A copy of the configuration data.
Return type:dict
getArgParseArgs()[source]
getCmdlineMapping()[source]
classmethod getConfFromCell(cell, conf=None, envar_prefix=None)[source]

Get a Config object from a Cell directly (either the ctor or the instance thereof).

Returns:A Config object.
Return type:Config
getEnvarMapping()[source]

Get a mapping of config values to envars.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

reqConfValid()[source]

Validate that the loaded configuration data is valid according to the schema.

Notes

The validation set does set any default values which are not currently set for configuration options.

Returns:This returns nothing.
Return type:None
reqConfValu(key)[source]

Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.

Parameters:key (str) – The key to require.
Returns:The requested value.
setConfFromEnvs()[source]

Set configuration options from environment variables.

Notes

Environment variables are resolved from configuration options after doing the following transform:

  • Replace : characters with _.
  • Add a config provided prefix, if set.
  • Uppercase the string.
  • Resolve the environment variable
  • If the environment variable is set, set the config value to the results of yaml.yaml_safeload() on the value.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

Examples

For the configuration value auth:passwd, the environment variable is resolved as AUTH_PASSWD. With the prefix cortex, the the environment variable is resolved as CORTEX_AUTH_PASSWD.

Returns:Returns None.
Return type:None
setConfFromOpts(opts)[source]

Set the opts for a conf object from a namespace object.

Parameters:
  • opts (argparse.Namespace) – A Namespace object made from parsing args with an ArgumentParser
  • with getArgumentParser. (made) –
Returns:

Returns None.

Return type:

None

synapse.lib.config.getJsSchema(confbase, confdefs)[source]

Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.

Parameters:
  • confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.
  • confdefs (dict) – A JSON Schema dictionary of properties for the object.

Notes

This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify

Returns:A complete JSON schema.
Return type:dict
synapse.lib.config.getJsValidator(schema)[source]

Get a fastjsonschema callable.

Parameters:schema (dict) – A JSON Schema object.
Returns:A callable function that can be used to validate data against the json schema.
Return type:callable
synapse.lib.config.make_envar_name(key, prefix=None)[source]

Convert a colon delimited string into an uppercase, underscore delimited string.

Parameters:
  • key (str) – Config key to convert.
  • prefix (str) – Optional string prefix to prepend the the config key.
Returns:

The string to lookup against a envar.

Return type:

str

synapse.lib.const module

synapse.lib.coro module

Async/Coroutine related utilities.

class synapse.lib.coro.GenrHelp(genr)[source]

Bases: object

list()[source]
spin()[source]
synapse.lib.coro.agen(item)[source]

Wrap an async_generator or generator in an async_generator.

Notes

Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.

synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None)[source]

Wait on an an asyncio event with an optional timeout

Returns:true if the event got set, False if timed out
synapse.lib.coro.executor(func, *args, **kwargs)[source]

Execute a non-coroutine function in the ioloop executor pool.

Parameters:
  • func – Function to execute.
  • *args – Args for the function.
  • **kwargs – Kwargs for the function.

Examples

Execute a blocking API call in the executor pool:

import requests

def block(url, params=None):
    return requests.get(url, params=params).json()

fut = s_coro.executor(block, 'http://some.tld/thign')
resp = await fut
Returns:An asyncio future.
Return type:asyncio.Future
synapse.lib.coro.forked(func, *args, **kwargs)[source]
synapse.lib.coro.genrhelp(f)[source]
synapse.lib.coro.iscoro(item)[source]
synapse.lib.coro.ornot(func, *args, **kwargs)[source]

Calls func and awaits it if a returns a coroutine.

Note

This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.

This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage:
ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)
synapse.lib.coro.set_pool_logging(logger_, logconf)[source]
synapse.lib.coro.spawn(todo, timeout=None, ctx=None)[source]

Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.

Parameters:
  • todo (tuple) – A tuple of function, *args, and **kwargs.
  • timeout (int) – The timeout to wait for the todo function to finish.
  • ctx (multiprocess.Context) – A optional multiprocessing context object.

Notes

The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.

Returns:The return value of executing the todo function.
synapse.lib.coro.waittask(task, timeout=None)[source]

Await a task without cancelling it when you time out.

Returns:True if the task completed before the timeout.
Return type:boolean

synapse.lib.datfile module

Utilities for handling data files embedded within python packages.

synapse.lib.datfile.openDatFile(datpath)[source]

Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.dyndeps module

synapse.lib.dyndeps.getDynLocal(name)[source]

Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()

synapse.lib.dyndeps.getDynMeth(name)[source]

Retrieve and return an unbound method by python path.

synapse.lib.dyndeps.getDynMod(name)[source]

Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’)

synapse.lib.dyndeps.runDynTask(task)[source]

Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)[source]

Dynamically import a module and call a function or raise an exception.

synapse.lib.dyndeps.tryDynLocal(name)[source]

Dynamically import a module and return a module local or raise an exception.

synapse.lib.dyndeps.tryDynMod(name)[source]

Dynamically import a python module or exception.

synapse.lib.encoding module

synapse.lib.encoding.addFormat(name, fn, opts)[source]

Add an additional ingest file format

synapse.lib.encoding.decode(name, byts, **opts)[source]

Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with +
to encode for that name/layer.
synapse.lib.encoding.encode(name, item, **opts)[source]
synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)[source]

Iterate through the data provided by a file like object.

Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd:
    for row in iterdata(fd, format='csv', encoding='utf8'):
        dostuff(row)
Parameters:
  • fd (file) – File like object to iterate over.
  • close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.
  • **opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.
Yields:

An item to process. The type of the item is dependent on the format parameters.

synapse.lib.gis module

synapse.lib.gis.bbox(lat, lon, dist)[source]

Calculate a min/max bounding box for the circle defined by lalo/dist.

Parameters:
  • lat (float) – The latitude in degrees
  • lon (float) – The longitude in degrees
  • dist (int) – A distance in geo:dist base units (mm)
Returns:

(latmin, latmax, lonmin, lonmax)

Return type:

(float,float,float,float)

synapse.lib.gis.dms2dec(degs, mins, secs)[source]

Convert degrees, minutes, seconds lat/long form to degrees float.

Parameters:
  • degs (int) – Degrees
  • mins (int) – Minutes
  • secs (int) – Seconds
Returns:

Degrees

Return type:

(float)

synapse.lib.gis.haversine(px, py, r=6371008800.0)[source]

Calculate the haversine distance between two points defined by (lat,lon) tuples.

Parameters:
  • px ((float,float)) – lat/long position 1
  • py ((float,float)) – lat/long position 2
  • r (float) – Radius of sphere
Returns:

Distance in mm.

Return type:

(int)

synapse.lib.gis.latlong(text)[source]

Chop a latlong string and return (float,float). Does not perform validation on the coordinates.

Parameters:text (str) – A longitude,latitude string.
Returns:A longitude, latitude float tuple.
Return type:(float,float)
synapse.lib.gis.near(point, dist, points)[source]

Determine if the given point is within dist of any of points.

Parameters:
  • point ((float,float)) – A latitude, longitude float tuple.
  • dist (int) – A distance in mm ( base units )
  • points (list) – A list of latitude, longitude float tuples to compare against.

synapse.lib.grammar module

synapse.lib.grammar.isCmdName(name)[source]
synapse.lib.grammar.isFormName(name)[source]
synapse.lib.grammar.isPropName(name)[source]
synapse.lib.grammar.isUnivName(name)[source]
synapse.lib.grammar.meh(txt, off, cset)[source]
synapse.lib.grammar.nom(txt, off, cset, trim=True)[source]

Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’)

name,off = nom(text,0,chars)

Note:

This really shouldn’t be used for new code

synapse.lib.grammar.parse_float(text, off)[source]

synapse.lib.hashitem module

synapse.lib.hashitem.hashitem(item)[source]

Generate a uniq hash for the JSON compatible primitive data structure.

synapse.lib.hashitem.normdict(item)[source]
synapse.lib.hashitem.normitem(item)[source]
synapse.lib.hashitem.normiter(item)[source]

synapse.lib.hashset module

class synapse.lib.hashset.HashSet[source]

Bases: object

digests()[source]

Get a list of (name, bytes) tuples for the hashes in the hashset.

eatfd(fd)[source]

Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd)

guid()[source]

Use elements from this hash set to create a unique (re)identifier.

update(byts)[source]

Update all the hashes in the set with the given bytes.

synapse.lib.health module

class synapse.lib.health.HealthCheck(iden)[source]

Bases: object

getStatus()[source]
pack()[source]
setStatus(valu)[source]
update(name, status, mesg='', data=None)[source]

Append a new component to the Healcheck object.

Parameters:
  • name (str) – Name of the reported component.
  • status (str) – nomdinal/degraded/failed status code.
  • mesg (str) – Optional message about the component status.
  • data (dict) – Optional arbitrary dictionary of additional metadata about the component.
Returns:

None

synapse.lib.hive module

class synapse.lib.hive.Hive[source]

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.

add(full, valu)[source]

Atomically increments a node’s value.

dict(full, nexs=False)[source]

Open a HiveDict at the given full path.

Parameters:full (tuple) – A full path tuple.
Returns:A HiveDict for the full path.
Return type:HiveDict
dir(full)[source]

List subnodes of the given Hive path.

Parameters:full (tuple) – A full path tuple.

Notes

This returns None if there is not a node at the path.

Returns:A list of tuples. Each tuple contains the name, node value, and the number of children nodes.
Return type:list
exists(full)[source]

Returns whether the Hive path has already been created.

get(full, defv=None)[source]

Get the value of a node at a given path.

Parameters:full (tuple) – A full path tuple.
Returns:Arbitrary node value.
getHiveAuth()[source]

Retrieve a HiveAuth for hive standalone or non-cell uses.

Note

This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell

getTeleApi(link, mesg, path)[source]

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

loadHiveTree(tree, path=(), trim=False)[source]
open(full)[source]

Open and return a hive Node().

Parameters:full (tuple) – A full path tuple.
Returns:A Hive node.
Return type:Node
pop(full, nexs=False)[source]

Remove and return the value for the given node.

rename(oldpath, newpath)[source]

Moves a node at oldpath and all its descendant nodes to newpath. newpath must not exist

saveHiveTree(path=())[source]
set(full, valu, nexs=False)[source]

A set operation at the hive level (full path).

storNodeDele(path)[source]
storNodeValu(full, valu)[source]
class synapse.lib.hive.HiveApi[source]

Bases: synapse.lib.base.Base

addAndSync(path, valu, iden)[source]
edits()[source]
get(full)[source]
loadHiveTree(tree, path=(), trim=False)[source]
popAndSync(path, iden, nexs=False)[source]
saveHiveTree(path=())[source]
setAndSync(path, valu, iden, nexs=False)[source]
treeAndSync(path, iden)[source]
class synapse.lib.hive.HiveDict[source]

Bases: synapse.lib.base.Base

get(name, default=None)[source]
items()[source]
pack()[source]
pop(name, default=None)[source]
set(name, valu)[source]
setdefault(name, valu)[source]
values()[source]
class synapse.lib.hive.Node[source]

Bases: synapse.lib.base.Base

A single node within the Hive tree.

add(valu)[source]

Increments existing node valu

dict(nexs=False)[source]

Get a HiveDict for this Node.

Returns:A HiveDict for this Node.
Return type:HiveDict
dir()[source]
get(name)[source]
name()[source]
open(path)[source]

Open a child Node of the this Node.

Parameters:path (tuple) – A child path of the current node.
Returns:A Node at the child path.
Return type:Node
parent()[source]
pop(path=())[source]
set(valu)[source]
class synapse.lib.hive.SlabHive[source]

Bases: synapse.lib.hive.Hive

storNodeDele(full)[source]
storNodeValu(full, valu)[source]
class synapse.lib.hive.TeleHive[source]

Bases: synapse.lib.hive.Hive

A Hive that acts as a consistent read cache for a telepath proxy Hive

add(path, valu)[source]

Atomically increments a node’s value.

get(path)[source]

Get the value of a node at a given path.

Parameters:full (tuple) – A full path tuple.
Returns:Arbitrary node value.
open(path)[source]

Open and return a hive Node().

Parameters:full (tuple) – A full path tuple.
Returns:A Hive node.
Return type:Node
pop(path, nexs=False)[source]

Remove and return the value for the given node.

set(path, valu, nexs=False)[source]

A set operation at the hive level (full path).

synapse.lib.hive.iterpath(path)[source]
synapse.lib.hive.opendir(dirn, conf=None)[source]
synapse.lib.hive.openurl(url, **opts)[source]

synapse.lib.hiveauth module

class synapse.lib.hiveauth.Auth[source]

Bases: synapse.lib.nexus.Pusher

Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.

Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).

Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.

Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.

Node layout:

Auth root (passed into constructor)
├ roles
│   ├ <role iden 1>
│   ├ ...
│   └ last role
├ users
│   ├ <user iden 1>
│   ├ ...
│   └ last user
└ authgates
    ├ <iden 1>
    │   ├ roles
    │   │   ├ <role iden 1>
    │   │   ├ ...
    │   │   └ last role
    │   └ users
    │       ├ <user iden 1>
    │       ├ ...
    │       └ last user
    ├ <iden 2>
    │   ├ ...
    └ ... last authgate
addAuthGate(iden, authgatetype)[source]

Retrieve AuthGate by iden. Create if not present.

Note

Not change distributed

Returns:(HiveAuthGate)
addRole(name)[source]
addUser(name, passwd=None, email=None, iden=None)[source]

Add a User to the Hive.

Parameters:
  • name (str) – The name of the User.
  • passwd (str) – A optional password for the user.
  • email (str) – A optional email for the user.
  • iden (str) – A optional iden to use as the user iden.
Returns:

A Hive User.

Return type:

HiveUser

delAuthGate(iden)[source]

Delete AuthGate by iden.

Note

Not change distributed

delRole(iden)[source]
delUser(iden)[source]
getAuthGate(iden)[source]
getAuthGates()[source]
getRoleByName(name)[source]
getUserByName(name)[source]

Get a user by their username.

Parameters:name (str) – Name of the user to get.
Returns:A Hive User. May return None if there is no user by the requested name.
Return type:HiveUser
getUserIdenByName(name)[source]
reqAuthGate(iden)[source]
reqRole(iden)[source]
reqRoleByName(name)[source]
reqUser(iden)[source]
reqUserByName(name)[source]
role(iden)[source]
roles()[source]
setRoleInfo(iden, name, valu, gateiden=None)[source]
setRoleName(iden, name)[source]
setUserInfo(iden, name, valu, gateiden=None)[source]
setUserName(iden, name)[source]
user(iden)[source]
users()[source]
class synapse.lib.hiveauth.AuthGate[source]

Bases: synapse.lib.base.Base

The storage object for object specific rules for users/roles.

delete()[source]
genRoleInfo(iden)[source]
genUserInfo(iden)[source]
pack()[source]
class synapse.lib.hiveauth.HiveRole[source]

Bases: synapse.lib.hiveauth.HiveRuler

A role within the Hive authorization subsystem.

A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.

clearAuthCache()[source]
genGateInfo(gateiden)[source]
pack()[source]
setName(name)[source]
class synapse.lib.hiveauth.HiveRuler[source]

Bases: synapse.lib.base.Base

A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those

addRule(rule, indx=None, gateiden=None, nexs=True)[source]
delRule(rule, gateiden=None)[source]
getRules(gateiden=None)[source]
setRules(rules, gateiden=None, nexs=True)[source]
class synapse.lib.hiveauth.HiveUser[source]

Bases: synapse.lib.hiveauth.HiveRuler

A user (could be human or computer) of the system within HiveAuth.

Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.

allowed(perm, default=None, gateiden=None)[source]
clearAuthCache()[source]
confirm(perm, default=None, gateiden=None)[source]
genGateInfo(gateiden)[source]
getRoles()[source]
grant(roleiden, indx=None)[source]
hasRole(iden)[source]
isAdmin(gateiden=None)[source]
isLocked()[source]
pack(packroles=False)[source]
raisePermDeny(perm, gateiden=None)[source]
revoke(iden, nexs=True)[source]
setAdmin(admin, gateiden=None, logged=True)[source]
setArchived(archived)[source]
setLocked(locked, logged=True)[source]
setName(name)[source]
setPasswd(passwd, nexs=True)[source]
setRoles(roleidens)[source]

Replace all the roles for a given user with a new list of roles.

Parameters:roleidens (list) – A list of roleidens.

Notes

The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.

Returns:None
tryPasswd(passwd)[source]
synapse.lib.hiveauth.getShadow(passwd)[source]

synapse.lib.httpapi module

class synapse.lib.httpapi.ActiveV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.AuthAddRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.AuthAddUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.AuthDelRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.AuthGrantV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

get()[source]
post()[source]
class synapse.lib.httpapi.AuthRevokeV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

get()[source]
post()[source]
class synapse.lib.httpapi.AuthRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get(iden)[source]
post(iden)[source]
class synapse.lib.httpapi.AuthRolesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.AuthUserPasswdV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post(iden)[source]
class synapse.lib.httpapi.AuthUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get(iden)[source]
post(iden)[source]
class synapse.lib.httpapi.AuthUsersV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.CoreInfoV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/core/info

get()[source]
class synapse.lib.httpapi.FeedV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/feed

{
‘name’: ‘syn.nodes’, ‘view’: null, ‘items’: […],

}

post()[source]
class synapse.lib.httpapi.Handler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

on_connection_close()[source]

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

prepare()[source]

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

class synapse.lib.httpapi.HandlerBase[source]

Bases: object

allowed(perm, gateiden=None)[source]

Return true if there is a logged in user with the given permissions.

NOTE: This API sets up HTTP response values if it returns False.

NOTE: This API uses the Handler.getAuthCell() abstraction and is safe for use
in split-auth cells.
authenticated()[source]
check_origin(origin)[source]
getAuthCell()[source]

Return a reference to the cell used for auth operations.

getCustomHeaders()[source]
getJsonBody(validator=None)[source]
getUserBody()[source]

Helper function to confirm that there is a auth user and a valid JSON body in the request.

Returns:
The user and body of the request as deserialized JSON, or a tuple of s_common.novalu
objects if there was no user or json body.
Return type:(User, object)
initialize(cell)[source]
isOrigHost(origin)[source]
loadJsonMesg(byts, validator=None)[source]
options()[source]
reqAuthAdmin()[source]
reqAuthUser()[source]
sendAuthRequired()[source]
sendRestErr(code, mesg)[source]
sendRestExc(e)[source]
sendRestRetn(valu)[source]
sess(gen=True)[source]
set_default_headers()[source]
user()[source]
useriden()[source]

Return the user iden of the current session user.

NOTE: APIs should migrate toward using this rather than the heavy
Handler.user() API to facilitate reuse of handler objects with telepath references.
class synapse.lib.httpapi.HealthCheckV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.LoginV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.ModelNormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
post()[source]
class synapse.lib.httpapi.ModelV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.OnePassIssueV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/onepass/issue

post()[source]
class synapse.lib.httpapi.ReqValidStormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.StormHandler

get()[source]
post()[source]
class synapse.lib.httpapi.RobotHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

get()[source]
class synapse.lib.httpapi.Sess[source]

Bases: synapse.lib.base.Base

addWebSock(sock)[source]
delWebSock(sock)[source]
login(user)[source]
logout()[source]
class synapse.lib.httpapi.StormCallV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.StormHandler

get()[source]
post()[source]
class synapse.lib.httpapi.StormExportV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.StormHandler

get()[source]
post()[source]
class synapse.lib.httpapi.StormHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

getCore()[source]
class synapse.lib.httpapi.StormNodesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.StormHandler

get()[source]
post()[source]
class synapse.lib.httpapi.StormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.StormHandler

get()[source]
post()[source]
class synapse.lib.httpapi.StormVarsGetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

get()[source]
class synapse.lib.httpapi.StormVarsPopV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.StormVarsSetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

post()[source]
class synapse.lib.httpapi.StreamHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.Handler

Subclass for Tornado streaming uploads.

Notes

  • Async method prepare() is called after headers are read but before body processing.
  • Sync method on_finish() can be used to cleanup after a request.
  • Sync method on_connection_close() can be used to cleanup after a client disconnect.
  • Async methods post(), put(), etc are called after the streaming has completed.
data_received(chunk)[source]

Implement this method to handle streamed request data.

Requires the .stream_request_body decorator.

May be a coroutine for flow control.

class synapse.lib.httpapi.WatchSockV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.WebSocket

A web-socket based API endpoint for distributing cortex events.

onWatchMesg(byts)[source]
on_message(byts)[source]

Handle incoming messages on the WebSocket

This method must be overridden.

Changed in version 4.5: on_message can be a coroutine.

class synapse.lib.httpapi.WebSocket(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs)[source]

Bases: synapse.lib.httpapi.HandlerBase, tornado.websocket.WebSocketHandler

xmit(name, **info)[source]

synapse.lib.ingest module

synapse.lib.interval module

A few utilities for dealing with intervals.

synapse.lib.interval.fold(*vals)[source]

Initialize a new (min,max) tuple interval from values.

Parameters:*vals ([int,..]) – A list of values (or Nones)
Returns:A (min,max) interval tuple or None
Return type:((int,int))
synapse.lib.interval.overlap(ival0, ival1)[source]

Determine if two interval tuples have overlap.

Parameters:
  • iv0 ((int,int)) – An interval tuple
  • iv1 ((int,int)) –
Returns:

True if the intervals overlap, otherwise False

Return type:

(bool)

synapse.lib.interval.parsetime(text)[source]

Parse an interval time string and return a (min,max) tuple.

Parameters:text (str) – A time interval string
Returns:A epoch millis epoch time string
Return type:((int,int))

synapse.lib.jsonstor module

class synapse.lib.jsonstor.JsonStor[source]

Bases: synapse.lib.base.Base

A filesystem like storage mechanism that allows hirarchical lookup of reference counted “objects” that have individually editable properties.

#TODO json validation by path glob matches? (persists?) #TODO GUID ACCESS with index generation by type #TODO registered types jsonschema with optional write-back validation

cmpDelPathObjProp(path, prop, valu)[source]
delPathObj(path)[source]

Remove a path and decref the object it references.

delPathObjProp(path, prop)[source]
getPathList(path)[source]
getPathObj(path)[source]
getPathObjProp(path, prop)[source]
getPathObjs(path)[source]

Add a link from the given srcpath to the dstpath. NOTE: This causes the item at dstpath to be incref’d

setPathObj(path, item)[source]

Set (and/or reinitialize) the object at the given path.

NOTE: This will break any links by creating a new object.

setPathObjProp(path, prop, valu)[source]
class synapse.lib.jsonstor.JsonStorApi[source]

Bases: synapse.lib.cell.CellApi

addQueue(name, info)[source]
cmpDelPathObjProp(path, name, valu)[source]
cullQueue(name, offs)[source]
delPathObj(path)[source]
delPathObjProp(path, name)[source]
delQueue(name)[source]
getPathList(path)[source]
getPathObj(path)[source]
getPathObjProp(path, prop)[source]
getPathObjs(path)[source]
getsQueue(name, offs, size=None, cull=True, wait=True)[source]
putsQueue(name, items)[source]
setPathObj(path, item)[source]
setPathObjProp(path, prop, valu)[source]
class synapse.lib.jsonstor.JsonStorCell[source]

Bases: synapse.lib.cell.Cell

addQueue(name, info)[source]
cellapi

alias of JsonStorApi

cmpDelPathObjProp(path, name, valu)[source]
cullQueue(name, offs)[source]
delPathObj(path)[source]
delPathObjProp(path, name)[source]
delQueue(name)[source]
getPathList(path)[source]
getPathObj(path)[source]
getPathObjProp(path, prop)[source]
getPathObjs(path)[source]
getsQueue(name, offs, size=None, cull=True, wait=True)[source]
initServiceStorage()[source]
putsQueue(name, items, reqid=None)[source]
setPathObj(path, item)[source]
setPathObjProp(path, prop, valu)[source]

synapse.lib.jupyter module

class synapse.lib.jupyter.CmdrCore[source]

Bases: synapse.lib.base.Base

A helper for jupyter/storm CLI interaction

addFeedData(name, items, *, viewiden=None)[source]

Add feed data to the cortex.

eval(text, opts=None, num=None, cmdr=False)[source]

A helper for executing a storm command and getting a list of packed nodes.

Parameters:
  • text (str) – Storm command to execute.
  • opts (dict) – Opt to pass to the cortex during execution.
  • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.
  • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True.

Returns:A list of packed nodes.
Return type:list
runCmdLine(text)[source]

Run a line of text directly via cmdr.

storm(text, opts=None, num=None, cmdr=False, suppress_logging=False)[source]

A helper for executing a storm command and getting a list of storm messages.

Parameters:
  • text (str) – Storm command to execute.
  • opts (dict) – Opt to pass to the cortex during execution.
  • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.
  • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.
  • suppress_logging (bool) – If True, suppresses some logging related to Storm runtime exceptions.

Notes

The opts dictionary will not be used if cmdr=True.

Returns:A list of storm messages.
Return type:list
suppress_logging(suppress)[source]

Context manager to suppress specific loggers.

synapse.lib.jupyter.genTempCoreProxy(mods=None)[source]

Get a temporary cortex proxy.

synapse.lib.jupyter.genTempStormsvcProxy(cmdrcore, svcname, svcctor, conf=None)[source]
synapse.lib.jupyter.getDocData(fp, root=None)[source]
Parameters:
  • fn (str) – Name of the file to retrieve the data of.
  • root (str) – Optional root path to look for a docdata directory in.

Notes

Will detect json/jsonl/yaml/mpk extensions and automatically decode that data if found; otherwise it returns bytes.

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns:May be deserialized data or bytes.
Return type:data
Raises:ValueError if the file does not exist or directory traversal attempted..
synapse.lib.jupyter.getDocPath(fn, root=None)[source]

Helper for getting a documentation data file paths.

Parameters:
  • fn (str) – Name of the file to retrieve the full path for.
  • root (str) – Optional root path to look for a docdata in.

Notes

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns:A file path.
Return type:str
Raises:ValueError if the file does not exist or directory traversal attempted..
synapse.lib.jupyter.getItemCmdr(prox, outp=None, locs=None)[source]

Get a Cmdr instance with prepopulated locs

synapse.lib.jupyter.getTempCoreCmdr(mods=None, outp=None)[source]

Get a CmdrCore instance which is backed by a temporary Cortex.

Parameters:
  • mods (list) – A list of additional CoreModules to load in the Cortex.
  • outp – A output helper. Will be used for the Cmdr instance.

Notes

The CmdrCore returned by this should be fini()’d to tear down the temporary Cortex.

Returns:A CmdrCore instance.
Return type:CmdrCore
synapse.lib.jupyter.getTempCoreCmdrStormsvc(svcname, svcctor, svcconf=None, outp=None)[source]

Get a proxy to a Storm service and a CmdrCore instance backed by a temporary Cortex with the service added.

Parameters:
  • svcname (str) – Storm service name
  • svcctor – Storm service constructor (e.g. Example.anit)
  • svcconf – Optional conf for the Storm service
  • outp – A output helper for the Cmdr instance

Notes

Both the CmdrCore and Storm service proxy should be fini()’d for proper teardown

Returns:A CmdrCore instance and proxy to the Storm service
Return type:(CmdrCore, Proxy)
synapse.lib.jupyter.getTempCoreProx(mods=None)[source]

Get a Telepath Proxt to a Cortex instance which is backed by a temporary Cortex.

Parameters:mods (list) – A list of additional CoreModules to load in the Cortex.

Notes

The Proxy returned by this should be fini()’d to tear down the temporary Cortex.

Returns:s_telepath.Proxy

synapse.lib.layer module

The Layer 2.0 archtecture introduces several optimized node/message serialization formats used by the layers to optimize returning primitives and facilitate efficient node construction:

Note

This interface is subject to change between minor revisions.

Storage Types (<stortype>)

In Layers 2.0, each node property from the model has an associated “storage type”. Each storage type determines how the data is indexed and represented within the Layer. This formalizes the separation of “data model” from “storage model”. Each data model type has a “stortype” property which coresponds to one of the STOR_TYPE_XXX values. The knowledge of the mapping of data model types to storage types is the responsibility of the data model, making the Layer implementation fully decoupled from the data model.

Node Edits / Edits

A node edit consists of a (<buid>, <form>, [edits]) tuple. An edit is Tuple of (<type>, <info>, List[NodeEdits]) where the first element is an int that matches to an EDIT_* constant below, the info is a tuple that varies depending on the first element, and the third element is a list of dependent NodeEdits that will only be applied if the edit actually makes a change.

Storage Node (<sode>)

A storage node is a layer/storage optimized node representation which is similar to a “packed node”. A storage node may be partial ( as it is produced by a given layer ) and are joined by the view/snap into “full” storage nodes which are used to construct Node() instances.

Sode format:

(<buid>, {

    'ndef': (<formname>, <formvalu>),

    'props': {
        <propname>: <propvalu>,
    }

    'tags': {
        <tagname>: <tagvalu>,
    }

    'tagprops: {
        <tagname>: {
            <propname>: <propvalu>,
        },
    }

    # changes that were *just* made.
    'edits': [
        <edit>
    ]

}),
class synapse.lib.layer.IndxBy(layr, abrv, db)[source]

Bases: object

IndxBy sub-classes encapsulate access methods and encoding details for various types of properties within the layer to be lifted/compared by storage types.

buidsByDups(indx)[source]
buidsByPref(indx=b'')[source]
buidsByRange(minindx, maxindx)[source]
buidsByRangeBack(minindx, maxindx)[source]
getNodeValu(buid)[source]
keyBuidsByDups(indx)[source]
keyBuidsByPref(indx=b'')[source]
keyBuidsByRange(minindx, maxindx)[source]
keyBuidsByRangeBack(minindx, maxindx)[source]

Yields backwards from maxindx to minindx

scanByDups(indx)[source]
scanByPref(indx=b'')[source]
scanByRange(minindx, maxindx)[source]
class synapse.lib.layer.IndxByForm(layr, form)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByProp(layr, form, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByPropArray(layr, form, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.IndxByTag(layr, form, tag)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValuForm(buid)[source]
class synapse.lib.layer.IndxByTagProp(layr, form, tag, prop)[source]

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)[source]
class synapse.lib.layer.Layer[source]

Bases: synapse.lib.nexus.Pusher

The base class for a cortex layer.

clone(newdirn)[source]

Copy the contents of this layer to a new layer

delete()[source]

Delete the underlying storage

getAbrvProp(abrv)[source]
getEdgeVerbs()[source]
getEdges(verb=None)[source]
getEditIndx()[source]

Returns what will be the next (i.e. 1 past the last) nodeedit log index.

getEditOffs()[source]

Return the offset of the last recorded log entry. Returns -1 if nodeedit log is disabled or empty.

getFormCounts()[source]
getLayerSize()[source]

Get the total storage size for the layer.

getModelVers()[source]
getNodeData(buid, name)[source]

Return a single element of a buid’s node data

getNodeEditWindow()[source]
getNodeForm(buid)[source]
getNodeTag(buid, tag)[source]
getNodeValu(buid, prop=None)[source]

Retrieve either the form valu or a prop valu for the given node by buid.

getPropAbrv(form, prop)[source]
getPropCount(formname, propname=None, maxsize=None)[source]

Return the number of property rows in the layer for the given form/prop.

getStorIndx(stortype, valu)[source]
getStorNode(buid)[source]
getStorNodes()[source]

Yield (buid, sode) tuples for all the nodes with props/tags/tagprops stored in this layer.

getTagCount(tagname, formname=None)[source]

Return the number of tag rows in the layer for the given tag/form.

getTagPropAbrv(*args)[source]
getUnivPropCount(propname, maxsize=None)[source]

Return the number of universal property rows in the layer for the given prop.

hasNodeData(buid, name)[source]
hasTagProp(name)[source]
initUpstreamSync(url)[source]
iterFormRows(form, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes of a single form, optionally (re)starting at startvalu.

Parameters:
  • form (str) – A form name.
  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.
Returns:

AsyncIterator[Tuple(buid, valu)]

iterLayerNodeEdits()[source]

Scan the full layer and yield artificial sets of nodeedits.

iterNodeData(buid)[source]

Return a generator of all a buid’s node data

iterNodeEdgesN1(buid, verb=None)[source]
iterNodeEdgesN2(buid, verb=None)[source]
iterNodeEditLog(offs=0)[source]

Iterate the node edit log and yield (offs, edits, meta) tuples.

iterNodeEditLogBack(offs=0)[source]

Iterate the node edit log and yield (offs, edits, meta) tuples in reverse.

iterPropRows(form, prop, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes with a particular secondary property, optionally (re)starting at startvalu.

Parameters:
  • form (str) – A form name.
  • prop (str) – A universal property name.
  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.
Returns:

AsyncIterator[Tuple(buid, valu)]

iterTagPropRows(tag, prop, form=None, stortype=None, startvalu=None)[source]

Yields (buid, valu) that match a tag:prop, optionally (re)starting at startvalu.

Parameters:
  • tag (str) – tag name
  • prop (str) – prop name
  • form (Optional[str]) – optional form name
  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.
Returns:

AsyncIterator[Tuple(buid, valu)]

iterTagRows(tag, form=None, starttupl=None)[source]

Yields (buid, (valu, form)) values that match a tag and optional form, optionally (re)starting at starttupl.

Parameters:
  • tag (str) – the tag to match
  • form (Optional[str]) – if present, only yields buids of nodes that match the form.
  • starttupl (Optional[Tuple[buid, form]]) – if present, (re)starts the stream of values there.
Returns:

AsyncIterator[Tuple(buid, (valu, form))]

Note

This yields (buid, (tagvalu, form)) instead of just buid, valu in order to allow resuming an interrupted call by feeding the last value retrieved into starttupl

iterUnivRows(prop, stortype=None, startvalu=None)[source]

Yields buid, valu tuples of nodes with a particular universal property, optionally (re)starting at startvalu.

Parameters:
  • prop (str) – A universal property name.
  • stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop
  • startvalu (Any) – The value to start at. May only be not None if stortype is not None.
Returns:

AsyncIterator[Tuple(buid, valu)]

liftByDataName(name)[source]
liftByFormValu(form, cmprvals)[source]
liftByProp(form, prop)[source]
liftByPropArray(form, prop, cmprvals)[source]
liftByPropValu(form, prop, cmprvals)[source]
liftByTag(tag, form=None)[source]
liftByTagProp(form, tag, prop)[source]
liftByTagPropValu(form, tag, prop, cmprvals)[source]

Note: form may be None

liftByTagValu(tag, cmpr, valu, form=None)[source]
liftTagProp(name)[source]

Note

This will lift all syn:tag nodes.

makeSplices(offs, nodeedits, meta, reverse=False)[source]

Flatten a set of nodeedits into splices.

mayDelBuid(buid, sode)[source]
nodeeditctor

alias of synapse.lib.slabseqn.SlabSeqn

pack()[source]
setLayerInfo(name, valu)[source]

Set a mutable layer property.

setModelVers(vers)[source]
setPropAbrv(form, prop)[source]
setSodeDirty(buid, sode, form)[source]
setTagPropAbrv(*args)[source]
splices(offs=None, size=None)[source]

Yield (offs, splice) tuples from the nodeedit log starting from the given offset.

Nodeedits will be flattened into splices before being yielded.

splicesBack(offs=None, size=None)[source]
stat()[source]
storNodeEdits(nodeedits, meta)[source]
storNodeEditsNoLift(nodeedits, meta)[source]

Execute a series of node edit operations.

Does not return the updated nodes.

syncIndexEvents(offs, matchdef, wait=True)[source]

Yield (offs, (buid, form, ETYPE, VALS, META)) tuples from the nodeedit log starting from the given offset. Only edits that match the filter in matchdef will be yielded.

Notes

ETYPE is an constant EDIT_* above. VALS is a tuple whose format depends on ETYPE, outlined in the comment next to the constant. META is a dict that may contain keys ‘user’ and ‘time’ to represent the iden of the user that initiated the change, and the time that it took place, respectively.

Additionally, every 1000 entries, an entry (offs, (None, None, EDIT_PROGRESS, (), ())) message is emitted.

The matchdef dict may contain the following keys: forms, props, tags, tagprops. The value must be a sequence of strings. Each key/val combination is treated as an “or”, so each key and value yields more events. forms: EDIT_NODE_ADD and EDIT_NODE_DEL events. Matches events for nodes with forms in the value list. props: EDIT_PROP_SET and EDIT_PROP_DEL events. Values must be in form:prop or .universal form tags: EDIT_TAG_SET and EDIT_TAG_DEL events. Values must be the raw tag with no #. tagprops: EDIT_TAGPROP_SET and EDIT_TAGPROP_DEL events. Values must be just the prop or tag:prop.

Will not yield any values if this layer was not created with logedits enabled

Parameters:
  • offs (int) – starting nexus/editlog offset
  • matchdef (Dict[str, Sequence[str]]) – a dict describing which events are yielded
  • wait (bool) – whether to pend and stream value until this layer is fini’d
syncNodeEdits(offs, wait=True)[source]

Identical to syncNodeEdits2, but doesn’t yield meta

syncNodeEdits2(offs, wait=True)[source]

Once caught up with storage, yield them in realtime.

Returns:Tuple of offset(int), nodeedits, meta(dict)
truncate()[source]

Nuke all the contents in the layer, leaving an empty layer

waitEditOffs(offs, timeout=None)[source]

Wait for the node edit log to write an entry at/past the given offset.

waitForHot()[source]

Wait for the layer’s slab to be prefaulted and locked into memory if lockmemory is true, otherwise return.

waitUpstreamOffs(iden, offs)[source]
class synapse.lib.layer.LayerApi[source]

Bases: synapse.lib.cell.CellApi

getEditIndx()[source]

Returns what will be the next nodeedit log index.

getIden()[source]
iterLayerNodeEdits()[source]

Scan the full layer and yield artificial nodeedit sets.

splices(offs=None, size=None)[source]

Yield (offs, splice) tuples from the nodeedit log starting from the given offset.

Nodeedits will be flattened into splices before being yielded.

storNodeEdits(nodeedits, meta=None)[source]
storNodeEditsNoLift(nodeedits, meta=None)[source]
syncNodeEdits(offs, wait=True)[source]

Yield (offs, nodeedits) tuples from the nodeedit log starting from the given offset.

Once caught up with storage, yield them in realtime.

class synapse.lib.layer.StorType(layr, stortype)[source]

Bases: object

decodeIndx(valu)[source]
indx(valu)[source]
indxBy(liftby, cmpr, valu)[source]
indxByForm(form, cmpr, valu)[source]
indxByProp(form, prop, cmpr, valu)[source]
indxByPropArray(form, prop, cmpr, valu)[source]
indxByTagProp(form, tag, prop, cmpr, valu)[source]
class synapse.lib.layer.StorTypeFloat(layr, stortype, size=8)[source]

Bases: synapse.lib.layer.StorType

FloatPackNegMax = b'\x80\x00\x00\x00\x00\x00\x00\x00'
FloatPackNegMin = b'\xff\xf0\x00\x00\x00\x00\x00\x00'
FloatPackPosMax = b'\x7f\xf0\x00\x00\x00\x00\x00\x00'
FloatPackPosMin = b'\x00\x00\x00\x00\x00\x00\x00\x00'
FloatPacker = <Struct object>
decodeIndx(bytz)[source]
fpack()

S.pack(v1, v2, …) -> bytes

Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.

indx(valu)[source]
class synapse.lib.layer.StorTypeFqdn(layr)[source]

Bases: synapse.lib.layer.StorTypeUtf8

decodeIndx(bytz)[source]
indx(norm)[source]
class synapse.lib.layer.StorTypeGuid(layr)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeHier(layr, stortype, sepr='.')[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
getHierIndx(valu)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeHugeNum(layr, stortype)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
getHugeIndx(norm)[source]
indx(norm)[source]
class synapse.lib.layer.StorTypeInt(layr, stortype, size, signed)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
getIntIndx(valu)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeIpv6(layr)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
getIPv6Indx(valu)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeIval(layr)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeLatLon(layr)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
indx(valu)[source]
class synapse.lib.layer.StorTypeLoc(layr)[source]

Bases: synapse.lib.layer.StorTypeHier

class synapse.lib.layer.StorTypeMsgp(layr)[source]

Bases: synapse.lib.layer.StorType

indx(valu)[source]
class synapse.lib.layer.StorTypeTag(layr)[source]

Bases: synapse.lib.layer.StorTypeHier

static getTagFilt(cmpr, valu)[source]
class synapse.lib.layer.StorTypeTime(layr)[source]

Bases: synapse.lib.layer.StorTypeInt

class synapse.lib.layer.StorTypeUtf8(layr)[source]

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)[source]
indx(valu)[source]
synapse.lib.layer.getFlatEdits(nodeedits)[source]

synapse.lib.lmdbslab module

class synapse.lib.lmdbslab.GuidStor(slab, name)[source]

Bases: object

gen(iden)[source]
class synapse.lib.lmdbslab.Hist(slab, name)[source]

Bases: object

A class for storing items in a slab by time.

Each added item is inserted into the specified db within the slab using the current epoch-millis time stamp as the key.

add(item)[source]
carve(tick, tock=None)[source]
class synapse.lib.lmdbslab.HotCount[source]

Bases: synapse.lib.lmdbslab.HotKeyVal

Like HotKeyVal, but optimized for integer/count vals

static DecFunc(b)

Decode a signed 64-bit int from 8 byte big-endian

static EncFunc(i)

Encode a signed 64-bit int into 8 byte big-endian bytes

get(name: str, defv=0)[source]
inc(name: str, valu=1)[source]
set(name: str, valu)[source]
class synapse.lib.lmdbslab.HotKeyVal[source]

Bases: synapse.lib.base.Base

A hot-loop capable keyval that only syncs on commit.

static DecFunc(byts)

Use msgpack to de-serialize a python object.

Parameters:byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns:The de-serialized object
Return type:obj
static EncFunc(item)

Use msgpack to serialize a compatible python object.

Parameters:item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns:The serialized bytes in msgpack format.
Return type:bytes
delete(name: str)[source]
get(name: str, defv=None)[source]
pack()[source]
set(name: str, valu)[source]
sync()[source]
class synapse.lib.lmdbslab.MultiQueue[source]

Bases: synapse.lib.base.Base

Allows creation/consumption of multiple durable queues in a slab.

add(name, info)[source]
cull(name, offs)[source]

Remove up-to (and including) the queue entry at offs.

dele(name, minoffs, maxoffs)[source]

Remove queue entries from minoffs, up-to (and including) the queue entry at maxoffs.

exists(name)[source]
get(name, offs, wait=False, cull=True)[source]

Return (nextoffs, item) tuple or (-1, None) for the given offset.

gets(name, offs, size=None, cull=False, wait=False)[source]

Yield (offs, item) tuples from the message queue.

list()[source]
offset(name)[source]
pop(name, offs)[source]

Pop a single entry from the named queue by offset.

put(name, item, reqid=None)[source]
puts(name, items, reqid=None)[source]
rem(name)[source]
sets(name, offs, items)[source]

Overwrite queue entries with the values in items, starting at offs.

size(name)[source]
status(name)[source]
class synapse.lib.lmdbslab.Scan(slab, db)[source]

Bases: object

A state-object used by Slab. Not to be instantiated directly.

Parameters:
  • slab (Slab) – which slab the scan is over
  • db (str) – name of open database on the slab
bump()[source]
first()[source]
isatitem()[source]

Returns if the cursor is at the value in atitem

iterfunc()[source]
iternext()[source]
resume()[source]
set_key(lkey)[source]
set_range(lkey, valu=None)[source]
class synapse.lib.lmdbslab.ScanBack(slab, db)[source]

Bases: synapse.lib.lmdbslab.Scan

A state-object used by Slab. Not to be instantiated directly.

Scans backwards.

first()[source]
iterfunc()[source]
resume()[source]
set_key(lkey)[source]
set_range(lkey)[source]
class synapse.lib.lmdbslab.ScanKeys(slab, db)[source]

Bases: synapse.lib.lmdbslab.Scan

An iterator over the keys of the database. If the database is dupsort, a key with multiple values with be yielded once for each value.

isatitem()[source]

Returns if the cursor is at the value in atitem

iterfunc()[source]
iternext()[source]
resume()[source]
class synapse.lib.lmdbslab.Slab[source]

Bases: synapse.lib.base.Base

A “monolithic” LMDB instance for use in a asyncio loop thread.

COMMIT_PERIOD = 0.2
DEFAULT_GROWSIZE = None
DEFAULT_MAPSIZE = 1073741824
WARN_COMMIT_TIME_MS = 5000
allslabs = {}
copydb(sourcedbname, destslab, destdbname=None, progresscb=None)[source]

Copy an entire database in this slab to a new database in potentially another slab.

Parameters:
  • sourcedbname (str) – name of the db in the source environment
  • destslab (LmdbSlab) – which slab to copy rows to
  • destdbname (str) – the name of the database to copy rows to in destslab
  • progresscb (Callable[int]) – if not None, this function will be periodically called with the number of rows completed
Returns:

the number of rows copied

Return type:

(int)

Note

If any rows already exist in the target database, this method returns an error. This means that one cannot use destdbname=None unless there are no explicit databases in the destination slab.

copyslab(dstpath, compact=True)[source]
countByPref(byts, db=None, maxsize=None)[source]

Return the number of rows in the given db with the matching prefix bytes.

dbexists(name)[source]

The DB exists already if there’s a key in the default DB with the name of the database

delete(lkey, val=None, db=None)[source]
dropdb(name)[source]

Deletes an entire database (i.e. a table), losing all data.

fini()[source]

Shut down the object and notify any onfini() coroutines.

Returns:Remaining ref count
firstkey(db=None)[source]

Return the first key or None from the given db.

forcecommit()[source]

Note

This method may raise a MapFullError

get(lkey, db=None)[source]
getHotCount(name)[source]
getMultiQueue(name, nexsroot=None)[source]
getNameAbrv(name)[source]
getSeqn(name)[source]
classmethod getSlabStats()[source]
classmethod getSlabsInDir(dirn)[source]

Returns all open slabs under a directory

has(lkey, db=None)[source]
hasdup(lkey, lval, db=None)[source]
classmethod initSyncLoop(inst)[source]
initdb(name, dupsort=False, integerkey=False, dupfixed=False)[source]
last(db=None)[source]

Return the last key/value pair from the given db.

lastkey(db=None)[source]

Return the last key or None from the given db.

pop(lkey, db=None)[source]
prefexists(byts, db=None)[source]

Returns True if a prefix exists in the db.

put(lkey, lval, dupdata=False, overwrite=True, append=False, db=None)[source]
putmulti(kvpairs, dupdata=False, append=False, db=None)[source]
Returns:Tuple of number of items consumed, number of items added
rangeexists(lmin, lmax=None, db=None)[source]

Returns True if at least one key exists in the range.

replace(lkey, lval, db=None)[source]

Like put, but returns the previous value if existed

scanByDups(lkey, db=None)[source]
scanByDupsBack(lkey, db=None)[source]
scanByFull(db=None)[source]
scanByFullBack(db=None)[source]
scanByPref(byts, startkey=None, startvalu=None, db=None)[source]
Parameters:
  • byts (bytes) – prefix to match on
  • startkey (Optional[bytes]) – if present, will start scanning at key=byts+startkey
  • startvalu (Optional[bytes]) – if present, will start scanning at (key+startkey, startvalu)

Notes

startvalu only makes sense if byts+startkey matches an entire key. startvalu is only value for dupsort=True dbs

scanByPrefBack(byts, db=None)[source]
scanByRange(lmin, lmax=None, db=None)[source]
scanByRangeBack(lmax, lmin=None, db=None)[source]
scanKeys(db=None)[source]
scanKeysByPref(byts, db=None)[source]
stat(db=None)[source]
statinfo()[source]
sync()[source]
classmethod syncLoopOnce()[source]
classmethod syncLoopTask()[source]
syncevnt = None
synctask = None
trash()[source]

Deletes underlying storage

class synapse.lib.lmdbslab.SlabAbrv(slab, name)[source]

Bases: object

A utility for translating arbitrary bytes into fixed with id bytes

abrvToByts(abrv)[source]
abrvToName(byts)[source]
bytsToAbrv(byts)[source]
nameToAbrv(name)[source]
setBytsToAbrv(byts)[source]
class synapse.lib.lmdbslab.SlabDict(slab, db=None, pref=b'')[source]

Bases: object

A dictionary-like object which stores its props in a slab via a prefix.

It is assumed that only one SlabDict with a given prefix exists at any given time, but it is up to the caller to cache them.

get(name, defval=None)[source]

Get a name from the SlabDict.

Parameters:
  • name (str) – The key name.
  • defval (obj) – The default value to return.
Returns:

The return value, or None.

Return type:

(obj)

inc(name, valu=1)[source]
items()[source]

Return a tuple of (prop, valu) tuples from the SlabDict.

Returns:Tuple of (name, valu) tuples.
Return type:(((str, object), ..))
keys()[source]
pop(name, defval=None)[source]

Pop a name from the SlabDict.

Parameters:
  • name (str) – The name to remove.
  • defval (obj) – The default value to return if the name is not present.
Returns:

The object stored in the SlabDict, or defval if the object was not present.

Return type:

object

set(name, valu)[source]

Set a name in the SlabDict.

Parameters:
  • name (str) – The key name.
  • valu (obj) – A msgpack compatible value.
Returns:

None

synapse.lib.modelrev module

class synapse.lib.modelrev.ModelRev(core)[source]

Bases: object

revCoreLayers()[source]
revModel20210126(layers)[source]
revModel20210312(layers)[source]
revModel20210528(layers)[source]
revModel20210801(layers)[source]

synapse.lib.module module

class synapse.lib.module.CoreModule(core, conf=None)[source]

Bases: object

confdefs = ()
getConfPath()[source]

Get the path to the module specific config file (conf.yaml).

Notes

This creates the parent directory for the conf.yaml file if it does not exist. This API exists to allow a implementor to get the conf path during initCoreModule and drop a example config if needed. One use case of that is for missing configuration values, an example config can be written to the file and a exception raised.

Returns:Path to where the conf file is located at.
Return type:str
getModDir()[source]

Get the path to the module specific directory.

Notes

This creates the directory if it did not previously exist.

Returns:The filepath to the module specific directory.
Return type:str
getModName()[source]

Return the lowercased name of this module.

Notes

This pulls the mod_name attribute on the class. This allows an implementer to set a arbitrary name for the module. If this attribute is not set, it defaults to self.__class__.__name__.lower() and sets mod_name to that value.

Returns:The module name.
Return type:(str)
getModPath(*paths)[source]

Construct a path relative to this module’s working directory.

Parameters:*paths – A list of path strings

Notes

This creates the module specific directory if it does not exist.

Returns:The full path (or None if no cortex dir is configured).
Return type:(str)
getModelDefs()[source]
getStormCmds()[source]

Module implementers may override this to provide a list of Storm commands which will be loaded into the Cortex.

Returns:A list of Storm Command classes (not instances).
Return type:list
initCoreModule()[source]

Module implementers may override this method to initialize the module after the Cortex has completed and is accessible to perform storage operations.

Notes

This is the preferred function to override for implementing custom code that needs to be executed during Cortex startup.

Any exception raised within this method will remove the module from the list of currently loaded modules.

This is called for modules after getModelDefs() and getStormCmds() has been called, in order to allow for model loading and storm command loading prior to code execution offered by initCoreModule.

A failure during initCoreModule will not unload data model or storm commands registered by the module.

Returns:None
mod_name = None
preCoreModule()[source]

Module implementers may override this method to execute code immediately after a module has been loaded.

Notes

The initCoreModule function is preferred for overriding instead of preCoreModule().

No Cortex layer/storage operations will function in preCoreModule.

Any exception raised within this method will halt additional loading of the module.

Returns:None

synapse.lib.modules module

Module which implements the synapse module API/convention.

synapse.lib.msgpack module

class synapse.lib.msgpack.Unpk[source]

Bases: object

An extension of the msgpack streaming Unpacker which reports sizes.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

feed(byts)[source]

Feed bytes to the unpacker and return completed objects.

Parameters:byts (bytes) – Bytes to unpack.

Notes

It is intended that this function is called multiple times with bytes from some sort of a stream, as it will unpack and return objects as they are available.

Returns:List of tuples containing the item size and the unpacked item.
Return type:list
synapse.lib.msgpack.deepcopy(item)[source]

Copy a msgpack serializable by packing then unpacking it. For complex primitives, this runs in about 1/3 the time of copy.deepcopy()

synapse.lib.msgpack.dumpfile(item, path)[source]

Dump an object to a file by path.

Parameters:
  • item (object) – The object to serialize.
  • path (str) – The file path to save.
Returns:

None

synapse.lib.msgpack.en(item)[source]

Use msgpack to serialize a compatible python object.

Parameters:item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns:The serialized bytes in msgpack format.
Return type:bytes
synapse.lib.msgpack.isok(item)[source]

Returns True if the item can be msgpacked (by testing packing).

synapse.lib.msgpack.iterfd(fd)[source]

Generator which unpacks a file object of msgpacked content.

Parameters:fd – File object to consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields:Objects from a msgpack stream.
synapse.lib.msgpack.iterfile(path, since=-1)[source]

Generator which yields msgpack objects from a file path.

Parameters:path – File path to open and consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields:Objects from a msgpack stream.
synapse.lib.msgpack.loadfile(path)[source]

Load and upack the msgpack bytes from a file by path.

Parameters:path (str) – The file path to a message pack file.
Raises:msgpack.exceptions.ExtraData – If the file contains multiple objects.
Returns:The decoded python object.
Return type:(obj)
synapse.lib.msgpack.un(byts)[source]

Use msgpack to de-serialize a python object.

Parameters:byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns:The de-serialized object
Return type:obj

synapse.lib.multislabseqn module

class synapse.lib.multislabseqn.MultiSlabSeqn[source]

Bases: synapse.lib.base.Base

An append-optimized sequence of byte blobs stored across multiple slabs for fast rotating/culling

add(item: Any, indx=None) → int[source]

Add a single item to the sequence.

cull(offs: int) → bool[source]

Remove entries up to (and including) the given offset.

get(offs: int) → Any[source]

Retrieve a single row by offset

getOffsetEvent(offs: int) → asyncio.locks.Event[source]

Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

gets(offs, wait=True) → AsyncIterator[Tuple[int, Any]][source]

Just like iter, but optionally waits for new entries once the end is reached.

index() → int[source]

Return the current index to be used

iter(offs: int) → AsyncIterator[Tuple[int, Any]][source]

Iterate over items in a sequence from a given offset.

Parameters:offs (int) – The offset to begin iterating from.
Yields:(indx, valu) – The index and valu of the item.
last() → Optional[Tuple[int, Any]][source]
rotate() → int[source]

Rotate the Nexus log at the current index.

Note

After this executes the tailseqn will be empty. Waiting for this indx to be written will indicate when it is possible to cull 1 minus the return value such that the rotated seqn is deleted.

Returns:The starting index of the new seqn
Return type:int
setIndex(indx: int) → None[source]
static slabFilename(dirn: str, indx: int)[source]
waitForOffset(offs: int, timeout=None) → bool[source]
Returns:true if the event got set, False if timed out

synapse.lib.nexus module

class synapse.lib.nexus.ChangeDist[source]

Bases: synapse.lib.base.Base

A utility class to distribute new change entries to mirrors/followers

update() → bool[source]
class synapse.lib.nexus.NexsRoot[source]

Bases: synapse.lib.base.Base

cull(offs)[source]
eat(nexsiden, event, args, kwargs, meta)[source]

Actually mutate for the given nexsiden instance.

getChangeDist(offs: int) → AsyncIterator[synapse.lib.nexus.ChangeDist][source]
index()[source]
issue(nexsiden: str, event: str, args: Tuple[Any, ...], kwargs: Dict[str, Any], meta: Optional[Dict[KT, VT]] = None) → Any[source]

If I’m not a follower, mutate, otherwise, ask the leader to make the change and wait for the follower loop to hand me the result through a future.

iter(offs: int) → AsyncIterator[Any][source]

Returns an iterator of change entries in the log

promote()[source]
recover() → None[source]

Replays the last entry in the nexus log in case we crashed between writing the log and applying it.

Notes

This must be called at cell startup after subsystems are initialized but before any write transactions might happen.

The log can only have recorded 1 entry ahead of what is applied. All log actions are idempotent, so replaying the last action that (might have) already happened is harmless.

rotate()[source]
runMirrorLoop(proxy)[source]
setindex(indx)[source]
startup(mirurl, celliden=None)[source]
waitOffs(offs, timeout=None)[source]
class synapse.lib.nexus.Pusher[source]

Bases: synapse.lib.base.Base

A mixin-class to manage distributing changes where one might plug in mirroring or consensus protocols

classmethod onPush(event: str, passitem=False) → Callable[source]

Decorator that registers a method to be a handler for a named event

Parameters:
  • event – string that distinguishes one handler from another. Must be unique per Pusher subclass
  • passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”
classmethod onPushAuto(event: str, passitem=False) → Callable[source]

Decorator that does the same as onPush, except automatically creates the top half method

Parameters:
  • event – string that distinguishes one handler from another. Must be unique per Pusher subclass
  • passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”
setNexsRoot(nexsroot)[source]
class synapse.lib.nexus.RegMethType(name: str, bases: List[type], attrs: Dict[str, Any])[source]

Bases: type

Metaclass that collects all methods in class with _regme prop into a class member called _regclstupls

synapse.lib.node module

class synapse.lib.node.Node(snap, sode, bylayer=None)[source]

Bases: object

A Cortex hypergraph node.

NOTE: This object is for local Cortex use during a single Xact.

addEdge(verb, n2iden)[source]
addTag(tag, valu=(None, None))[source]

Add a tag to a node.

Parameters:
  • tag (str) – The tag to add to the node.
  • valu – The optional tag value. If specified, this must be a value that norms as a valid time interval as an ival.
Returns:

This returns None.

Return type:

None

delEdge(verb, n2iden)[source]
delTag(tag, init=False)[source]

Delete a tag from the node.

delTagProp(tag, name)[source]
delete(force=False)[source]

Delete a node from the cortex.

The following tear-down operations occur in order:

  • validate that you have permissions to delete the node
  • validate that you have permissions to delete all tags
  • validate that there are no remaining references to the node.
  • delete all the tags (bottom up)
    • fire onDelTag() handlers
    • delete tag properties from storage
    • log tag:del splices
  • delete all secondary properties
    • fire onDelProp handler
    • delete secondary property from storage
    • log prop:del splices
  • delete the primary property
    • fire onDel handlers for the node
    • delete primary property from storage
    • log node:del splices
filter(runt, text, opts=None, path=None)[source]
get(name)[source]

Return a secondary property value from the Node.

Parameters:name (str) – The name of a secondary property.
Returns:The secondary property value or None.
Return type:(obj)
getByLayer()[source]

Return a dictionary that translates the node’s bylayer dict to a primitive.

getData(name)[source]
getEmbeds(embeds)[source]

Return a dictionary of property embeddings.

getNodeRefs()[source]

Return a list of (prop, (form, valu)) refs out for the node.

getStorNodes()[source]

Return a list of the raw storage nodes for each layer.

getTag(name, defval=None)[source]
getTagProp(tag, prop, defval=None)[source]

Return the value (or defval) of the given tag property.

getTagProps(tag)[source]
getTags(leaf=False)[source]
has(name)[source]
hasData(name)[source]
hasTag(name)[source]
hasTagProp(tag, prop)[source]

Check if a #foo.bar:baz tag property exists on the node.

iden()[source]
iterData()[source]
iterEdgesN1(verb=None)[source]
iterEdgesN2(verb=None)[source]
pack(dorepr=False)[source]

Return the serializable/packed version of the node.

Parameters:dorepr (bool) – Include repr information for human readable versions of properties.
Returns:An (ndef, info) node tuple.
Return type:(tuple)
pop(name, init=False)[source]

Remove a property from a node and return the value

popData(name)[source]
repr(name=None, defv=None)[source]
reprs()[source]

Return a dictionary of repr values for props whose repr is different than the system mode value.

seen(tick, source=None)[source]

Update the .seen interval and optionally a source specific seen node.

set(name, valu, init=False)[source]

Set a property on the node.

Parameters:
  • name (str) – The name of the property.
  • valu (obj) – The value of the property.
  • init (bool) – Set to True to disable read-only enforcement
Returns:

True if the property was changed.

Return type:

(bool)

setData(name, valu)[source]
setTagProp(tag, name, valu)[source]

Set the value of the given tag property.

storm(runt, text, opts=None, path=None)[source]
Parameters:path (Path) – If set, then vars from path are copied into the new runtime, and vars are copied back out into path at the end

Note

If opts is not None and opts[‘vars’] is set and path is not None, then values of path vars take precedent

tagpropreprs()[source]

Return a dictionary of repr values for tagprops whose repr is different than the system mode value.

class synapse.lib.node.Path(vars, nodes)[source]

Bases: object

A path context tracked through the storm runtime.

clone()[source]
finiframe()[source]

Pop a scope frame from the path, restoring runt if at the top :param runt: A storm runtime to restore if we’re at the top :type runt: Runtime :param merge: Set to true to merge vars back up into the next frame :type merge: bool

fork(node)[source]
getVar(name, defv=<synapse.common.NoValu object>)[source]
initframe(initvars=None)[source]
meta(name, valu)[source]

Add node specific metadata to be returned with the node.

pack(path=False)[source]
popVar(name)[source]
setVar(name, valu)[source]
trace()[source]

Construct and return a Trace object for this path.

class synapse.lib.node.Trace(path)[source]

Bases: object

A trace for pivots taken and nodes involved from a given path’s subsequent forks.

addFork(path)[source]
addPath(path)[source]
synapse.lib.node.iden(pode)[source]

Return the iden (buid) of the packed node.

Parameters:pode (tuple) – A packed node.
Returns:The node iden.
Return type:str
synapse.lib.node.ndef(pode)[source]

Return a node definition (<form>,<valu>) tuple from the node.

Parameters:pode (tuple) – A packed node.
Returns:The (<form>,<valu>) tuple for the node
Return type:((str,obj))
synapse.lib.node.prop(pode, prop)[source]

Return the valu of a given property on the node.

Parameters:
  • pode (tuple) – A packed node.
  • prop (str) – Property to retrieve.

Notes

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns:

synapse.lib.node.props(pode)[source]

Get the props from the node.

Parameters:pode (tuple) – A packed node.

Notes

This will include any universal props present on the node.

Returns:A dictionary of properties.
Return type:dict
synapse.lib.node.reprNdef(pode)[source]

Get the ndef of the pode with a human readable value.

Parameters:pode (tuple) – A packed node.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

Returns:A tuple of form and the human readable value.
Return type:(str, str)
synapse.lib.node.reprProp(pode, prop)[source]

Get the human readable value for a secondary property from the pode.

Parameters:
  • pode (tuple) – A packed node.
  • prop

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns:The human readable property value. If the property is not present, returns None.
Return type:str
synapse.lib.node.reprTag(pode, tag)[source]

Get the human readable value for the tag timestamp from the pode.

Parameters:
  • pode (tuple) – A packed node.
  • tag (str) – The tag to get the value for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have a timestamp, this returns a empty string.

Returns:The human readable value for the tag. If the tag is not present, returns None.
Return type:str
synapse.lib.node.reprTagProps(pode, tag)[source]

Get the human readable values for any tagprops on a tag for a given node.

Parameters:
  • pode (tuple) – A packed node.
  • tag (str) – The tag to get the tagprops reprs for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have any tagprops associated with it, this returns an empty list.

Returns:A list of tuples, containing the name of the tagprop and the repr value.
Return type:list
synapse.lib.node.tagged(pode, tag)[source]

Check if a packed node has a given tag.

Parameters:
  • pode (tuple) – A packed node.
  • tag (str) – The tag to check.

Examples

Check if a node is tagged with “woot” and dostuff if it is.

if s_node.tagged(node,’woot’):
dostuff()

Notes

If the tag starts with #, this is removed prior to checking.

Returns:True if the tag is present. False otherwise.
Return type:bool
synapse.lib.node.tags(pode, leaf=False)[source]

Get all the tags for a given node.

Parameters:
  • pode (tuple) – A packed node.
  • leaf (bool) – If True, only return leaf tags
Returns:

A list of tag strings.

Return type:

list

synapse.lib.node.tagsnice(pode)[source]

Get all the leaf tags and the tags that have values or tagprops.

Parameters:pode (tuple) – A packed node.
Returns:A list of tag strings.
Return type:list

synapse.lib.output module

Tools for easily hookable output from cli-like tools.

class synapse.lib.output.OutPut[source]

Bases: object

printf(mesg, addnl=True)[source]
class synapse.lib.output.OutPutBytes[source]

Bases: synapse.lib.output.OutPutFd

class synapse.lib.output.OutPutFd(fd, enc='utf8')[source]

Bases: synapse.lib.output.OutPut

class synapse.lib.output.OutPutStr[source]

Bases: synapse.lib.output.OutPut

synapse.lib.parser module

class synapse.lib.parser.AstConverter(text)[source]

Bases: lark.visitors.Transformer

Convert AST from parser into synapse AST, depth first.

If a method with a name that matches the current rule exists, that will be called, otherwise __default__ will be used

argvquery(kids, meta)[source]
baresubquery(kids, meta)[source]
cmdrargs(kids)[source]
embedquery(kids, meta)[source]
evalvalu(kids, meta)[source]
funcargs(kids, meta)[source]

A list of function parameters (as part of a function definition)

funccall(kids, meta)[source]
lookup(kids, meta)[source]
operrelprop_join(kids)[source]
operrelprop_pivot(kids, isjoin=False)[source]
query(kids, meta)[source]
stormcmdargs(kids)[source]
subquery(kids, meta)[source]
switchcase(kids)[source]
tagname(kids)[source]
varderef(kids)[source]
varlist(kids)[source]
yieldvalu(kids)[source]
class synapse.lib.parser.CmdStringer(visit_tokens=True)[source]

Bases: lark.visitors.Transformer

alist(kids, meta)[source]
cmdstring(kids, meta)[source]
valu(kids, meta)[source]
class synapse.lib.parser.Parser(text, offs=0)[source]

Bases: object

Storm query parser

cmdrargs()[source]

Parse command args that might have storm queries as arguments

eval()[source]
lookup()[source]
query()[source]

Parse the storm query

Returns (s_ast.Query): instance of parsed query

synapse.lib.parser.massage_vartokn(x)[source]
synapse.lib.parser.parseEval(text)[source]
synapse.lib.parser.parseQuery(text, mode='storm')[source]

Parse a storm query and return the Lark AST. Cached here to speed up unit tests

synapse.lib.parser.parse_cmd_string(text, off)[source]

Parse a command line string which may be quoted.

synapse.lib.parser.unescape(valu)[source]

Parse a string for backslash-escaped characters and omit them. The full list of escaped characters can be found at https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

synapse.lib.provenance module

class synapse.lib.provenance.ProvStor[source]

Bases: synapse.lib.base.Base

Persistently stores provstacks so they can be retrieved by iden and sequence

PROV_FN = 'prov.lmdb'
PROV_MAP_SIZE = 67108864
getProvStack(iden: bytes)[source]

Returns the provenance stack given the iden to it

precommit()[source]

Determine the iden for the current provenance stack and return it

Returns the iden

provStacks(offs, size)[source]

Returns a stream of provenance stacks at the given offset

stor()[source]

Writes the current provenance stack to storage if it wasn’t already there

Returns (iden, provstack) if was written, (None, None) if it was already there

synapse.lib.provenance.claim(typ, **info)[source]

Add an entry to the provenance stack for the duration of the context

synapse.lib.provenance.dupstack(newtask)[source]

Duplicate the current provenance stack onto another task

synapse.lib.provenance.get()[source]
Returns:A tuple of (stack iden (or None if not set), the current provenance stack)
synapse.lib.provenance.reset()[source]

Reset the stack to its initial state

For testing purposes

synapse.lib.provenance.setiden(iden, waswritten)[source]

Sets the cached stack iden, waswritten for the current provenance stack. We use waswritten to cache whether we’ve written the stack and so we can tell the snap whether to fire a prov:new event

synapse.lib.queue module

class synapse.lib.queue.AQueue[source]

Bases: synapse.lib.base.Base

An async queue with chunk optimized sync compatible consumer.

put(item)[source]

Add an item to the queue.

slice()[source]
class synapse.lib.queue.Queue(maxsize=None)[source]

Bases: object

An asyncio Queue with batch methods and graceful close.

close()[source]
put(item)[source]
puts(items)[source]
size()[source]
slice(size=1000)[source]
slices(size=1000)[source]
class synapse.lib.queue.Window[source]

Bases: synapse.lib.base.Base

A Queue like object which yields added items. If the queue ever reaches its maxsize, it will be fini()d. On fini(), the Window will continue to yield results until empty and then return.

put(item)[source]

Add a single item to the Window.

puts(items)[source]

Add multiple items to the window.

synapse.lib.ratelimit module

class synapse.lib.ratelimit.RateLimit(rate, per)[source]

Bases: object

A RateLimit class may be used to detect/enforce rate limits.

Example

# allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10)

Notes

It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”.

allows()[source]

Returns True if the rate limit has not been reached.

Example

if not rlimit.allows():
rasie RateExceeded()

# ok to go…

synapse.lib.reflect module

synapse.lib.reflect.getClsNames(item)[source]

Return a list of “fully qualified” class names for an instance.

Example

for name in getClsNames(foo):
print(name)
synapse.lib.reflect.getItemLocals(item)[source]

Iterate the locals of an item and yield (name,valu) pairs.

Example

for name,valu in getItemLocals(item):
dostuff()
synapse.lib.reflect.getMethName(meth)[source]

Return a fully qualified string for the <mod>.<class>.<func> name of a given method.

synapse.lib.reflect.getShareInfo(item)[source]

Get a dictionary of special annotations for a Telepath Proxy.

Parameters:item – Item to inspect.

Notes

This will set the _syn_telemeth attribute on the item and the items class, so this data is only computed once.

Returns:A dictionary of methods requiring special handling by the proxy.
Return type:dict

synapse.lib.rstorm module

class synapse.lib.rstorm.OutPutRst[source]

Bases: synapse.lib.output.OutPutStr

Rst specific helper for output intended to be indented in RST text as a literal block.

prefix = ' '
printf(mesg, addnl=True)[source]
class synapse.lib.rstorm.StormCliOutput[source]

Bases: synapse.tools.storm.StormCli

handleErr(mesg)[source]
printf(mesg, addnl=True, color=None)[source]
runRstCmdLine(text, ctx, stormopts=None)[source]
class synapse.lib.rstorm.StormOutput(core, ctx, stormopts=None, opts=None)[source]

Bases: synapse.cmds.cortex.StormCmd

Produce standard output from a stream of storm runtime messages. Must be instantiated for a single query with a rstorm context.

printf(mesg, addnl=True, color=None)[source]
runCmdLine(line)[source]

Run a line of command input for this command.

Parameters:line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)
runCmdOpts(opts)[source]

Perform the command actions. Must be implemented by Cmd implementers.

Parameters:opts (dict) – Options dictionary.
class synapse.lib.rstorm.StormRst[source]

Bases: synapse.lib.base.Base

run()[source]

Parses the specified RST file with Storm directive handling.

Returns:List of line strings for the RST output
Return type:list
synapse.lib.rstorm.getCell(ctor, conf)[source]

synapse.lib.scope module

class synapse.lib.scope.Scope(*frames, **vals)[source]

Bases: object

The Scope object assists in creating nested varible scopes.

Example

with Scope() as scope:

scope.set(‘foo’,10)

with scope:
scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20…

dostuff(scope) # ‘foo’ is 10 again…

add(name, *vals)[source]

Add values as iter() compatible items in the current scope frame.

ctor(name, func, *args, **kwargs)[source]

Add a constructor to be called when a specific property is not present.

Example

scope.ctor(‘foo’,FooThing) … foo = scope.get(‘foo’)

enter(vals=None)[source]

Add an additional scope frame.

get(name, defval=None)[source]

Retrieve a value from the closest scope frame.

iter(name)[source]

Iterate through values added with add() from each scope frame.

leave()[source]

Pop the current scope frame.

pop(name, defval=None)[source]

Pop and return a value (from the last frame) of the scope.

Parameters:name (str) – The name of the scope variable.
Returns:The scope variable value or None
Return type:obj
set(name, valu)[source]

Set a value in the current scope frame.

update(vals)[source]

Set multiple values in the current scope frame.

synapse.lib.scope.ctor(name, func, *args, **kwargs)[source]

Add a ctor callback to the global scope.

synapse.lib.scope.enter(vals=None)[source]

Return the task’s local scope for use in a with block

synapse.lib.scope.get(name, defval=None)[source]

Access this task’s scope with default values from glob.

synapse.lib.scope.pop(name)[source]

Pop and return a task scope variable. :param name: The task scope variable name. :type name: str

Returns:The scope value or None
Return type:obj
synapse.lib.scope.set(name, valu)[source]

Set a value in the current frame of the local task scope.

synapse.lib.scope.update(vals)[source]

synapse.lib.scrape module

synapse.lib.scrape.fqdn_prefix_check(match: _regex.Match)[source]
synapse.lib.scrape.refang_text(txt)[source]

Remove address de-fanging in text blobs, .e.g. example[.]com to example.com

Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM

Returns:Re-fanged text blob
Return type:(str)
synapse.lib.scrape.scrape(text, ptype=None, refang=True, first=False)[source]

Scrape types from a blob of text and return node tuples.

Parameters:
  • text (str) – Text to scrape.
  • ptype (str) – Optional ptype to scrape. If present, only scrape rules which match the provided type.
  • refang (bool) – Whether to remove de-fanging schemes from text before scraping.
Returns:

Yield tuples of node ndef values.

Return type:

(str, object)

synapse.lib.share module

class synapse.lib.share.Share[source]

Bases: synapse.lib.base.Base

Class to wrap a dynamically shared object.

synapse.lib.slaboffs module

class synapse.lib.slaboffs.SlabOffs(slab: synapse.lib.lmdbslab.Slab, db: str)[source]

Bases: object

A helper for storing offset integers by iden.

As with all slab objects, this is meant for single-thread async loop use.

delete(iden)[source]
get(iden)[source]
set(iden, offs)[source]

synapse.lib.slabseqn module

class synapse.lib.slabseqn.SlabSeqn(slab, name: str)[source]

Bases: object

An append optimized sequence of byte blobs.

Parameters:
  • lenv (lmdb.Environment) – The LMDB Environment.
  • name (str) – The name of the sequence.
add(item, indx=None)[source]

Add a single item to the sequence.

cull(offs)[source]

Remove entries up to (and including) the given offset.

first()[source]
get(offs)[source]

Retrieve a single row by offset

getByIndxByts(indxbyts)[source]
getOffsetEvent(offs)[source]

Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

gets(offs, wait=True)[source]

Returns an async generator of indx/valu tuples, optionally waiting and continuing to yield them as new entries are added

Parameters:
  • offs (int) – The offset to begin iterating from.
  • wait (bool) – Whether to continue yielding tupls when it hits the end of the sequence.
Yields:

(indx, valu) – The index and valu of the item.

index()[source]

Return the current index to be used

iter(offs)[source]

Iterate over items in a sequence from a given offset.

Parameters:offs (int) – The offset to begin iterating from.
Yields:(indx, valu) – The index and valu of the item.
iterBack(offs)[source]

Iterate backwards over items in a sequence from a given offset.

Parameters:offs (int) – The offset to begin iterating from.
Yields:(indx, valu) – The index and valu of the item.
last()[source]
nextindx()[source]

Determine the next insert offset according to storage.

Returns:The next insert offset.
Return type:int
pop(offs)[source]

Pop a single entry at the given offset.

rows(offs)[source]

Iterate over raw indx, bytes tuples from a given offset.

save(items)[source]

Save a series of items to a sequence.

Parameters:items (tuple) – The series of items to save into the sequence.
Returns:The index of the first item
slice(offs, size)[source]
sliceBack(offs, size)[source]
stat()[source]
trim(offs)[source]

Delete entries starting at offset and moving forward.

waitForOffset(offs, timeout=None)[source]
Returns:true if the event got set, False if timed out

synapse.lib.snap module

class synapse.lib.snap.Scrubber(rules)[source]

Bases: object

scrub(pode)[source]
class synapse.lib.snap.Snap[source]

Bases: synapse.lib.base.Base

A “snapshot” is a transaction across multiple Cortex layers.

The Snap object contains the bulk of the Cortex API to facilitate performance through careful use of transaction boundaries.

Transactions produce the following EventBus events:

(…any splice…) (‘log’, {‘level’: ‘mesg’: }) (‘print’, {}),

addFeedData(name, items)[source]
addFeedNodes(name, items)[source]

Call a feed function and return what it returns (typically yields Node()s).

Parameters:
  • name (str) – The name of the feed record type.
  • items (list) – A list of records of the given feed type.
Returns:

The return value from the feed function. Typically Node() generator.

Return type:

(object)

addNode(name, valu, props=None)[source]

Add a node by form name and value with optional props.

Parameters:
  • name (str) – The form of node to add.
  • valu (obj) – The value for the node.
  • props (dict) – Optional secondary properties for the node.

Notes

If a props dictionary is provided, it may be mutated during node construction.

Returns:A Node object. It may return None if the snap is unable to add or lift the node.
Return type:s_node.Node
addNodes(nodedefs)[source]

Add/merge nodes in bulk.

The addNodes API is designed for bulk adds which will also set properties, add tags, add edges, and set nodedata to existing nodes. Nodes are specified as a list of the following tuples:

( (form, valu), {‘props’:{}, ‘tags’:{}})
Parameters:nodedefs (list) – A list of nodedef tuples.
Returns:A list of xact messages.
Return type:(list)
addTagNode(name)[source]

Ensure that the given syn:tag node exists.

applyNodeEdit(edit)[source]
applyNodeEdits(edits)[source]

Sends edits to the write layer and evaluates the consequences (triggers, node object updates)

clearCache()[source]
disableTriggers()[source]
eval(text, opts=None, user=None)[source]

Run a storm query and yield Node() objects.

getNodeAdds(form, valu, props, addnode=T