synapse.lib package
Subpackages
- synapse.lib.crypto package
- synapse.lib.platforms package
- synapse.lib.stormlib package
- Submodules
- synapse.lib.stormlib.auth module
- synapse.lib.stormlib.backup module
- synapse.lib.stormlib.basex module
- synapse.lib.stormlib.cell module
- synapse.lib.stormlib.ethereum module
- synapse.lib.stormlib.gen module
- synapse.lib.stormlib.graph module
- synapse.lib.stormlib.hashes module
- synapse.lib.stormlib.hex module
- synapse.lib.stormlib.imap module
- synapse.lib.stormlib.infosec module
- synapse.lib.stormlib.ipv6 module
- synapse.lib.stormlib.iters module
- synapse.lib.stormlib.json module
- synapse.lib.stormlib.log module
- synapse.lib.stormlib.macro module
- synapse.lib.stormlib.math module
- synapse.lib.stormlib.mime module
- synapse.lib.stormlib.model module
- synapse.lib.stormlib.modelext module
- synapse.lib.stormlib.notifications module
- synapse.lib.stormlib.oauth module
- synapse.lib.stormlib.project module
- synapse.lib.stormlib.random module
- synapse.lib.stormlib.scrape module
- synapse.lib.stormlib.smtp module
- synapse.lib.stormlib.stix module
- synapse.lib.stormlib.storm module
- synapse.lib.stormlib.version module
- synapse.lib.stormlib.xml module
- synapse.lib.stormlib.yaml module
Submodules
synapse.lib.agenda module
- class synapse.lib.agenda.Agenda[source]
Bases:
synapse.lib.base.Base
Organize and execute all the scheduled storm queries in a cortex.
- async add(cdef)[source]
Persistently adds an appointment
- Parameters
cdef (dict) – Dictionary containing the Cron definition.
Notes
The cron definition may contain the following keys:
- creator (str)
Iden of the creating user.
- iden (str)
Iden of the appointment.
- storm (str)
The Storm query to run.
- reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…])
One or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.
- incunit (Union[None, TimeUnit])
The unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.
- incvals (Union[None, int, Iterable[int])
Count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)
If the values for req and incvals are both lists, all combinations of all values (the product) are used.
- Returns
Packed appointment definition
- class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)[source]
Bases:
object
Represents a single element of a single combination of an appointment
synapse.lib.aha module
- class synapse.lib.aha.AhaApi[source]
Bases:
synapse.lib.cell.CellApi
- async addAhaSvc(name, info, network=None)[source]
Register a service with the AHA discovery server.
- NOTE: In order for the service to remain marked “up” a caller
must maintain the telepath link.
- addAhaSvcProv(name, provinfo=None)[source]
Provision the given relative service name within the configured network name.
- addAhaUserEnroll(name, userinfo=None, again=False)[source]
Create and return a one-time user enroll key.
- async getAhaSvc(name, filters=None)[source]
Return an AHA service description dictionary for a service name.
- async getAhaSvcMirrors(name)[source]
Return list of AHA svcinfo dictionaries for mirrors of a service.
- class synapse.lib.aha.AhaCell[source]
Bases:
synapse.lib.cell.Cell
- cellapi
alias of
synapse.lib.aha.AhaApi
- confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': False, 'hidedocs': False, 'type': ['string', 'null']}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. (may take a while)', 'type': 'boolean'}}
- confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'provision:listen': {'description': 'A telepath URL for the AHA provisioning listener.', 'type': ['string', 'null']}}
- class synapse.lib.aha.AhaProvisionServiceV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.aha.ProvDmon[source]
Bases:
synapse.daemon.Daemon
synapse.lib.ast module
- class synapse.lib.ast.AbsProp(valu, kids=())[source]
Bases:
synapse.lib.ast.Const
- class synapse.lib.ast.AbsPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.AndCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
<cond> and <cond>
- class synapse.lib.ast.ArgvQuery(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.ArrayCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.AstNode(kids=())[source]
Bases:
object
Base class for all nodes in the Storm abstract syntax tree.
- class synapse.lib.ast.Bool(valu, kids=())[source]
Bases:
synapse.lib.ast.Const
- class synapse.lib.ast.BreakOper(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.CallArgs(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.CallKwarg(kids=())[source]
Bases:
synapse.lib.ast.CallArgs
- class synapse.lib.ast.CallKwargs(kids=())[source]
Bases:
synapse.lib.ast.CallArgs
- class synapse.lib.ast.CaseEntry(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.CatchBlock(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.CmdOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.Cmpr(valu, kids=())[source]
Bases:
synapse.lib.ast.Const
- class synapse.lib.ast.Cond(kids=())[source]
Bases:
synapse.lib.ast.Value
A condition that is evaluated to filter nodes.
- class synapse.lib.ast.Const(valu, kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.ContinueOper(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.DollarExpr(kids=())[source]
Bases:
synapse.lib.ast.Value
Top level node for $(…) expressions
- class synapse.lib.ast.Edit(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.EditEdgeAdd(kids=(), n2=False)[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditEdgeDel(kids=(), n2=False)[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditNodeAdd(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditParens(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditPropDel(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditPropSet(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditTagAdd(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditTagDel(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EditTagPropDel(kids=())[source]
Bases:
synapse.lib.ast.Edit
[ -#foo.bar:baz ]
- class synapse.lib.ast.EditTagPropSet(kids=())[source]
Bases:
synapse.lib.ast.Edit
[ #foo.bar:baz=10 ]
- class synapse.lib.ast.EditUnivDel(kids=())[source]
Bases:
synapse.lib.ast.Edit
- class synapse.lib.ast.EmbedQuery(valu, kids=())[source]
Bases:
synapse.lib.ast.Const
- class synapse.lib.ast.Emit(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.ExprAndNode(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.ExprDict(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.ExprList(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.ExprNode(kids=())[source]
Bases:
synapse.lib.ast.Value
A binary (i.e. two argument) expression node
- class synapse.lib.ast.ExprOrNode(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.FiltByArray(kids=())[source]
Bases:
synapse.lib.ast.FiltOper
+:foo*[^=visi]
- class synapse.lib.ast.FiltOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.FiniBlock(kids=())[source]
Bases:
synapse.lib.ast.AstNode
An AST node that runs only once after all nodes have been consumed.
Example
Using a fini block:
fini { // stuff here runs *once* after the last node yield (even if there are no nodes) }
Notes
A fini block must be runtsafe.
- class synapse.lib.ast.ForLoop(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.FormName(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.FormPivot(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
-> foo:bar
- class synapse.lib.ast.FormTagProp(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.FormatString(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.FuncArgs(kids=())[source]
Bases:
synapse.lib.ast.AstNode
Represents the function arguments in a function definition
- class synapse.lib.ast.FuncCall(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.Function(kids=())[source]
Bases:
synapse.lib.ast.AstNode
( name, args, body )
// use args/kwargs syntax function bar(x, v=$(30)) { }
# we auto-detect the behavior of the target function
# return a value function bar(x, y) { return ($(x + y)) }
# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }
$foo = $bar(10, v=20)
- class synapse.lib.ast.HasAbsPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.HasRelPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.HasTagPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.IfClause(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.IfStmt(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.InitBlock(kids=())[source]
Bases:
synapse.lib.ast.AstNode
An AST node that runs only once before yielding nodes.
Example
Using a init block:
init { // stuff here runs *once* before the first node yield (even if there are no nodes) }
- class synapse.lib.ast.LiftByArray(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
:prop*[range=(200, 400)]
- class synapse.lib.ast.LiftFormTag(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
- class synapse.lib.ast.LiftFormTagProp(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
hehe:haha#foo.bar:baz [ = x ]
- class synapse.lib.ast.LiftOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.LiftProp(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
- class synapse.lib.ast.LiftPropBy(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
- class synapse.lib.ast.LiftTag(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
- class synapse.lib.ast.LiftTagProp(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
#foo.bar:baz [ = x ]
- class synapse.lib.ast.LiftTagTag(kids=())[source]
Bases:
synapse.lib.ast.LiftOper
##foo.bar
- class synapse.lib.ast.List(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.LookList(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.Lookup(kids, autoadd=False)[source]
Bases:
synapse.lib.ast.Query
When storm input mode is “lookup”
- class synapse.lib.ast.N1Walk(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.N1WalkNPivo(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOut
- class synapse.lib.ast.N2Walk(kids=())[source]
Bases:
synapse.lib.ast.N1Walk
- class synapse.lib.ast.N2WalkNPivo(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotIn
- class synapse.lib.ast.NotCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
not <cond>
- class synapse.lib.ast.Oper(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.OrCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
<cond> or <cond>
- class synapse.lib.ast.PivotIn(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
<- *
- class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
<- foo:edge
- class synapse.lib.ast.PivotOper(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.PivotOut(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
-> *
- class synapse.lib.ast.PivotToTags(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present
- class synapse.lib.ast.PropName(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.PropPivot(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
:foo -> bar:foo
- class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
:prop -> *
- class synapse.lib.ast.PropValue(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.Query(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.RawPivot(kids=(), isjoin=False)[source]
Bases:
synapse.lib.ast.PivotOper
-> { <varsfrompath> }
- class synapse.lib.ast.RelProp(kids=())[source]
Bases:
synapse.lib.ast.PropName
- class synapse.lib.ast.RelPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
(:foo:bar or .univ) <cmpr> <value>
- class synapse.lib.ast.RelPropValue(kids=())[source]
Bases:
synapse.lib.ast.PropValue
- class synapse.lib.ast.Return(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.Search(kids=())[source]
Bases:
synapse.lib.ast.Query
- class synapse.lib.ast.SetItemOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz
- class synapse.lib.ast.SetVarOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.Stop(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.SubGraph(rules)[source]
Bases:
object
An Oper like object which generates a subgraph.
Notes
The rules format for the subgraph is shaped like the following:
rules = { 'degrees': 1, 'edges': True, 'filterinput': True, 'yieldfiltered': False, 'filters': [ '-(#foo or #bar)', '-(foo:bar or baz:faz)', ], 'pivots': [ '-> * | limit 100', '<- * | limit 100', ] 'forms': { 'inet:fqdn':{ 'filters': [], 'pivots': [], } '*': { 'filters': [], 'pivots': [], }, }, }
Nodes which were original seeds have path.meta(‘graph:seed’).
All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.
- class synapse.lib.ast.SubQuery(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.SubqCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.SwitchCase(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.TagCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
#foo.bar
- class synapse.lib.ast.TagMatch(kids=())[source]
Bases:
synapse.lib.ast.TagName
Like TagName, but can have asterisks
- class synapse.lib.ast.TagName(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.TagProp(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.TagPropCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.TagPropValue(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.TagValuCond(kids=())[source]
Bases:
synapse.lib.ast.Cond
- class synapse.lib.ast.TagValue(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.TryCatch(kids=())[source]
Bases:
synapse.lib.ast.AstNode
- class synapse.lib.ast.UnaryExprNode(kids=())[source]
Bases:
synapse.lib.ast.Value
A unary (i.e. single-argument) expression node
- class synapse.lib.ast.UnivProp(kids=())[source]
Bases:
synapse.lib.ast.RelProp
- class synapse.lib.ast.UnivPropValue(kids=())[source]
Bases:
synapse.lib.ast.PropValue
- class synapse.lib.ast.Value(kids=())[source]
Bases:
synapse.lib.ast.AstNode
The base class for all values and value expressions.
- class synapse.lib.ast.VarDeref(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.VarEvalOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)
- class synapse.lib.ast.VarList(valu, kids=())[source]
Bases:
synapse.lib.ast.Const
- class synapse.lib.ast.VarListSetOper(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.VarValue(kids=())[source]
Bases:
synapse.lib.ast.Value
- class synapse.lib.ast.WhileLoop(kids=())[source]
Bases:
synapse.lib.ast.Oper
- class synapse.lib.ast.YieldValu(kids=())[source]
Bases:
synapse.lib.ast.Oper
synapse.lib.autodoc module
- synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)[source]
Process a list of StormTypes doc information to add them to a a RstHelp object.
- Notes
This will create internal hyperlink link targets for each header item. The link prefix string must be given with the
linkprefix
argument.
- Parameters
page (RstHelp) – The RST page to add .
docinfo (dict) – A Stormtypes Doc.
linkprefix (str) – The RST link prefix string to use.
islib (bool) – Treat the data as a library. This will preface the header and attribute values with
$
and use full paths for attributes.lvl (int) – The base header level to use when adding headers to the page.
- Returns
None
- synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None, isstor=False)[source]
synapse.lib.base module
- class synapse.lib.base.Base[source]
Bases:
object
Base class for Synapse objects.
Acts as an observable, enables async init and fini.
Example
class Foo(Base):
async def __anit__(self, x, y):
await Base.__anit__(self)
await stuff(x, y)
foo = await Foo.anit(10)
Note
One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.
- async addSignalHandlers()[source]
Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.
- async dist(mesg)[source]
Distribute an existing event tuple.
- Parameters
mesg ((str,dict)) – An event tuple.
Example
await base.dist( (‘foo’,{‘bar’:’baz’}) )
- async enter_context(item)[source]
Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.
- Returns
The result of item’s own __aenter__ or __enter__() method.
- async fini()[source]
Shut down the object and notify any onfini() coroutines.
- Returns
Remaining ref count
- async fire(evtname, **info)[source]
Fire the given event name on the Base. Returns a list of the return values of each callback.
Example
- for ret in d.fire(‘woot’,foo=’asdf’):
print(‘got: %r’ % (ret,))
- incref()[source]
Increment the reference count for this base. This API may be optionally used to control fini().
- link(func)[source]
Add a callback function to receive all events.
Example
base1 = Base() base2 = Base()
base1.link( base2.dist )
# all events on base1 are also propagated on base2
- async main()[source]
Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )
Note
This API may only be used when the ioloop is also the main thread.
- off(evnt, func)[source]
Remove a previously registered event handler function.
Example
base.off( ‘foo’, onFooFunc )
- on(evnt, func, base=None)[source]
Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.
- Parameters
evnt (str) – An event name
func (function) – A callback function to receive event tufo
Examples
Add a callback function and fire it:
- async def baz(event):
x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y
d.on(‘foo’, baz)
# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)
- Return type
None
- onWith(evnt, func)[source]
A context manager which can be used to add a callback and remove it when using a
with
statement.- Parameters
evnt (str) – An event name
func (function) – A callback function to receive event tufo
- async postAnit()[source]
Method called after self.__anit__() has completed, but before anit() returns the object to the caller.
- schedCallSafe(func, *args, **kwargs)[source]
Schedule a function to run as soon as possible on the same event loop that this Base is running on.
This function does not pend on the function completion.
- Parameters
func –
*args –
**kwargs –
Notes
This method may be called from outside of the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.
- Returns
A Future representing the eventual function execution.
- Return type
concurrent.futures.Future
- schedCoro(coro)[source]
Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.
- Parameters
coro – The coroutine to schedule.
Notes
This function is not threadsafe and must be run on the Base’s event loop. Tasks created by this function do inherit the synapse.lib.scope Scope from the current task.
- Returns
An asyncio.Task object.
- Return type
asyncio.Task
- schedCoroSafe(coro)[source]
Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.
This function does not pend on coroutine completion.
Notes
This method may be run outside the event loop on a different thread. This function will break any task scoping done with synapse.lib.scope.
- Returns
A Future representing the eventual coroutine execution.
- Return type
concurrent.futures.Future
- schedCoroSafePend(coro)[source]
Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on
Note
This method may not be run inside an event loop
- unlink(func)[source]
Remove a callback function previously added with link()
Example
base.unlink( callback )
- waiter(count, *names)[source]
Construct and return a new Waiter for events on this base.
Example
# wait up to 3 seconds for 10 foo:bar events…
waiter = base.waiter(10,’foo:bar’)
# .. fire task that will cause foo:bar events
events = await waiter.wait(timeout=3)
- if events == None:
# handle the timeout case…
- for event in events:
# parse the events if you need…
Note
Use this with caution. It’s easy to accidentally construct race conditions with this mechanism ;)
- class synapse.lib.base.BaseRef[source]
Bases:
synapse.lib.base.Base
An object for managing multiple Base instances by name.
- async gen(name)[source]
Atomically get/gen a Base and incref. (requires ctor during BaseRef init)
- Parameters
name (str) – The name/iden of the Base instance.
- get(name)[source]
Retrieve a Base instance by name.
- Parameters
name (str) – The name/iden of the Base
- Returns
The Base instance (or None)
- Return type
(Base)
- pop(name)[source]
Remove and return a Base from the BaseRef.
- Parameters
name (str) – The name/iden of the Base instance
- Returns
The named base ( or None )
- Return type
(Base)
synapse.lib.boss module
- class synapse.lib.boss.Boss[source]
Bases:
synapse.lib.base.Base
An object to track “promoted” async tasks.
- async execute(coro, name, user, info=None, iden=None)[source]
Create a synapse task from the given coroutine.
- async promote(name, user, info=None, taskiden=None)[source]
Promote the currently running task.
- Parameters
name (str) – The name of the task.
user – The User who owns the task.
taskiden – An optional GUID for the task.
info – An optional information dictionary containing information about the task.
- Returns
The Synapse Task object.
- Return type
s_task.Task
synapse.lib.cache module
A few speed optimized (lockless) cache helpers. Use carefully.
- class synapse.lib.cache.LruDict(size=10000)[source]
Bases:
collections.abc.MutableMapping
Maintains the last n accessed keys
- class synapse.lib.cache.TagGlobs[source]
Bases:
object
An object that manages multiple tag globs and values for caching.
- synapse.lib.cache.memoizemethod(size=16384)[source]
A version of memoize that doesn’t cause GC cycles when applied to a method.
- synapse.lib.cache.regexizeTagGlob(tag)[source]
- Returns
a regular expression string with ** and * interpreted as tag globs
- Precondition:
tag is a valid tagmatch
Notes
A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.
The returned string does not contain a starting ‘^’ or trailing ‘$’.
synapse.lib.cell module
- class synapse.lib.cell.Cell[source]
Bases:
synapse.lib.nexus.Pusher
,synapse.telepath.Aware
A Cell() implements a synapse micro-service.
- A Cell has 5 phases of startup:
Universal cell data structures
Service specific storage/data (pre-nexs)
Nexus subsystem initialization
Service specific startup (with nexus)
Networking and mirror services
- BACKUP_SPAWN_TIMEOUT = 60.0
- COMMIT = ''
- FREE_SPACE_CHECK_FREQ = 60.0
- VERSION = (2, 125, 0)
- VERSTRING = '2.125.0'
- addActiveCoro(func, iden=None, base=None)[source]
Add a function callback to be run as a coroutine when the Cell is active.
- Parameters
func (coroutine function) – The function run as a coroutine.
iden (str) – The iden to use for the coroutine.
base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d
- Returns
A GUID string that identifies the coroutine for delActiveCoro()
- Return type
str
Note
This will re-fire the coroutine if it exits and the Cell is still active.
- cellapi
alias of
synapse.lib.cell.CellApi
- confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:provision': {'description': 'The telepath URL of the aha provisioning service.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'hidecmdl': True, 'hidedocs': True, 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'aha:user': {'description': 'The username of this service when connecting to others.', 'type': 'string'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class. Used by stemcell.', 'hideconf': True, 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the service.', 'hideconf': True, 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'description': 'Data used to drive configuration of the service upon first startup.', 'hidedocs': True, 'properties': {'roles': {'items': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}, 'users': {'items': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'limit:disk:free': {'default': 5, 'description': 'Minimum disk free space percentage before setting the cell read-only.', 'maximum': 100, 'minimum': 0, 'type': ['integer', 'null']}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': True, 'hidedocs': True, 'type': ['string', 'null']}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'hidecmdl': True, 'hidedocs': True, 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to a stream file on disk. Required for mirroring (on both sides).', 'type': 'boolean'}, 'onboot:optimize': {'default': False, 'description': 'Delay startup to optimize LMDB databases during boot to recover free space and increase performance. (may take a while)', 'type': 'boolean'}}
- confdefs = {}
- async delActiveCoro(iden)[source]
Remove an Active coroutine previously added with addActiveCoro().
- Parameters
iden (str) – The iden returned by addActiveCoro()
- async classmethod execmain(argv, outp=None)[source]
The main entry point for running the Cell as an application.
- Parameters
argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.
Notes
This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.
- Returns
None.
- async feedBeholder(name, info, gates=None, perms=None)[source]
Feed a named event onto the
cell:beholder
message bus that will sent to any listeners.- Parameters
info (dict) – An information dictionary to be sent to any consumers.
gates (list) – List of gate idens, whose details will be added to the outbound message(s).
perms (list) – List of permission names, whose details will be added to the outbound message(s).
- Returns
None
- classmethod getArgParser(conf=None)[source]
Get an
argparse.ArgumentParser
for the Cell.- Parameters
conf (s_config.Config) – Optional, a Config object which
Notes
Boot time configuration data is placed in the argument group called
config
. This adds defaultdirn
,--telepath
,--https
and--name
arguements to the argparser instance. Configuration values which have thehideconf
orhidecmdl
value set to True are not added to the argparser instance.- Returns
A ArgumentParser for the Cell.
- Return type
argparse.ArgumentParser
- async getCellApi(link, user, path)[source]
Get an instance of the telepath Client object for a given user, link and path.
- Parameters
link (s_link.Link) – The link object.
user (s_hive.HiveUser) – The heavy user object.
path (str) – The path requested.
Notes
This defaults to the self.cellapi class. Implementors may override the default class attribute for cellapi to share a different interface.
- Returns
The shared object for this cell.
- Return type
object
- async getCellInfo()[source]
Return metadata specific for the Cell.
Notes
By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.
It is expected that implementers override the following Class attributes in order to provide meaningful version information:
COMMIT
- A Git CommitVERSION
- A Version tuple.VERSTRING
- A Version string.- Returns
A Dictionary of metadata.
- Return type
- async getLogExtra(**kwargs)[source]
Get an extra dictionary for structured logging which can be used as a extra argument for loggers.
- Parameters
**kwargs – Additional key/value items to add to the log.
- Returns
A dictionary
- Return type
- async getSystemInfo()[source]
Get info about the system in which the cell is running
- Returns
volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
cellstarttime - Cell start time in epoch milliseconds
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system
cpucount - Number of CPUs on system
- Return type
A dictionary with the following keys. All size values are in bytes
- async getTeleApi(link, mesg, path)[source]
Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)
- async handoff(turl, timeout=30)[source]
Hand off leadership to a mirror in a transactional fashion.
- classmethod initCellConf(conf=None)[source]
Create a Config object for the Cell.
- Parameters
conf (s_config.Config) – An optional config structure. This has _opts_data taken from it.
Notes
The Config object has a
envar_prefix
set according to the results ofcls.getEnvPrefix()
.- Returns
A Config helper object.
- Return type
s_config.Config
- async classmethod initFromArgv(argv, outp=None)[source]
Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.
- Parameters
argv (list) – A list of command line arguments to launch the Cell with.
outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.
Notes
- This does the following items:
Create a Config object from the Cell class.
Creates an Argument Parser from the Cell class and Config object.
Parses the provided arguments.
Loads configuration data from the parsed options and environment variables.
Sets logging for the process.
Creates the Cell from the Cell Ctor.
Adds a Telepath listener, HTTPs port listeners and Telepath share names.
Returns the Cell.
- Returns
This returns an instance of the Cell.
- Return type
- modCellConf(conf)[source]
Modify the Cell’s ondisk configuration overrides file and runtime configuration.
- Parameters
conf (dict) – A dictionary of items to set.
Notes
This does require the data being set to be schema valid.
- Returns
None.
- popCellConf(name)[source]
Remove a key from the Cell’s ondisk configuration overrides file and runtime configuration.
- Parameters
name (str) – Name of the value to remove.
Notes
This does not modify the cell.yaml file. This does re-validate the configuration after removing the value, so if the value removed had a default populated by schema, that default would be reset.
- Returns
None
- async popHiveKey(path)[source]
Remove and return the value of a key in the cell default hive.
Note: this is for expert emergency use only.
- async promote(graceful=False)[source]
Transform this cell from a passive follower to an active cell that writes changes locally.
- class synapse.lib.cell.CellApi[source]
Bases:
synapse.lib.base.Base
- async allowed(perm, default=None)[source]
Check if the user has the requested permission.
- Parameters
perm – permission path components to check
default – Value returned if no value stored
Examples
Form a path and check the permission from a remote proxy:
perm = ('node', 'add', 'inet:ipv4') allowed = await prox.allowed(perm) if allowed: dostuff()
- Returns
True if the user has permission, False if explicitly denied, None if no entry
- Return type
Optional[bool]
- cullNexsLog(offs)[source]
Remove Nexus log entries up to (and including) the given offset.
Note
If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.
Only rotated logs where the last index is less than the provided offset will be removed from disk.
- Parameters
offs (int) – The offset to remove entries up to.
- Returns
Whether the cull was executed
- Return type
bool
- delBackup(name)[source]
Delete a backup by name.
- Parameters
name (str) – The name of the backup to delete.
- getAuthUsers(archived=False)[source]
- Parameters
archived (bool) – If true, list all users, else list non-archived users
- getBackupInfo()[source]
Get information about recent backup activity.
- Returns
currduration - If backup currently running, time in ms since backup started, otherwise None
laststart - Last time (in epoch milliseconds) a backup started
lastend - Last time (in epoch milliseconds) a backup ended
lastduration - How long last backup took in ms
lastsize - Disk usage of last backup completed
lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive
lastexception - Tuple of exception information if last backup failed, otherwise None
- Return type
(dict) It has the following keys
Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.
- getBackups()[source]
Retrieve a list of backups.
- Returns
A list of backup names.
- Return type
list[str]
- getGcInfo()[source]
For diagnostic purposes only!
NOTE: This API is not supported and can be removed at any time!
- getSystemInfo()[source]
Get info about the system in which the cell is running
- Returns
volsize - Volume where cell is running total space
volfree - Volume where cell is running free space
backupvolsize - Backup directory volume total space
backupvolfree - Backup directory volume free space
celluptime - Cell uptime in milliseconds
cellrealdisk - Cell’s use of disk, equivalent to du
cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l
osversion - OS version/architecture
pyversion - Python version
totalmem - Total memory in the system
availmem - Available memory in the system
- Return type
A dictionary with the following keys. All size values are in bytes
- iterBackupArchive(name)[source]
Retrieve a backup by name as a compressed stream of bytes.
Note: Compression and streaming will occur from a separate process.
- Parameters
name (str) – The name of the backup to retrieve.
- iterNewBackupArchive(name=None, remove=False)[source]
Run a new backup and return it as a compressed stream of bytes.
Note: Compression and streaming will occur from a separate process.
- Parameters
name (str) – The name of the backup to retrieve.
remove (bool) – Delete the backup after streaming.
- rotateNexsLog()[source]
Rotate the Nexus log at the current offset.
- Returns
The starting index of the active Nexus log
- Return type
int
- runBackup(name=None, wait=True)[source]
Run a new backup.
- Parameters
name (str) – The optional name of the backup.
wait (bool) – On True, wait for backup to complete before returning.
- Returns
The name of the newly created backup.
- Return type
str
- runGcCollect(generation=2)[source]
For diagnostic purposes only!
NOTE: This API is not supported and can be removed at any time!
- setCellUser(iden)[source]
Switch to another user (admin only).
This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.
- trimNexsLog(consumers=None, timeout=60)[source]
Rotate and cull the Nexus log (and those of any consumers) at the current offset.
Note
If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.
- Parameters
consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.
timeout (int) – Time in seconds to wait for downstream consumers to be caught up.
- Returns
The offset that the Nexus log was culled up to and including.
- Return type
int
- synapse.lib.cell.SLAB_MAP_SIZE = 134217728
Base classes for the synapse “cell” microservice architecture.
synapse.lib.certdir module
- class synapse.lib.certdir.CertDir(path=None)[source]
Bases:
object
Certificate loading/generation/signing utilities.
- Features:
Locates and load certificates, keys, and certificate signing requests (CSRs).
Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.
Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).
Signs certificate signing requests (CSRs).
Generates PKCS#12 archives for use in browser.
- Parameters
path (str) – Optional path which can override the default path directory.
Notes
All certificates will be loaded from and written to ~/.syn/certs by default. Set the environment variable SYN_CERT_DIR to override.
All certificate generation methods create 4096 bit RSA keypairs.
All certificate signing methods use sha256 as the signature algorithm.
CertDir does not currently support signing CA CSRs.
- genCaCert(name, signas=None, outp=None, save=True)[source]
Generates a CA keypair.
- Parameters
name (str) – The name of the CA keypair.
signas (str) – The CA keypair to sign the new CA with.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Make a CA named “myca”:
mycakey, mycacert = cdir.genCaCert(‘myca’)
- Returns
Tuple containing the private key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- genCaCrl(name)[source]
Get the CRL for a given CA.
- Parameters
name (str) – The CA name.
- Returns
The CRL object.
- Return type
- genClientCert(name, outp=None)[source]
Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material.
- Parameters
name (str) – The name of the user keypair.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Make the PKC12 object for user “myuser”:
myuserpkcs12 = cdir.genClientCert(‘myuser’)
- Returns
The PKCS #12 archive.
- Return type
OpenSSL.crypto.PKCS12
- genCodeCert(name, signas=None, outp=None, save=True)[source]
Generates a code signing keypair.
- Parameters
name (str) – The name of the code signing cert.
signas (str) – The CA keypair to sign the new code keypair with.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Generate a code signing cert for the name “The Vertex Project”:
myuserkey, myusercert = cdir.genCodeCert(‘The Vertex Project’)
- Returns
Tuple containing the key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- genHostCert(name, signas=None, outp=None, csr=None, sans=None, save=True)[source]
Generates a host keypair.
- Parameters
name (str) – The name of the host keypair.
signas (str) – The CA keypair to sign the new host keypair with.
outp (synapse.lib.output.Output) – The output buffer.
csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.
sans (list) – List of subject alternative names.
Examples
Make a host keypair named “myhost”:
myhostkey, myhostcert = cdir.genHostCert(‘myhost’)
- Returns
Tuple containing the private key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- genHostCsr(name, outp=None)[source]
Generates a host certificate signing request.
- Parameters
name (str) – The name of the host CSR.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Generate a CSR for the host key named “myhost”:
cdir.genHostCsr(‘myhost’)
- Returns
The bytes of the CSR.
- Return type
bytes
- genUserCert(name, signas=None, outp=None, csr=None, save=True)[source]
Generates a user keypair.
- Parameters
name (str) – The name of the user keypair.
signas (str) – The CA keypair to sign the new user keypair with.
outp (synapse.lib.output.Output) – The output buffer.
csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.
Examples
Generate a user cert for the user “myuser”:
myuserkey, myusercert = cdir.genUserCert(‘myuser’)
- Returns
Tuple containing the key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- genUserCsr(name, outp=None)[source]
Generates a user certificate signing request.
- Parameters
name (str) – The name of the user CSR.
outp (synapse.lib.output.Output) – The output buffer.
Examples
Generate a CSR for the user “myuser”:
cdir.genUserCsr(‘myuser’)
- Returns
The bytes of the CSR.
- Return type
bytes
- getCaCert(name)[source]
Loads the X509 object for a given CA.
- Parameters
name (str) – The name of the CA keypair.
Examples
Get the certificate for the CA “myca”
mycacert = cdir.getCaCert(‘myca’)
- Returns
The certificate, if exists.
- Return type
OpenSSL.crypto.X509
- getCaCertPath(name)[source]
Gets the path to a CA certificate.
- Parameters
name (str) – The name of the CA keypair.
Examples
Get the path to the CA certificate for the CA “myca”:
mypath = cdir.getCACertPath(‘myca’)
- Returns
The path if exists.
- Return type
str
- getCaCerts()[source]
Return a list of CA certs from the CertDir.
- Returns
List of CA certificates.
- Return type
[OpenSSL.crypto.X509]
- getCaKey(name)[source]
Loads the PKey object for a given CA keypair.
- Parameters
name (str) – The name of the CA keypair.
Examples
Get the private key for the CA “myca”:
mycakey = cdir.getCaKey(‘myca’)
- Returns
The private key, if exists.
- Return type
OpenSSL.crypto.PKey
- getCaKeyPath(name)[source]
Gets the path to a CA key.
- Parameters
name (str) – The name of the CA keypair.
Examples
Get the path to the private key for the CA “myca”:
mypath = cdir.getCAKeyPath(‘myca’)
- Returns
The path if exists.
- Return type
str
- getClientCert(name)[source]
Loads the PKCS12 archive object for a given user keypair.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the PKCS12 object for the user “myuser”:
mypkcs12 = cdir.getClientCert(‘myuser’)
Notes
The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool
- Returns
The PKCS12 archive, if exists.
- Return type
OpenSSL.crypto.PKCS12
- getClientCertPath(name)[source]
Gets the path to a client certificate.
- Parameters
name (str) – The name of the client keypair.
Examples
Get the path to the client certificate for “myuser”:
mypath = cdir.getClientCertPath(‘myuser’)
- Returns
The path if exists.
- Return type
str
- getClientSSLContext(certname=None)[source]
Returns an ssl.SSLContext appropriate for initiating a TLS session
- Parameters
certname – If specified, use the user certificate with the matching name to authenticate to the remote service.
- Returns
A SSLContext object.
- Return type
ssl.SSLContext
- getHostCaPath(name)[source]
Gets the path to the CA certificate that issued a given host keypair.
- Parameters
name (str) – The name of the host keypair.
Examples
Get the path to the CA cert which issue the cert for “myhost”:
mypath = cdir.getHostCaPath(‘myhost’)
- Returns
The path if exists.
- Return type
str
- getHostCert(name)[source]
Loads the X509 object for a given host keypair.
- Parameters
name (str) – The name of the host keypair.
Examples
Get the certificate object for the host “myhost”:
myhostcert = cdir.getHostCert(‘myhost’)
- Returns
The certificate, if exists.
- Return type
OpenSSL.crypto.X509
- getHostCertPath(name)[source]
Gets the path to a host certificate.
- Parameters
name (str) – The name of the host keypair.
Examples
Get the path to the host certificate for the host “myhost”:
mypath = cdir.getHostCertPath(‘myhost’)
- Returns
The path if exists.
- Return type
str
- getHostKey(name)[source]
Loads the PKey object for a given host keypair.
- Parameters
name (str) – The name of the host keypair.
Examples
Get the private key object for the host “myhost”:
myhostkey = cdir.getHostKey(‘myhost’)
- Returns
The private key, if exists.
- Return type
OpenSSL.crypto.PKey
- getHostKeyPath(name)[source]
Gets the path to a host key.
- Parameters
name (str) – The name of the host keypair.
Examples
Get the path to the host key for the host “myhost”:
mypath = cdir.getHostKeyPath(‘myhost’)
- Returns
The path if exists.
- Return type
str
- getServerSSLContext(hostname=None, caname=None)[source]
Returns an ssl.SSLContext appropriate to listen on a socket
- Parameters
hostname – If None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory.
caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.
- Returns
A SSLContext object.
- Return type
ssl.SSLContext
- getUserCaPath(name)[source]
Gets the path to the CA certificate that issued a given user keypair.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the path to the CA cert which issue the cert for “myuser”:
mypath = cdir.getUserCaPath(‘myuser’)
- Returns
The path if exists.
- Return type
str
- getUserCert(name)[source]
Loads the X509 object for a given user keypair.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the certificate object for the user “myuser”:
myusercert = cdir.getUserCert(‘myuser’)
- Returns
The certificate, if exists.
- Return type
OpenSSL.crypto.X509
- getUserCertPath(name)[source]
Gets the path to a user certificate.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the path for the user cert for “myuser”:
mypath = cdir.getUserCertPath(‘myuser’)
- Returns
The path if exists.
- Return type
str
- getUserForHost(user, host)[source]
Gets the name of the first existing user cert for a given user and host.
- Parameters
user (str) – The name of the user.
host (str) – The name of the host.
Examples
Get the name for the “myuser” user cert at “cool.vertex.link”:
usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)
- Returns
The cert name, if exists.
- Return type
str
- getUserKey(name)[source]
Loads the PKey object for a given user keypair.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the key object for the user key for “myuser”:
myuserkey = cdir.getUserKey(‘myuser’)
- Returns
The private key, if exists.
- Return type
OpenSSL.crypto.PKey
- getUserKeyPath(name)[source]
Gets the path to a user key.
- Parameters
name (str) – The name of the user keypair.
Examples
Get the path to the user key for “myuser”:
mypath = cdir.getUserKeyPath(‘myuser’)
- Returns
The path if exists.
- Return type
str
- importFile(path, mode, outp=None)[source]
Imports certs and keys into the Synapse cert directory
- Parameters
path (str) – The path of the file to be imported.
mode (str) – The certdir subdirectory to import the file into.
Examples
Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory.
certdir.importFile(‘mycoolca.crt’, ‘cas’)
Notes
importFile does not perform any validation on the files it imports.
- Returns
None
- isCaCert(name)[source]
Checks if a CA certificate exists.
- Parameters
name (str) – The name of the CA keypair.
Examples
Check if the CA certificate for “myca” exists:
exists = cdir.isCaCert(‘myca’)
- Returns
True if the certificate is present, False otherwise.
- Return type
bool
- isClientCert(name)[source]
Checks if a user client certificate (PKCS12) exists.
- Parameters
name (str) – The name of the user keypair.
Examples
Check if the client certificate “myuser” exists:
exists = cdir.isClientCert(‘myuser’)
- Returns
True if the certificate is present, False otherwise.
- Return type
bool
- isHostCert(name)[source]
Checks if a host certificate exists.
- Parameters
name (str) – The name of the host keypair.
Examples
Check if the host cert “myhost” exists:
exists = cdir.isUserCert(‘myhost’)
- Returns
True if the certificate is present, False otherwise.
- Return type
bool
- isUserCert(name)[source]
Checks if a user certificate exists.
- Parameters
name (str) – The name of the user keypair.
Examples
Check if the user cert “myuser” exists:
exists = cdir.isUserCert(‘myuser’)
- Returns
True if the certificate is present, False otherwise.
- Return type
bool
- loadCertByts(byts)[source]
Load a X509 certificate from its PEM encoded bytes.
- Parameters
byts (bytes) – The PEM encoded bytes of the certificate.
- Returns
The X509 certificate.
- Return type
OpenSSL.crypto.X509
- Raises
BadCertBytes – If the certificate bytes are invalid.
- selfSignCert(cert, pkey)[source]
Self-sign a certificate.
- Parameters
cert (OpenSSL.crypto.X509) – The certificate to sign.
pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.
Examples
Sign a given certificate with a given private key:
cdir.selfSignCert(mycert, myotherprivatekey)
- Returns
None
- signCertAs(cert, signas)[source]
Signs a certificate with a CA keypair.
- Parameters
cert (OpenSSL.crypto.X509) – The certificate to sign.
signas (str) – The CA keypair name to sign the new keypair with.
Examples
Sign a certificate with the CA “myca”:
cdir.signCertAs(mycert, ‘myca’)
- Returns
None
- signHostCsr(xcsr, signas, outp=None, sans=None, save=True)[source]
Signs a host CSR with a CA keypair.
- Parameters
xcsr (OpenSSL.crypto.X509Req) – The certificate signing request.
signas (str) – The CA keypair name to sign the CSR with.
outp (synapse.lib.output.Output) – The output buffer.
sans (list) – List of subject alternative names.
Examples
Sign a host key with the CA “myca”:
cdir.signHostCsr(mycsr, ‘myca’)
- Returns
Tuple containing the public key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- signUserCsr(xcsr, signas, outp=None, save=True)[source]
Signs a user CSR with a CA keypair.
- Parameters
xcsr (OpenSSL.crypto.X509Req) – The certificate signing request.
signas (str) – The CA keypair name to sign the CSR with.
outp (synapse.lib.output.Output) – The output buffer.
Examples
cdir.signUserCsr(mycsr, ‘myca’)
- Returns
Tuple containing the public key and certificate objects.
- Return type
((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))
- valCodeCert(byts)[source]
Verify a code cert is valid according to certdir’s available CAs and CRLs.
- Parameters
byts (bytes) – The certificate bytes.
- Returns
The certificate.
- Return type
OpenSSL.crypto.X509
- valUserCert(byts, cacerts=None)[source]
Validate the PEM encoded x509 user certificate bytes and return it.
- Parameters
byts (bytes) – The bytes for the User Certificate.
cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates.
- Raises
BadCertVerify – If the certificate is not valid.
- Returns
The certificate, if it is valid.
- Return type
OpenSSL.crypto.X509
- synapse.lib.certdir.getCertDir() synapse.lib.certdir.CertDir [source]
Get the singleton CertDir instance.
- Returns
A certdir object.
- Return type
- synapse.lib.certdir.getCertDirn() str [source]
Get the expanded default path used by the singleton CertDir instance.
- Returns
The path string.
- Return type
str
- synapse.lib.certdir.getServerSSLContext() ssl.SSLContext [source]
Get a server SSLContext object.
This object has a minimum TLS version of 1.2, a subset of ciphers in use, and disabled client renegotiation.
This object has no certificates loaded in it.
- Returns
The context object.
- Return type
ssl.SSLContext
synapse.lib.chop module
- synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)
Shared primitive routines for chopping up strings and values.
- synapse.lib.chop.hexstr(text)[source]
Ensure a string is valid hex.
- Parameters
text (str) – String to normalize.
Examples
Norm a few strings:
hexstr(‘0xff00’) hexstr(‘ff00’)
Notes
Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.
- Returns
Normalized hex string.
- Return type
str
- synapse.lib.chop.replaceUnicodeDashes(valu)[source]
Replace unicode dashes in a string with regular dashes.
- Parameters
valu (str) – A string.
- Returns
A new string with replaced dashes.
- Return type
str
synapse.lib.cli module
- class synapse.lib.cli.Cli[source]
Bases:
synapse.lib.base.Base
A modular / event-driven CLI base object.
- async addSignalHandlers()[source]
Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.
- histfile = 'cmdr_history'
- async runCmdLine(line)[source]
Run a single command line.
- Parameters
line (str) – Line to execute.
Examples
Execute the ‘woot’ command with the ‘help’ switch:
await cli.runCmdLine(‘woot –help’)
- Returns
Arbitrary data from the cmd class.
- Return type
object
- class synapse.lib.cli.Cmd(cli, **opts)[source]
Bases:
object
Base class for modular commands in the synapse CLI.
- getCmdOpts(text)[source]
Use the _cmd_syntax def to split/parse/normalize the cmd line.
- Parameters
text (str) – Command to process.
Notes
This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type
- Returns
An opts dictionary.
- Return type
dict
- class synapse.lib.cli.CmdHelp(cli, **opts)[source]
Bases:
synapse.lib.cli.Cmd
List commands and display help output.
Example
help foocmd
- class synapse.lib.cli.CmdLocals(cli, **opts)[source]
Bases:
synapse.lib.cli.Cmd
List the current locals for a given CLI object.
- class synapse.lib.cli.CmdQuit(cli, **opts)[source]
Bases:
synapse.lib.cli.Cmd
Quit the current command line interpreter.
Example
quit
synapse.lib.cmd module
synapse.lib.cmdr module
- async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)[source]
Construct and return a cmdr for the given remote cell.
- Parameters
cell – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.
Examples
Get the cmdr for a proxy:
cmdr = await getItemCmdr(foo)
- Returns
A Cli instance with Cmds loaeded into it.
- Return type
s_cli.Cli
- async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)[source]
Create a cmdr for the given item and run the cmd loop.
- Parameters
item – Cell proxy being commanded.
outp – Output helper object.
color (bool) – If true, enable colorized output.
**opts – Additional options pushed into the Cmdr locs.
Notes
This function does not return while the command loop is run.
Examples
Run the Cmdr for a proxy:
await runItemCmdr(foo)
- Returns
This function returns None.
- Return type
None
synapse.lib.config module
- class synapse.lib.config.Config(schema, conf=None, envar_prefixes=None)[source]
Bases:
collections.abc.MutableMapping
Synapse configuration helper based on JSON Schema.
- Parameters
schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.
conf (dict) – Optional, a set of configuration data to preload.
envar_prefixes (list) – Optional, a list of prefix strings used when collecting configuration data from environment variables.
Notes
This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.
The default values provided in the schema must be able to be recreated from the repr() of their Python value.
Default values are not loaded into the configuration data until the
reqConfValid()
method is called.- asDict()[source]
Get a copy of configuration data.
- Returns
A copy of the configuration data.
- Return type
dict
- classmethod getConfFromCell(cell, conf=None, envar_prefixes=None)[source]
Get a Config object from a Cell directly (either the ctor or the instance thereof).
- Returns
A Config object.
- Return type
- getEnvarMapping(prefix=None)[source]
Get a mapping of config values to envars.
Configuration values which have the
hideconf
value set to True are not resolved from environment variables.
- reqConfValid()[source]
Validate that the loaded configuration data is valid according to the schema.
Notes
The validation set does set any default values which are not currently set for configuration options.
- Returns
This returns nothing.
- Return type
None
- reqConfValu(key)[source]
Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.
- Parameters
key (str) – The key to require.
- Returns
The requested value.
- reqKeyValid(key, value)[source]
Test if a key is valid for the provided schema it is associated with.
- Parameters
key (str) – Key to check.
value – Value to check.
- Raises
BadArg – If the key has no associated schema.
BadConfValu – If the data is not schema valid.
- Returns
None when valid.
- setConfFromEnvs()[source]
Set configuration options from environment variables.
Notes
Environment variables are resolved from configuration options after doing the following transform:
Replace
:
characters with_
.Add a config provided prefix, if set.
Uppercase the string.
Resolve the environment variable
If the environment variable is set, set the config value to the results of
yaml.yaml_safeload()
on the value.
Configuration values which have the
hideconf
value set to True are not resolved from environment variables.Examples
For the configuration value
auth:passwd
, the environment variable is resolved asAUTH_PASSWD
. With the prefixcortex
, the the environment variable is resolved asCORTEX_AUTH_PASSWD
.- Returns
Returns a dictionary of values which were set from enviroment variables.
- Return type
dict
- synapse.lib.config.getJsSchema(confbase, confdefs)[source]
Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.
- Parameters
confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.
confdefs (dict) – A JSON Schema dictionary of properties for the object.
Notes
This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify
- Returns
A complete JSON schema.
- Return type
dict
- synapse.lib.config.getJsValidator(schema, use_default=True)[source]
Get a fastjsonschema callable.
- Parameters
schema (dict) – A JSON Schema object.
use_default (bool) – Whether to insert “default” key arguments into the validated data structure.
- Returns
A callable function that can be used to validate data against the json schema.
- Return type
callable
- synapse.lib.config.make_envar_name(key, prefix=None)[source]
Convert a colon delimited string into an uppercase, underscore delimited string.
- Parameters
key (str) – Config key to convert.
prefix (str) – Optional string prefix to prepend the the config key.
- Returns
The string to lookup against a envar.
- Return type
str
synapse.lib.const module
synapse.lib.coro module
Async/Coroutine related utilities.
- async synapse.lib.coro.agen(item)[source]
Wrap an async_generator or generator in an async_generator.
Notes
Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.
- async synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None)[source]
Wait on an an asyncio event with an optional timeout
- Returns
true if the event got set, False if timed out
- synapse.lib.coro.executor(func, *args, **kwargs)[source]
Execute a non-coroutine function in the ioloop executor pool.
- Parameters
func – Function to execute.
*args – Args for the function.
**kwargs – Kwargs for the function.
Examples
Execute a blocking API call in the executor pool:
import requests def block(url, params=None): return requests.get(url, params=params).json() fut = s_coro.executor(block, 'http://some.tld/thign') resp = await fut
- Returns
An asyncio future.
- Return type
asyncio.Future
- async synapse.lib.coro.forked(func, *args, **kwargs)[source]
Execute a target function in the forked process pool.
- Parameters
func – The target function.
*args – Function positional arguments.
**kwargs – Function keyword arguments.
- Returns
The target function return.
- Raises
The function may raise from the target function, or raise a s_exc.FatalErr in the event of a broken forked –
process pool. The fatalerr represents a unrecoverable application state. –
- async synapse.lib.coro.ornot(func, *args, **kwargs)[source]
Calls func and awaits it if a returns a coroutine.
Note
This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.
This is also useful when calling a callback that might either be a coroutine function or a regular function.
- Usage:
ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)
- async synapse.lib.coro.spawn(todo, timeout=None, ctx=None, log_conf=None)[source]
Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.
- Parameters
todo (tuple) – A tuple of function,
*args
, and**kwargs
.timeout (int) – The timeout to wait for the todo function to finish.
ctx (multiprocess.Context) – A optional multiprocessing context object.
log_conf (dict) – An optional logging configuration for the spawned process.
Notes
The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.
- Returns
The return value of executing the todo function.
synapse.lib.datfile module
Utilities for handling data files embedded within python packages.
synapse.lib.dyndeps module
- synapse.lib.dyndeps.getDynLocal(name)[source]
Dynamically import a python module and return a local.
Example
cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()
- synapse.lib.dyndeps.getDynMod(name)[source]
Dynamically import a python module and return a ref (or None).
Example
mod = getDynMod(‘foo.bar’)
- synapse.lib.dyndeps.runDynTask(task)[source]
Run a dynamic task and return the result.
Example
foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )
- synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)[source]
Dynamically import a module and call a function or raise an exception.
synapse.lib.encoding module
- synapse.lib.encoding.decode(name, byts, **opts)[source]
Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.
Example
byts = s_encoding.decode(‘base64’,byts)
- Note: Decoder names may also be prefixed with +
to encode for that name/layer.
- synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)[source]
Iterate through the data provided by a file like object.
Optional parameters may be used to control how the data is deserialized.
Examples
The following example show use of the iterdata function.:
with open('foo.csv','rb') as fd: for row in iterdata(fd, format='csv', encoding='utf8'): dostuff(row)
- Parameters
fd (file) – File like object to iterate over.
close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.
**opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.
- Yields
An item to process. The type of the item is dependent on the format parameters.
synapse.lib.gis module
- synapse.lib.gis.bbox(lat, lon, dist)[source]
Calculate a min/max bounding box for the circle defined by lalo/dist.
- Parameters
lat (float) – The latitude in degrees
lon (float) – The longitude in degrees
dist (int) – A distance in geo:dist base units (mm)
- Returns
(latmin, latmax, lonmin, lonmax)
- Return type
(float,float,float,float)
- synapse.lib.gis.dms2dec(degs, mins, secs)[source]
Convert degrees, minutes, seconds lat/long form to degrees float.
- Parameters
degs (int) – Degrees
mins (int) – Minutes
secs (int) – Seconds
- Returns
Degrees
- Return type
(float)
- synapse.lib.gis.haversine(px, py, r=6371008800.0)[source]
Calculate the haversine distance between two points defined by (lat,lon) tuples.
- Parameters
px ((float,float)) – lat/long position 1
py ((float,float)) – lat/long position 2
r (float) – Radius of sphere
- Returns
Distance in mm.
- Return type
(int)
- synapse.lib.gis.latlong(text)[source]
Chop a latlong string and return (float,float). Does not perform validation on the coordinates.
- Parameters
text (str) – A longitude,latitude string.
- Returns
A longitude, latitude float tuple.
- Return type
(float,float)
- synapse.lib.gis.near(point, dist, points)[source]
Determine if the given point is within dist of any of points.
- Parameters
point ((float,float)) – A latitude, longitude float tuple.
dist (int) – A distance in mm ( base units )
points (list) – A list of latitude, longitude float tuples to compare against.
synapse.lib.grammar module
synapse.lib.hashitem module
synapse.lib.hashset module
synapse.lib.health module
- class synapse.lib.health.HealthCheck(iden)[source]
Bases:
object
- update(name, status, mesg='', data=None)[source]
Append a new component to the Healcheck object.
- Parameters
name (str) – Name of the reported component.
status (str) – nomdinal/degraded/failed status code.
mesg (str) – Optional message about the component status.
data (dict) – Optional arbitrary dictionary of additional metadata about the component.
- Returns
None
synapse.lib.hive module
- class synapse.lib.hive.Hive[source]
Bases:
synapse.lib.nexus.Pusher
,synapse.telepath.Aware
An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.
- async dict(full, nexs=False)[source]
Open a HiveDict at the given full path.
- Parameters
full (tuple) – A full path tuple.
- Returns
A HiveDict for the full path.
- Return type
- dir(full)[source]
List subnodes of the given Hive path.
- Parameters
full (tuple) – A full path tuple.
Notes
This returns None if there is not a node at the path.
- Returns
A list of tuples. Each tuple contains the name, node value, and the number of children nodes.
- Return type
list
- async get(full, defv=None)[source]
Get the value of a node at a given path.
- Parameters
full (tuple) – A full path tuple.
- Returns
Arbitrary node value.
- async getHiveAuth()[source]
Retrieve a HiveAuth for hive standalone or non-cell uses.
Note
This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell
- async getTeleApi(link, mesg, path)[source]
Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)
- async open(full)[source]
Open and return a hive Node().
- Parameters
full (tuple) – A full path tuple.
- Returns
A Hive node.
- Return type
- class synapse.lib.hive.HiveApi[source]
Bases:
synapse.lib.base.Base
- class synapse.lib.hive.HiveDict[source]
Bases:
synapse.lib.base.Base
- class synapse.lib.hive.Node[source]
Bases:
synapse.lib.base.Base
A single node within the Hive tree.
- async dict(nexs=False)[source]
Get a HiveDict for this Node.
- Returns
A HiveDict for this Node.
- Return type
- class synapse.lib.hive.SlabHive[source]
Bases:
synapse.lib.hive.Hive
- class synapse.lib.hive.TeleHive[source]
Bases:
synapse.lib.hive.Hive
A Hive that acts as a consistent read cache for a telepath proxy Hive
- async get(path)[source]
Get the value of a node at a given path.
- Parameters
full (tuple) – A full path tuple.
- Returns
Arbitrary node value.
synapse.lib.hiveauth module
- class synapse.lib.hiveauth.Auth[source]
Bases:
synapse.lib.nexus.Pusher
Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.
Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).
Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.
Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.
Node layout:
Auth root (passed into constructor) ├ roles │ ├ <role iden 1> │ ├ ... │ └ last role ├ users │ ├ <user iden 1> │ ├ ... │ └ last user └ authgates ├ <iden 1> │ ├ roles │ │ ├ <role iden 1> │ │ ├ ... │ │ └ last role │ └ users │ ├ <user iden 1> │ ├ ... │ └ last user ├ <iden 2> │ ├ ... └ ... last authgate
- async addAuthGate(iden, authgatetype)[source]
Retrieve AuthGate by iden. Create if not present.
Note
Not change distributed
- Returns
(HiveAuthGate)
- async addUser(name, passwd=None, email=None, iden=None)[source]
Add a User to the Hive.
- Parameters
name (str) – The name of the User.
passwd (str) – A optional password for the user.
email (str) – A optional email for the user.
iden (str) – A optional iden to use as the user iden.
- Returns
A Hive User.
- Return type
- class synapse.lib.hiveauth.AuthGate[source]
Bases:
synapse.lib.base.Base
The storage object for object specific rules for users/roles.
- class synapse.lib.hiveauth.HiveRole[source]
Bases:
synapse.lib.hiveauth.HiveRuler
A role within the Hive authorization subsystem.
A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.
- class synapse.lib.hiveauth.HiveRuler[source]
Bases:
synapse.lib.base.Base
A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those
- class synapse.lib.hiveauth.HiveUser[source]
Bases:
synapse.lib.hiveauth.HiveRuler
A user (could be human or computer) of the system within HiveAuth.
Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.
- async setRoles(roleidens)[source]
Replace all the roles for a given user with a new list of roles.
- Parameters
roleidens (list) – A list of roleidens.
Notes
The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.
- Returns
None
synapse.lib.httpapi module
- class synapse.lib.httpapi.ActiveV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthAddRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthAddUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthDelRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthGrantV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
/api/v1/auth/grant?user=iden&role=iden
- class synapse.lib.httpapi.AuthRevokeV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
/api/v1/auth/grant?user=iden&role=iden
- class synapse.lib.httpapi.AuthRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthRolesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthUserPasswdV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.AuthUsersV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.BeholdSockV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
- class synapse.lib.httpapi.CoreInfoV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
/api/v1/core/info
- class synapse.lib.httpapi.FeedV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
/api/v1/feed
Examples
Example data:
{ 'name': 'syn.nodes', 'view': null, 'items': [...], }
- class synapse.lib.httpapi.Handler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.HandlerBase
,tornado.web.RequestHandler
- on_connection_close()[source]
Called in async handlers if the client closed the connection.
Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.
Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.
- prepare()[source]
Called at the beginning of a request before get/post/etc.
Override this method to perform common initialization regardless of the request method.
Asynchronous support: Use
async def
or decorate this method with .gen.coroutine to make it asynchronous. If this method returns anAwaitable
execution will not proceed until theAwaitable
is done.New in version 3.1: Asynchronous support.
- class synapse.lib.httpapi.HandlerBase[source]
Bases:
object
- async allowed(perm, gateiden=None)[source]
Check if the authenticated user has the given permission.
- Parameters
perm (tuple) – The permission tuple to check.
gateiden (str) – The gateiden to check the permission against.
Notes
This API sets up HTTP response values if it returns False.
- Returns
True if the user has the requested permission.
- Return type
bool
- async authenticated()[source]
Check if the request has an authenticated user or not.
- Returns
True if the request has an authenticated user, false otherwise.
- Return type
bool
- async getUserBody()[source]
Helper function to confirm that there is an auth user and a valid JSON body in the request.
Deprecated, use the getUseridenBody() function instead.
- async getUseridenBody(validator=None)[source]
Helper function to confirm that there is an auth user and a valid JSON body in the request.
- Parameters
validator – Validator function run on the deserialized JSON body.
- Returns
The user definition and body of the request as deserialized JSON, or a tuple of s_common.novalu objects if there was no user or json body.
- Return type
(str, object)
- async handleBasicAuth()[source]
Handle basic authentication in the handler.
Notes
Implementors may override this to disable or implement their own basic auth schemes. This is expected to set web_useriden and web_username upon successful authentication.
- Returns
The user iden of the logged in user.
- Return type
str
- async isUserAdmin()[source]
Check if the current authenticated user is an admin or not.
- Returns
True if the user is an admin, false otherwise.
- Return type
bool
- async reqAuthAdmin()[source]
Require the current authenticated user to be an admin.
Notes
If this returns False, an error message has already been sent and no additional processing for the request should be done.
- Returns
True if the user is an admin, false otherwise.
- Return type
bool
- async sess(gen=True)[source]
Get the heavy Session object for the request.
- Parameters
gen (bool) – If set to True, generate a new session if there is no sess cookie.
Notes
This stores the identifier in the
sess
cookie for with a 14 day expiration, stored in the Cell. Valid requests with thatsess
cookie will resolve to the same Session object.- Returns
A heavy session object. If the sess cookie is invalid or gen is false, this returns None.
- Return type
- class synapse.lib.httpapi.HealthCheckV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.LoginV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.ModelNormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.ModelV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.OnePassIssueV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
/api/v1/auth/onepass/issue
- class synapse.lib.httpapi.ReqValidStormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
- class synapse.lib.httpapi.RobotHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.HandlerBase
,tornado.web.RequestHandler
- class synapse.lib.httpapi.Sess[source]
Bases:
synapse.lib.base.Base
- class synapse.lib.httpapi.StormCallV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
- class synapse.lib.httpapi.StormExportV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
- class synapse.lib.httpapi.StormHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
Bases:
synapse.lib.httpapi.Handler
- class synapse.lib.httpapi.StormNodesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)[source]
- class synapse.lib.httpapi.StormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest,