synapse package

The synapse intelligence analysis framework.

Subpackages

Submodules

synapse.axon module

class synapse.axon.Axon[source]

Bases: Cell

byterange = False

cellapi: alias of AxonApi

confdefs = {'http:proxy': {'description': 'An aiohttp-socks compatible proxy URL to use in the wget API.', 'type': 'string'}, 'max:bytes': {'description': 'The maximum number of bytes that can be stored in the Axon.', 'hidecmdl': True, 'minimum': 1, 'type': 'integer'}, 'max:count': {'description': 'The maximum number of files that can be stored in the Axon.', 'hidecmdl': True, 'minimum': 1, 'type': 'integer'}, 'tls:ca:dir': {'description': 'An optional directory of CAs which are added to the TLS CA chain for wget and wput APIs.', 'type': 'string'}}

async csvrows(sha256, dialect='excel', **fmtparams)[source]

async del_(sha256)[source]

Remove the given bytes from the Axon by sha256.

Parameters:: sha256 (bytes) – The sha256, in bytes, to remove from the Axon.
Returns:: True if the file is removed; false if the file is not present.
Return type:: boolean

async dels(sha256s)[source]

Given a list of sha256 hashes, delete the files from the Axon.

Parameters:: sha256s (list) – A list of sha256 hashes in bytes form.
Returns:: A list of booleans, indicating if the file was deleted or not.
Return type:: list

async get(sha256, offs=None, size=None)[source]

Get bytes of a file.

Parameters:

sha256 (bytes) – The sha256 hash of the file in bytes.
offs (int) – The offset to start reading from.
size (int) – The total number of bytes to read.

Examples

Get the bytes from an Axon and process them:

buf = b''
async for bytz in axon.get(sha256):
    buf =+ bytz

await dostuff(buf)

Yields:: bytes – Chunks of the file bytes.
Raises:: synapse.exc.NoSuchFile – If the file does not exist.

async getCellInfo()[source]

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns:: A Dictionary of metadata.
Return type:: Dict

async has(sha256)[source]

Check if the Axon has a file.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: True if the Axon has the file; false otherwise.
Return type:: boolean

async hashes(offs, wait=False, timeout=None)[source]

Yield hash rows for files that exist in the Axon in added order starting at an offset.

Parameters:

offs (int) – The index offset.
wait (boolean) – Wait for new results and yield them in realtime.
timeout (int) – Max time to wait for new results.

Yields:

(int, (bytes, int)) – An index offset and the file SHA-256 and size.

Note

If the same hash was deleted and then added back, the same hash will be yielded twice.

async hashset(sha256)[source]

Calculate additional hashes for a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: A dictionary containing hashes of the file.
Return type:: dict

async history(tick, tock=None)[source]

Yield hash rows for files that existing in the Axon after a given point in time.

Parameters:

tick (int) – The starting time (in epoch milliseconds).
tock (int) – The ending time to stop iterating at (in epoch milliseconds).

Yields:

(int, (bytes, int)) – A tuple containing time of the hash was added and the file SHA-256 and size.

holdHashLock(hashbyts)[source]

A context manager that synchronizes edit access to a blob.

Parameters:: hashbyts (bytes) – The blob to hold the lock for.

async initServiceRuntime()[source]

async initServiceStorage()[source]

async iterMpkFile(sha256)[source]

Yield items from a MsgPack (.mpk) file in the Axon.

Parameters:: sha256 (str) – The sha256 hash of the file as a string.
Yields:: Unpacked items from the bytes.

async jsonlines(sha256)[source]

async metrics()[source]

Get the runtime metrics of the Axon.

Returns:: A dictionary of runtime data about the Axon.
Return type:: dict

async postfiles(fields, url, params=None, headers=None, method='POST', ssl=True, timeout=None, proxy=None)[source]

Send files from the axon as fields in a multipart/form-data HTTP request.

Parameters:

fields (list) – List of dicts containing the fields to add to the request as form-data.
url (str) – The URL to retrieve.
params (dict) – Additional parameters to add to the URL.
headers (dict) – Additional HTTP headers to add in the request.
method (str) – The HTTP method to use.
ssl (bool) – Perform SSL verification.
timeout (int) – The timeout of the request, in seconds.
proxy (bool|str|null) – Use a specific proxy or disable proxy use.

Notes

The dictionaries in the fields list may contain the following values:

{
    'name': <str> - Name of the field.
    'sha256': <str> - SHA256 hash of the file to submit for this field.
    'value': <str> - Value for the field. Ignored if a sha256 has been specified.
    'filename': <str> - Optional filename for the field.
    'content_type': <str> - Optional content type for the field.
    'content_transfer_encoding': <str> - Optional content-transfer-encoding header for the field.
}

The dictionary returned by this may contain the following values:

{
    'ok': <boolean> - False if there were exceptions retrieving the URL.
    'err': <str> - An error message if there was an exception when retrieving the URL.
    'url': <str> - The URL retrieved (which could have been redirected)
    'code': <int> - The response code.
    'body': <bytes> - The response body.
    'headers': <dict> - The response headers as a dictionary.
}

Returns:: An information dictionary containing the results of the request.
Return type:: dict

async put(byts)[source]

Store bytes in the Axon.

Parameters:: byts (bytes) – The bytes to store in the Axon.

Notes

This API should not be used for files greater than 128 MiB in size.

Returns:: A tuple with the file size and sha256 hash of the bytes.
Return type:: tuple(int, bytes)

async puts(files)[source]

Store a set of bytes in the Axon.

Parameters:: files (list) – A list of bytes to store in the Axon.

Notes

This API should not be used for storing more than 128 MiB of bytes at once.

Returns:: A list containing tuples of file size and sha256 hash of the saved bytes.
Return type:: list(tuple(int, bytes))

async readlines(sha256)[source]

async save(sha256, genr, size)[source]

Save a generator of bytes to the Axon.

Parameters:

sha256 (bytes) – The sha256 hash of the file in bytes.
genr – The bytes generator function.

Returns:

The size of the bytes saved.

Return type:

int

async size(sha256)[source]

Get the size of a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: The size of the file, in bytes. If not present, None is returned.
Return type:: int

async upload()[source]

Get an Upload object.

Notes

The UpLoad object should be used to manage uploads greater than 128 MiB in size.

Examples

Use an UpLoad object to upload a file to the Axon:

async with await axon.upload() as upfd:
    # Assumes bytesGenerator yields bytes
    async for byts in bytsgenerator():
        await upfd.write(byts)
    await upfd.save()

Use a single UpLoad object to save multiple files:

async with await axon.upload() as upfd:
    for fp in file_paths:
        # Assumes bytesGenerator yields bytes
        async for byts in bytsgenerator(fp):
            await upfd.write(byts)
        await upfd.save()

Returns:: An Upload manager object.
Return type:: UpLoad

async wants(sha256s)[source]

Get a list of sha256 values the axon does not have from a input list.

Parameters:: sha256s (list) – A list of sha256 values as bytes.
Returns:: A list of bytes containing the sha256 hashes the Axon does not have.
Return type:: list

async wget(url, params=None, headers=None, json=None, body=None, method='GET', ssl=True, timeout=None, proxy=None)[source]

Stream a file download directly into the Axon.

Parameters:

url (str) – The URL to retrieve.
params (dict) – Additional parameters to add to the URL.
headers (dict) – Additional HTTP headers to add in the request.
json – A JSON body which is included with the request.
body – The body to be included in the request.
method (str) – The HTTP method to use.
ssl (bool) – Perform SSL verification.
timeout (int) – The timeout of the request, in seconds.
proxy (bool|str|null) – Use a specific proxy or disable proxy use.

Notes

The response body will be stored, regardless of the response code. The ok value in the reponse does not reflect that a status code, such as a 404, was encountered when retrieving the URL.

The dictionary returned by this may contain the following values:

{
    'ok': <boolean> - False if there were exceptions retrieving the URL.
    'url': <str> - The URL retrieved (which could have been redirected). This is a url-decoded string.
    'code': <int> - The response code.
    'mesg': <str> - An error message if there was an exception when retrieving the URL.
    'headers': <dict> - The response headers as a dictionary.
    'size': <int> - The size in bytes of the response body.
    'hashes': {
        'md5': <str> - The MD5 hash of the response body.
        'sha1': <str> - The SHA1 hash of the response body.
        'sha256': <str> - The SHA256 hash of the response body.
        'sha512': <str> - The SHA512 hash of the response body.
    },
    'request': {
        'url': The request URL. This is a url-decoded string.
        'headers': The request headers.
        'method': The request method.
    }
    'history': A sequence of response bodies to track any redirects, not including hashes.
}

Returns:: An information dictionary containing the results of the request.
Return type:: dict

async wput(sha256, url, params=None, headers=None, method='PUT', ssl=True, timeout=None, filename=None, filemime=None, proxy=None)[source]: Stream a blob from the axon as the body of an HTTP request.

class synapse.axon.AxonApi[source]

Bases: CellApi, Share

async csvrows(sha256, dialect='excel', **fmtparams)[source]

Yield CSV rows from a CSV file.

Parameters:

sha256 (bytes) – The sha256 hash of the file.
dialect (str) – The CSV dialect to use.
**fmtparams – The CSV dialect format parameters.

Notes

The dialect and fmtparams expose the Python csv.reader() parameters.

Examples

Get the rows from a CSV file and process them:

async for row in axon.csvrows(sha256):
    await dostuff(row)

Get the rows from a tab separated file and process them:

async for row in axon.csvrows(sha256, delimiter='       '):
    await dostuff(row)

Yields:: list – Decoded CSV rows.

async del_(sha256)[source]

Remove the given bytes from the Axon by sha256.

Parameters:: sha256 (bytes) – The sha256, in bytes, to remove from the Axon.
Returns:: True if the file is removed; false if the file is not present.
Return type:: boolean

async dels(sha256s)[source]

Given a list of sha256 hashes, delete the files from the Axon.

Parameters:: sha256s (list) – A list of sha256 hashes in bytes form.
Returns:: A list of booleans, indicating if the file was deleted or not.
Return type:: list

async get(sha256, offs=None, size=None)[source]

Get bytes of a file.

Parameters:

sha256 (bytes) – The sha256 hash of the file in bytes.
offs (int) – The offset to start reading from.
size (int) – The total number of bytes to read.

Examples

Get the bytes from an Axon and process them:

buf = b''
async for bytz in axon.get(sha256):
    buf =+ bytz

await dostuff(buf)

Yields:: bytes – Chunks of the file bytes.
Raises:: synapse.exc.NoSuchFile – If the file does not exist.

async has(sha256)[source]

Check if the Axon has a file.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: True if the Axon has the file; false otherwise.
Return type:: boolean

async hashes(offs, wait=False, timeout=None)[source]

Yield hash rows for files that exist in the Axon in added order starting at an offset.

Parameters:

offs (int) – The index offset.
wait (boolean) – Wait for new results and yield them in realtime.
timeout (int) – Max time to wait for new results.

Yields:

(int, (bytes, int)) – An index offset and the file SHA-256 and size.

async hashset(sha256)[source]

Calculate additional hashes for a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: A dictionary containing hashes of the file.
Return type:: dict

async history(tick, tock=None)[source]

Yield hash rows for files that existing in the Axon after a given point in time.

Parameters:

tick (int) – The starting time (in epoch milliseconds).
tock (int) – The ending time to stop iterating at (in epoch milliseconds).

Yields:

(int, (bytes, int)) – A tuple containing time of the hash was added and the file SHA-256 and size.

async iterMpkFile(sha256)[source]

Yield items from a MsgPack (.mpk) file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Yields:: Unpacked items from the bytes.

async jsonlines(sha256)[source]

Yield JSON objects from JSONL (JSON lines) file.

Parameters:: sha256 (bytes) – The sha256 hash of the file.
Yields:: object – Decoded JSON objects.

async metrics()[source]

Get the runtime metrics of the Axon.

Returns:: A dictionary of runtime data about the Axon.
Return type:: dict

async postfiles(fields, url, params=None, headers=None, method='POST', ssl=True, timeout=None, proxy=None)[source]

async put(byts)[source]

Store bytes in the Axon.

Parameters:: byts (bytes) – The bytes to store in the Axon.

Notes

This API should not be used for files greater than 128 MiB in size.

Returns:: A tuple with the file size and sha256 hash of the bytes.
Return type:: tuple(int, bytes)

async puts(files)[source]

Store a set of bytes in the Axon.

Parameters:: files (list) – A list of bytes to store in the Axon.

Notes

This API should not be used for storing more than 128 MiB of bytes at once.

Returns:: A list containing tuples of file size and sha256 hash of the saved bytes.
Return type:: list(tuple(int, bytes))

async readlines(sha256)[source]

Yield lines from a multi-line text file in the axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file.
Yields:: str – Lines of text

async size(sha256)[source]

Get the size of a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: The size of the file, in bytes. If not present, None is returned.
Return type:: int

async upload()[source]

Get an Upload object.

Notes

The UpLoad object should be used to manage uploads greater than 128 MiB in size.

Examples

Use an UpLoad object to upload a file to the Axon:

async with axonProxy.upload() as upfd:
    # Assumes bytesGenerator yields bytes
    async for byts in bytsgenerator():
        upfd.write(byts)
    upfd.save()

Use a single UpLoad object to save multiple files:

async with axonProxy.upload() as upfd:
    for fp in file_paths:
        # Assumes bytesGenerator yields bytes
        async for byts in bytsgenerator(fp):
            upfd.write(byts)
        upfd.save()

Returns:: An Upload manager object.
Return type:: UpLoadShare

async wants(sha256s)[source]

Get a list of sha256 values the axon does not have from a input list.

Parameters:: sha256s (list) – A list of sha256 values as bytes.
Returns:: A list of bytes containing the sha256 hashes the Axon does not have.
Return type:: list

async wget(url, params=None, headers=None, json=None, body=None, method='GET', ssl=True, timeout=None, proxy=None)[source]

Stream a file download directly into the Axon.

Parameters:

url (str) – The URL to retrieve.
params (dict) – Additional parameters to add to the URL.
headers (dict) – Additional HTTP headers to add in the request.
json – A JSON body which is included with the request.
body – The body to be included in the request.
method (str) – The HTTP method to use.
ssl (bool) – Perform SSL verification.
timeout (int) – The timeout of the request, in seconds.

Notes

The response body will be stored, regardless of the response code. The ok value in the reponse does not reflect that a status code, such as a 404, was encountered when retrieving the URL.

The dictionary returned by this may contain the following values:

{
    'ok': <boolean> - False if there were exceptions retrieving the URL.
    'url': <str> - The URL retrieved (which could have been redirected). This is a url-decoded string.
    'code': <int> - The response code.
    'mesg': <str> - An error message if there was an exception when retrieving the URL.
    'headers': <dict> - The response headers as a dictionary.
    'size': <int> - The size in bytes of the response body.
    'hashes': {
        'md5': <str> - The MD5 hash of the response body.
        'sha1': <str> - The SHA1 hash of the response body.
        'sha256': <str> - The SHA256 hash of the response body.
        'sha512': <str> - The SHA512 hash of the response body.
    },
    'request': {
        'url': The request URL. This is a url-decoded string.
        'headers': The request headers.
        'method': The request method.
    }
    'history': A sequence of response bodies to track any redirects, not including hashes.
}

Returns:: An information dictionary containing the results of the request.
Return type:: dict

async wput(sha256, url, params=None, headers=None, method='PUT', ssl=True, timeout=None, proxy=None)[source]

class synapse.axon.AxonFileHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonHandlerMixin, Handler

async getAxonInfo()[source]

class synapse.axon.AxonHandlerMixin[source]

Bases: object

getAxon()[source]: Get a reference to the Axon interface used by the handler.

class synapse.axon.AxonHttpBySha256InvalidV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonFileHandler

async delete(sha256)[source]

async get(sha256)[source]

async head(sha256)[source]

class synapse.axon.AxonHttpBySha256V1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonFileHandler

async delete(sha256)[source]

async get(sha256)[source]

async head(sha256)[source]

class synapse.axon.AxonHttpDelV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonHandlerMixin, Handler

async post()[source]

class synapse.axon.AxonHttpHasV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonHandlerMixin, Handler

async get(sha256)[source]

class synapse.axon.AxonHttpUploadV1(application: Application, request: HTTPServerRequest, **kwargs: Any)[source]

Bases: AxonHandlerMixin, StreamHandler

async data_received(chunk)[source]

Implement this method to handle streamed request data.

Requires the .stream_request_body decorator.

May be a coroutine for flow control.

on_connection_close()[source]

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

on_finish()[source]

Called after the end of a request.

Override this method to perform cleanup, logging, etc. This method is a counterpart to prepare. on_finish may not produce any output, as it is called after the response has been sent to the client.

async post()[source]: Called after all data has been read.

async prepare()[source]

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

async put()[source]

class synapse.axon.UpLoad[source]

Bases: Base

An object used to manage uploads to the Axon.

async save()[source]

Save the currently uploaded bytes to the Axon.

Notes

This resets the Upload object, so it can be reused.

Returns:: A tuple of sizes in bytes and the sha256 hash of the saved files.
Return type:: tuple(int, bytes)

async write(byts)[source]

Write bytes to the Upload object.

Parameters:: byts (bytes) – Bytes to write to the current Upload object.
Returns:: Returns None.
Return type:: (None)

class synapse.axon.UpLoadProxy[source]

Bases: Share

async save()[source]

async write(byts)[source]

class synapse.axon.UpLoadShare[source]

Bases: UpLoad, Share

typename = 'upload'

synapse.cells module

synapse.common module

class synapse.common.NoValu[source]: Bases: object

class synapse.common.aclosing(thing)[source]

Bases: AbstractAsyncContextManager

Async context manager for safely finalizing an asynchronously cleaned-up resource such as an async generator, calling its aclose() method.

Code like this:

async with aclosing(<module>.fetch(<arguments>)) as agen:
    <block>

is equivalent to this:

agen = <module>.fetch(<arguments>)
try:
    <block>
finally:
    await agen.aclose()

async synapse.common.agen(*items)[source]

async synapse.common.aspin(genr)[source]: Async version of spin

synapse.common.buid(valu=None)[source]

A binary GUID like sequence of 32 bytes.

Parameters:

valu (object) – Optional, if provided, the hash of the msgpack
to (encoded form of the object is returned. This can be used) –
buids. (create stable) –

Notes

By default, this returns a random 32 byte value.

Returns:: A 32 byte value.
Return type:: bytes

synapse.common.chunks(item, size)[source]

Divide an iterable into chunks.

Parameters:

item – Item to slice
size (int) – Maximum chunk size.

Notes

This supports Generator objects and objects which support calling the __getitem__() method with a slice object.

Yields:: Slices of the item containing up to “size” number of items.

synapse.common.config(conf, confdefs)[source]: Initialize a config dict using the given confdef tuples.

synapse.common.debase64(b)[source]

synapse.common.deprecated(name, curv='2.x', eolv='3.0.0')[source]

synapse.common.ehex(byts)[source]

Encode a bytes variable to a string using binascii.hexlify.

Parameters:: byts (bytes) – Bytes to encode.
Returns:: A string representing the bytes.
Return type:: str

synapse.common.enbase64(b)[source]

synapse.common.envbool(name, defval='false')[source]

Resolve an environment variable to a boolean value.

Parameters:

name (str) – Environment variable to resolve.
defval (str) – Default string value to resolve as.

Notes

False values will be consider strings “0” or “false” after lower casing.

Returns:: True if the envar is set, false if it is set to a false value.
Return type:: boolean

synapse.common.err(e, fulltb=False)[source]

synapse.common.errinfo(name, mesg)[source]

synapse.common.excinfo(e)[source]: Populate err,errmsg,errtrace info from exc.

synapse.common.firethread(f)[source]: A decorator for making a function fire a thread.

synapse.common.flatten(item)[source]

Normalize a primitive object for cryptographic signing.

Parameters:: item – The python primitive object to normalize.

Notes

Only None, bool, int, bytes, strings, lists, tuples and dictionaries are acceptable input. List objects will be converted to tuples. Dictionary objects must have keys which can be sorted.

Returns:: A new copy of the object.

synapse.common.gendir(*paths, **opts)[source]

Return the absolute path of the joining of the arguments, creating a directory at the resulting path if one does not exist.

Performs home directory(~) and environment variable expansion.

Parameters:

*paths ([str,...]) – A list of path elements
**opts – arguments as kwargs to os.makedirs

synapse.common.genfile(*paths)[source]

Create or open (for read/write) a file path join.

Parameters:: *paths – A list of paths to join together to make the file.

Notes

If the file already exists, the fd returned is opened in r+b mode. Otherwise, the fd is opened in w+b mode.

The file position is set to the start of the file. The user is responsible for truncating (fd.truncate()) if the existing file contents are not desired, or seeking to the end (fd.seek(0, 2)) to append.

Returns:: A file-object which can be read/written too.
Return type:: io.BufferedRandom

synapse.common.genpath(*paths)[source]

Return an absolute path of the joining of the arguments as path elements

Performs home directory(~) and environment variable expansion on the joined path

Parameters:: *paths ([str,...]) – A list of path elements

Note

All paths used by Synapse operations (i.e. everything but the data) shall use this function or one of its callers before storing as object properties.

synapse.common.getDirSize(*paths)[source]

Get the size of a directory.

Parameters:: *paths (str) – A list of path elements.

Notes

This is equivalent to du -B 1 -s and du -bs.

Returns:: Tuple of total real and total apparent size of all normal files and directories underneath *paths plus *paths itself.
Return type:: tuple

synapse.common.getSslCtx(cadir, purpose=Purpose.SERVER_AUTH)[source]

Create as SSL Context and load certificates from a given directory.

Parameters:

cadir (str) – Path to load certificates from.
purpose – SSLContext purposes flags.

Returns:

A SSL Context object.

Return type:

ssl.SSLContext

synapse.common.getSynDir(*paths)[source]

synapse.common.getSynPath(*paths)[source]

synapse.common.getTempDir(dirn=None)[source]

synapse.common.getbytes(*paths, **opts)[source]

synapse.common.getfile(*paths, **opts)[source]

Return a file at the path resulting from joining of the arguments, or None if the file does not exist.

Parameters:

*paths ([str,...]) – A list of path elements
**opts – arguments as kwargs to io.open

Returns:

A file-object which can be read/written too.

Return type:

io.BufferedRandom

synapse.common.guid(valu=None)[source]

Get a 16 byte guid value.

By default, this is a random guid value.

Parameters:: valu – Object used to construct the guid valu from. This must be able to be msgpack’d.
Returns:: 32 character, lowercase ascii string.
Return type:: str

synapse.common.hugeadd(x, y)[source]: Add two decimal.Decimal with proper precision to support synapse hugenums.

synapse.common.hugediv(x, y)[source]: Divide two decimal.Decimal with proper precision to support synapse hugenums.

synapse.common.hugemod(x, y)[source]

synapse.common.hugemul(x, y)[source]: Multiply two decimal.Decimal with proper precision to support synapse hugenums.

synapse.common.hugenum(valu)[source]: Return a decimal.Decimal with proper precision for use as a synapse hugenum.

synapse.common.hugepow(x, y)[source]: Return the first operand to the power of the second operand.

synapse.common.hugeround(x)[source]: Round a decimal.Decimal with proper precision for synapse hugenums.

synapse.common.hugescaleb(x, y)[source]: Return the first operand with its exponent adjusted by the second operand.

synapse.common.hugesub(x, y)[source]: Subtract two decimal.Decimal with proper precision to support synapse hugenums.

synapse.common.int64en(i)[source]: Encode an unsigned 64-bit int into 8 byte big-endian bytes

synapse.common.int64un(b)[source]: Decode an unsigned 64-bit int from 8 byte big-endian

synapse.common.intify(x)[source]

Ensure ( or coerce ) a value into being an integer or None.

Parameters:: x (obj) – An object to intify
Returns:: The int value ( or None )
Return type:: (int)

synapse.common.isbuidhex(text)[source]

synapse.common.isguid(text)[source]

synapse.common.iterfd(fd, size=10000000)[source]

Generator which yields bytes from a file descriptor.

Parameters:

fd (file) – A file-like object to read bytes from.
size (int) – Size, in bytes, of the number of bytes to read from the
time. (fd at a given) –

Notes

If the first read call on the file descriptor is a empty bytestring, that zero length bytestring will be yielded and the generator will then be exhuasted. This behavior is intended to allow the yielding of contents of a zero byte file.

Yields:: bytes – Bytes from the file descriptor.

synapse.common.iterzip(*args, fillvalue=None)[source]

synapse.common.jslines(*paths)[source]

synapse.common.jsload(*paths)[source]

synapse.common.jsonsafe_nodeedits(nodeedits)[source]: Hexlify the buid of each node:edits

synapse.common.jssave(js, *paths)[source]

synapse.common.listdir(*paths, glob=None)[source]

List the (optionally glob filtered) full paths from a dir.

Parameters:

*paths ([str,...]) – A list of path elements
glob (str) – An optional fnmatch glob str

synapse.common.lockfile(path)[source]

A file lock with-block helper.

Parameters:: path (str) – A path to a lock file.

Examples

Get the lock on a file and dostuff while having the lock:

path = '/hehe/haha.lock'
with lockfile(path):
    dostuff()

Notes

This is curently based on fcntl.lockf(), and as such, it is purely advisory locking. If multiple processes are attempting to obtain a lock on the same file, this will block until the process which has the current lock releases it.

Yields:: None

synapse.common.makedirs(path, mode=511)[source]

async synapse.common.merggenr(genrs, cmprkey)[source]

Iterate multiple sorted async generators and yield their results in order.

Parameters:

genrs (Sequence[AsyncGenerator[T]]) – a sequence of async generator that each yield sorted items
cmprkey (Callable[T, T, bool]) – a comparison function over the items yielded

Note

If the genrs yield increasing items, cmprkey should return True if the first parameter is less than the second parameter, e.g lambda x, y: x < y.

async synapse.common.merggenr2(genrs, cmprkey=None, reverse=False)[source]: Optimized version of merggenr based on heapq.merge

synapse.common.normLogLevel(valu)[source]

Norm a log level value to a integer.

Parameters:: valu – The value to norm ( a string or integer ).
Returns:: A valid Logging log level.
Return type:: int

synapse.common.now()[source]

Get the current epoch time in milliseconds.

This relies on time.time(), which is system-dependent in terms of resolution.

Examples

Get the current time and make a row for a Cortex:

tick = now()
row = (someiden, 'foo:prop', 1, tick)
core.addRows([row])

Returns:: Epoch time in milliseconds.
Return type:: int

synapse.common.reqJsonSafeStrict(item)[source]

Require the item to be safe to serialize to JSON without type coercion issues.

Parameters:: item – The python primitive to check.
Returns:: None
Raises:: s_exc.BadArg – If the item contains invalid data.

synapse.common.reqbytes(*paths)[source]

synapse.common.reqdir(*paths)[source]

Return the absolute path of the joining of the arguments, raising an exception if a directory does not exist at the resulting path.

Performs home directory(~) and environment variable expansion.

Parameters:: *paths ([str,...]) – A list of path elements

synapse.common.reqfile(*paths, **opts)[source]

Return a file at the path resulting from joining of the arguments, raising an exception if the file does not exist.

Parameters:

*paths ([str,...]) – A list of path elements
**opts – arguments as kwargs to io.open

Returns:

A file-object which can be read/written too.

Return type:

io.BufferedRandom

synapse.common.reqjsonsafe(item)[source]: Returns None if item is json serializable, otherwise raises an exception. Uses default type coercion from built-in json.dumps.

synapse.common.reqpath(*paths)[source]

Return the absolute path of the joining of the arguments, raising an exception if a file doesn’t exist at resulting path

Parameters:: *paths ([str,...]) – A list of path elements

synapse.common.result(retn)[source]: Return a value or raise an exception from a retn tuple.

synapse.common.retnexc(e)[source]: Construct a retn tuple for the given exception.

synapse.common.setlogging(mlogger, defval=None, structlog=None, log_setup=True)[source]

Configure synapse logging.

Parameters:

mlogger (logging.Logger) – Reference to a logging.Logger()
defval (str) – Default log level. May be an integer.
structlog (bool) – Enabled structured (jsonl) logging output.

Notes

This calls logging.basicConfig and should only be called once per process.

Returns:: None

synapse.common.signedint64en(i)[source]: Encode a signed 64-bit int into 8 byte big-endian bytes

synapse.common.signedint64un(b)[source]: Decode a signed 64-bit int from 8 byte big-endian

synapse.common.spin(genr)[source]

Crank through a generator but discard the yielded values.

Parameters:: genr – Any generator or iterable valu.

Notes

This generator is exhausted via the collections.dequeue() constructor with a maxlen=0, which will quickly exhaust an iterator staying in C code as much as possible.

Returns:: None

synapse.common.switchext(