Telepath API

class synmods.s3.axon.S3AxonApi

Bases: AxonApi

recoverIndices()

Update metadata and local indices from the configured bucket as if they were just uploaded without reuploading the objects themselves.

This is useful if the Axon’s local data was lost.

This may be run on an Axon with partial loss (for example, an older backup). If an object is already in the local indices, it is skipped.

Note

Some information cannot be recovered, namely, the time and order that the objects were added. After this call, future calls to history() and hashes() will return data as if the objects were added as of this call.

Object keys/hashes are not verified against the contents of the objects.

class synapse.axon.AxonApi

Bases: CellApi, Share

async csvrows(sha256, dialect='excel', errors='ignore', **fmtparams)

Yield CSV rows from a CSV file.

Parameters:

sha256 (bytes) – The sha256 hash of the file.
dialect (str) – The CSV dialect to use.
errors (str) – Specify how encoding errors should handled.
**fmtparams – The CSV dialect format parameters.

Notes

The dialect and fmtparams expose the Python csv.reader() parameters.

Examples

Get the rows from a CSV file and process them:

async for row in axon.csvrows(sha256):
    await dostuff(row)

Get the rows from a tab separated file and process them:

async for row in axon.csvrows(sha256, delimiter='       '):
    await dostuff(row)

Yields:: list – Decoded CSV rows.

async del_(sha256)

Remove the given bytes from the Axon by sha256.

Parameters:: sha256 (bytes) – The sha256, in bytes, to remove from the Axon.
Returns:: True if the file is removed; false if the file is not present.
Return type:: boolean

async dels(sha256s)

Given a list of sha256 hashes, delete the files from the Axon.

Parameters:: sha256s (list) – A list of sha256 hashes in bytes form.
Returns:: A list of booleans, indicating if the file was deleted or not.
Return type:: list

async get(sha256, offs=None, size=None)

Get bytes of a file.

Parameters:

sha256 (bytes) – The sha256 hash of the file in bytes.
offs (int) – The offset to start reading from.
size (int) – The total number of bytes to read.

Examples

Get the bytes from an Axon and process them:

buf = b''
async for bytz in axon.get(sha256):
    buf += bytz

await dostuff(buf)

Yields:: bytes – Chunks of the file bytes.
Raises:: synapse.exc.NoSuchFile – If the file does not exist.

async has(sha256)

Check if the Axon has a file.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: True if the Axon has the file; false otherwise.
Return type:: boolean

async hashes(offs, wait=False, timeout=None)

Yield hash rows for files that exist in the Axon in added order starting at an offset.

Parameters:

offs (int) – The index offset.
wait (boolean) – Wait for new results and yield them in realtime.
timeout (int) – Max time to wait for new results.

Yields:

(int, (bytes, int)) – An index offset and the file SHA-256 and size.

async hashset(sha256)

Calculate additional hashes for a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: A dictionary containing hashes of the file.
Return type:: dict

async history(tick, tock=None)

Yield hash rows for files that existing in the Axon after a given point in time.

Parameters:

tick (int) – The starting time (in epoch milliseconds).
tock (int) – The ending time to stop iterating at (in epoch milliseconds).

Yields:

(int, (bytes, int)) – A tuple containing time of the hash was added and the file SHA-256 and size.

async iterMpkFile(sha256)

Yield items from a MsgPack (.mpk) file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Yields:: Unpacked items from the bytes.

async jsonlines(sha256, errors='ignore')

Yield JSON objects from JSONL (JSON lines) file.

Parameters:

sha256 (bytes) – The sha256 hash of the file.
errors (str) – Specify how encoding errors should handled.

Yields:

object – Decoded JSON objects.

async metrics()

Get the runtime metrics of the Axon.

Returns:: A dictionary of runtime data about the Axon.
Return type:: dict

async postfiles(fields, url, params=None, headers=None, method='POST', ssl=True, timeout=None, proxy=True, ssl_opts=None)

async put(byts)

Store bytes in the Axon.

Parameters:: byts (bytes) – The bytes to store in the Axon.

Notes

This API should not be used for files greater than 128 MiB in size.

Returns:: A tuple with the file size and sha256 hash of the bytes.
Return type:: tuple(int, bytes)

async puts(files)

Store a set of bytes in the Axon.

Parameters:: files (list) – A list of bytes to store in the Axon.

Notes

This API should not be used for storing more than 128 MiB of bytes at once.

Returns:: A list containing tuples of file size and sha256 hash of the saved bytes.
Return type:: list(tuple(int, bytes))

async readlines(sha256, errors='ignore')

Yield lines from a multi-line text file in the axon.

Parameters:

sha256 (bytes) – The sha256 hash of the file.
errors (str) – Specify how encoding errors should handled.

Yields:

str – Lines of text

async size(sha256)

Get the size of a file in the Axon.

Parameters:: sha256 (bytes) – The sha256 hash of the file in bytes.
Returns:: The size of the file, in bytes. If not present, None is returned.
Return type:: int

async upload()

Get an Upload object.

Notes

The UpLoad object should be used to manage uploads greater than 128 MiB in size.

Examples

Use an UpLoad object to upload a file to the Axon:

async with axonProxy.upload() as upfd:
    # Assumes bytesGenerator yields bytes
    async for byts in bytsgenerator():
        upfd.write(byts)
    upfd.save()

Use a single UpLoad object to save multiple files:

async with axonProxy.upload() as upfd:
    for fp in file_paths:
        # Assumes bytesGenerator yields bytes
        async for byts in bytsgenerator(fp):
            upfd.write(byts)
        upfd.save()

Returns:: An Upload manager object.
Return type:: UpLoadShare

async wants(sha256s)

Get a list of sha256 values the axon does not have from an input list.

Parameters:: sha256s (list) – A list of sha256 values as bytes.
Returns:: A list of bytes containing the sha256 hashes the Axon does not have.
Return type:: list

async wget(url, params=None, headers=None, json=None, body=None, method='GET', ssl=True, timeout=None, proxy=True, ssl_opts=None)

Stream a file download directly into the Axon.

Parameters:

url (str) – The URL to retrieve.
params (dict) – Additional parameters to add to the URL.
headers (dict) – Additional HTTP headers to add in the request.
json – A JSON body which is included with the request.
body – The body to be included in the request.
method (str) – The HTTP method to use.
ssl (bool) – Perform SSL verification.
timeout (int) – The timeout of the request, in seconds.
proxy (str|bool) – The proxy value.
ssl_opts (dict) – Additional SSL/TLS options.

Notes

The response body will be stored, regardless of the response code. The ok value in the response does not reflect that a status code, such as a 404, was encountered when retrieving the URL.

The ssl_opts dictionary may contain the following values:

{
    'verify': <bool> - Perform SSL/TLS verification. Is overridden by the ssl argument.
    'client_cert': <str> - PEM encoded full chain certificate for use in mTLS.
    'client_key': <str> - PEM encoded key for use in mTLS. Alternatively, can be included in client_cert.
}

The following proxy arguments are supported:

None: Deprecated - Use the proxy defined by the http:proxy configuration option if set.
True: Use the proxy defined by the http:proxy configuration option if set.
False: Do not use the proxy defined by the http:proxy configuration option if set.
<str>: A proxy URL string.

The dictionary returned by this may contain the following values:

{
    'ok': <boolean> - False if there were exceptions retrieving the URL.
    'url': <str> - The URL retrieved (which could have been redirected). This is a url-decoded string.
    'code': <int> - The response code.
    'reason': <str> - The reason phrase for the HTTP status code.
    'mesg': <str> - An error message if there was an exception when retrieving the URL.
    'err': <tuple> - An error tuple if there was an exception when retrieving the URL.
    'headers': <dict> - The response headers as a dictionary.
    'size': <int> - The size in bytes of the response body.
    'hashes': {
        'md5': <str> - The MD5 hash of the response body.
        'sha1': <str> - The SHA1 hash of the response body.
        'sha256': <str> - The SHA256 hash of the response body.
        'sha512': <str> - The SHA512 hash of the response body.
    },
    'request': {
        'url': The request URL. This is a url-decoded string.
        'headers': The request headers.
        'method': The request method.
    }
    'history': A sequence of response bodies to track any redirects, not including hashes.
}

Returns:: An information dictionary containing the results of the request.
Return type:: dict

async wput(sha256, url, params=None, headers=None, method='PUT', ssl=True, timeout=None, proxy=True, ssl_opts=None)