Telepath API

class synmods.s3.axon.S3AxonApi

Bases: synapse.axon.AxonApi

recoverIndices()

Update metadata and local indices from the configured bucket as if they were just uploaded without reuploading the objects themselves.

This is useful if the Axon’s local data was lost.

This may be run on an Axon with partial loss (for example, an older backup). If an object is already in the local indices, it is skipped.

Note

Some information cannot be recovered, namely, the time and order that the objects were added. After this call, future calls to history() and hashes() will return data as if the objects were added as of this call.

Object keys/hashes are not verified against the contents of the objects.

class synapse.axon.AxonApi

Bases: synapse.lib.cell.CellApi, synapse.lib.share.Share

async csvrows(sha256, dialect='excel', **fmtparams)

Yield CSV rows from a CSV file.

Parameters
  • sha256 (bytes) – The sha256 hash of the file.

  • dialect (str) – The CSV dialect to use.

  • **fmtparams – The CSV dialect format parameters.

Notes

The dialect and fmtparams expose the Python csv.reader() parameters.

Examples

Get the rows from a CSV file and process them:

async for row in axon.csvrows(sha256):
    await dostuff(row)

Get the rows from a tab separated file and process them:

async for row in axon.csvrows(sha256, delimiter='       '):
    await dostuff(row)
Yields

list – Decoded CSV rows.

async del_(sha256)

Remove the given bytes from the Axon by sha256.

Parameters

sha256 (bytes) – The sha256, in bytes, to remove from the Axon.

Returns

True if the file is removed; false if the file is not present.

Return type

boolean

async dels(sha256s)

Given a list of sha256 hashes, delete the files from the Axon.

Parameters

sha256s (list) – A list of sha256 hashes in bytes form.

Returns

A list of booleans, indicating if the file was deleted or not.

Return type

list

async get(sha256, offs=None, size=None)

Get bytes of a file.

Parameters
  • sha256 (bytes) – The sha256 hash of the file in bytes.

  • offs (int) – The offset to start reading from.

  • size (int) – The total number of bytes to read.

Examples

Get the bytes from an Axon and process them:

buf = b''
async for bytz in axon.get(sha256):
    buf =+ bytz

await dostuff(buf)
Yields

bytes – Chunks of the file bytes.

Raises

synapse.exc.NoSuchFile – If the file does not exist.

async has(sha256)

Check if the Axon has a file.

Parameters

sha256 (bytes) – The sha256 hash of the file in bytes.

Returns

True if the Axon has the file; false otherwise.

Return type

boolean

async hashes(offs, wait=False, timeout=None)

Yield hash rows for files that exist in the Axon in added order starting at an offset.

Parameters
  • offs (int) – The index offset.

  • wait (boolean) – Wait for new results and yield them in realtime.

  • timeout (int) – Max time to wait for new results.

Yields

(int, (bytes, int)) – An index offset and the file SHA-256 and size.

async hashset(sha256)

Calculate additional hashes for a file in the Axon.

Parameters

sha256 (bytes) – The sha256 hash of the file in bytes.

Returns

A dictionary containing hashes of the file.

Return type

dict

async history(tick, tock=None)

Yield hash rows for files that existing in the Axon after a given point in time.

Parameters
  • tick (int) – The starting time (in epoch milliseconds).

  • tock (int) – The ending time to stop iterating at (in epoch milliseconds).

Yields

(int, (bytes, int)) – A tuple containing time of the hash was added and the file SHA-256 and size.

async iterMpkFile(sha256)

Yield items from a MsgPack (.mpk) file in the Axon.

Parameters

sha256 (bytes) – The sha256 hash of the file in bytes.

Yields

Unpacked items from the bytes.

async jsonlines(sha256)

Yield JSON objects from JSONL (JSON lines) file.

Parameters

sha256 (bytes) – The sha256 hash of the file.

Yields

object – Decoded JSON objects.

async metrics()

Get the runtime metrics of the Axon.

Returns

A dictionary of runtime data about the Axon.

Return type

dict

async postfiles(fields, url, params=None, headers=None, method='POST', ssl=True, timeout=None, proxy=None)
async put(byts)

Store bytes in the Axon.

Parameters

byts (bytes) – The bytes to store in the Axon.

Notes

This API should not be used for files greater than 128 MiB in size.

Returns

A tuple with the file size and sha256 hash of the bytes.

Return type

tuple(int, bytes)

async puts(files)

Store a set of bytes in the Axon.

Parameters

files (list) – A list of bytes to store in the Axon.

Notes

This API should not be used for storing more than 128 MiB of bytes at once.

Returns

A list containing tuples of file size and sha256 hash of the saved bytes.

Return type

list(tuple(int, bytes))

async readlines(sha256)

Yield lines from a multi-line text file in the axon.

Parameters

sha256 (bytes) – The sha256 hash of the file.

Yields

str – Lines of text

async size(sha256)

Get the size of a file in the Axon.

Parameters

sha256 (bytes) – The sha256 hash of the file in bytes.

Returns

The size of the file, in bytes. If not present, None is returned.

Return type

int

async upload()

Get an Upload object.

Notes

The UpLoad object should be used to manage uploads greater than 128 MiB in size.

Examples

Use an UpLoad object to upload a file to the Axon:

async with axonProxy.upload() as upfd:
    # Assumes bytesGenerator yields bytes
    async for byts in bytsgenerator():
        upfd.write(byts)
    upfd.save()

Use a single UpLoad object to save multiple files:

async with axonProxy.upload() as upfd:
    for fp in file_paths:
        # Assumes bytesGenerator yields bytes
        async for byts in bytsgenerator(fp):
            upfd.write(byts)
        upfd.save()
Returns

An Upload manager object.

Return type

UpLoadShare

async wants(sha256s)

Get a list of sha256 values the axon does not have from a input list.

Parameters

sha256s (list) – A list of sha256 values as bytes.

Returns

A list of bytes containing the sha256 hashes the Axon does not have.

Return type

list

async wget(url, params=None, headers=None, json=None, body=None, method='GET', ssl=True, timeout=None, proxy=None)

Stream a file download directly into the Axon.

Parameters
  • url (str) – The URL to retrieve.

  • params (dict) – Additional parameters to add to the URL.

  • headers (dict) – Additional HTTP headers to add in the request.

  • json – A JSON body which is included with the request.

  • body – The body to be included in the request.

  • method (str) – The HTTP method to use.

  • ssl (bool) – Perform SSL verification.

  • timeout (int) – The timeout of the request, in seconds.

Notes

The response body will be stored, regardless of the response code. The ok value in the reponse does not reflect that a status code, such as a 404, was encountered when retrieving the URL.

The dictionary returned by this may contain the following values:

{
    'ok': <boolean> - False if there were exceptions retrieving the URL.
    'url': <str> - The URL retrieved (which could have been redirected)
    'code': <int> - The response code.
    'mesg': <str> - An error message if there was an exception when retrieving the URL.
    'headers': <dict> - The response headers as a dictionary.
    'size': <int> - The size in bytes of the response body.
    'hashes': {
        'md5': <str> - The MD5 hash of the response body.
        'sha1': <str> - The SHA1 hash of the response body.
        'sha256': <str> - The SHA256 hash of the response body.
        'sha512': <str> - The SHA512 hash of the response body.
    }
}
Returns

An information dictionary containing the results of the request.

Return type

dict

async wput(sha256, url, params=None, headers=None, method='PUT', ssl=True, timeout=None, proxy=None)