Telepath API
- class synmods.s3.axon.S3AxonApi
Bases:
synapse.axon.AxonApi
- recoverIndices()
Update metadata and local indices from the configured bucket as if they were just uploaded without reuploading the objects themselves.
This is useful if the Axon’s local data was lost.
This may be run on an Axon with partial loss (for example, an older backup). If an object is already in the local indices, it is skipped.
Note
Some information cannot be recovered, namely, the time and order that the objects were added. After this call, future calls to history() and hashes() will return data as if the objects were added as of this call.
Object keys/hashes are not verified against the contents of the objects.
- class synapse.axon.AxonApi
Bases:
synapse.lib.cell.CellApi
,synapse.lib.share.Share
- async csvrows(sha256, dialect='excel', errors='ignore', **fmtparams)
Yield CSV rows from a CSV file.
- Parameters
sha256 (bytes) – The sha256 hash of the file.
dialect (str) – The CSV dialect to use.
errors (str) – Specify how encoding errors should handled.
**fmtparams – The CSV dialect format parameters.
Notes
The dialect and fmtparams expose the Python csv.reader() parameters.
Examples
Get the rows from a CSV file and process them:
async for row in axon.csvrows(sha256): await dostuff(row)
Get the rows from a tab separated file and process them:
async for row in axon.csvrows(sha256, delimiter=' '): await dostuff(row)
- Yields
list – Decoded CSV rows.
- async del_(sha256)
Remove the given bytes from the Axon by sha256.
- Parameters
sha256 (bytes) – The sha256, in bytes, to remove from the Axon.
- Returns
True if the file is removed; false if the file is not present.
- Return type
boolean
- async dels(sha256s)
Given a list of sha256 hashes, delete the files from the Axon.
- Parameters
sha256s (list) – A list of sha256 hashes in bytes form.
- Returns
A list of booleans, indicating if the file was deleted or not.
- Return type
list
- async get(sha256, offs=None, size=None)
Get bytes of a file.
- Parameters
sha256 (bytes) – The sha256 hash of the file in bytes.
offs (int) – The offset to start reading from.
size (int) – The total number of bytes to read.
Examples
Get the bytes from an Axon and process them:
buf = b'' async for bytz in axon.get(sha256): buf =+ bytz await dostuff(buf)
- Yields
bytes – Chunks of the file bytes.
- Raises
synapse.exc.NoSuchFile – If the file does not exist.
- async has(sha256)
Check if the Axon has a file.
- Parameters
sha256 (bytes) – The sha256 hash of the file in bytes.
- Returns
True if the Axon has the file; false otherwise.
- Return type
boolean
- async hashes(offs, wait=False, timeout=None)
Yield hash rows for files that exist in the Axon in added order starting at an offset.
- Parameters
offs (int) – The index offset.
wait (boolean) – Wait for new results and yield them in realtime.
timeout (int) – Max time to wait for new results.
- Yields
(int, (bytes, int)) – An index offset and the file SHA-256 and size.
- async hashset(sha256)
Calculate additional hashes for a file in the Axon.
- Parameters
sha256 (bytes) – The sha256 hash of the file in bytes.
- Returns
A dictionary containing hashes of the file.
- Return type
dict
- async history(tick, tock=None)
Yield hash rows for files that existing in the Axon after a given point in time.
- Parameters
tick (int) – The starting time (in epoch milliseconds).
tock (int) – The ending time to stop iterating at (in epoch milliseconds).
- Yields
(int, (bytes, int)) – A tuple containing time of the hash was added and the file SHA-256 and size.
- async iterMpkFile(sha256)
Yield items from a MsgPack (.mpk) file in the Axon.
- Parameters
sha256 (bytes) – The sha256 hash of the file in bytes.
- Yields
Unpacked items from the bytes.
- async jsonlines(sha256, errors='ignore')
Yield JSON objects from JSONL (JSON lines) file.
- Parameters
sha256 (bytes) – The sha256 hash of the file.
errors (str) – Specify how encoding errors should handled.
- Yields
object – Decoded JSON objects.
- async metrics()
Get the runtime metrics of the Axon.
- Returns
A dictionary of runtime data about the Axon.
- Return type
dict
- async postfiles(fields, url, params=None, headers=None, method='POST', ssl=True, timeout=None, proxy=None)
- async put(byts)
Store bytes in the Axon.
- Parameters
byts (bytes) – The bytes to store in the Axon.
Notes
This API should not be used for files greater than 128 MiB in size.
- Returns
A tuple with the file size and sha256 hash of the bytes.
- Return type
tuple(int, bytes)
- async puts(files)
Store a set of bytes in the Axon.
- Parameters
files (list) – A list of bytes to store in the Axon.
Notes
This API should not be used for storing more than 128 MiB of bytes at once.
- Returns
A list containing tuples of file size and sha256 hash of the saved bytes.
- Return type
list(tuple(int, bytes))
- async readlines(sha256, errors='ignore')
Yield lines from a multi-line text file in the axon.
- Parameters
sha256 (bytes) – The sha256 hash of the file.
errors (str) – Specify how encoding errors should handled.
- Yields
str – Lines of text
- async size(sha256)
Get the size of a file in the Axon.
- Parameters
sha256 (bytes) – The sha256 hash of the file in bytes.
- Returns
The size of the file, in bytes. If not present, None is returned.
- Return type
int
- async upload()
Get an Upload object.
Notes
The UpLoad object should be used to manage uploads greater than 128 MiB in size.
Examples
Use an UpLoad object to upload a file to the Axon:
async with axonProxy.upload() as upfd: # Assumes bytesGenerator yields bytes async for byts in bytsgenerator(): upfd.write(byts) upfd.save()
Use a single UpLoad object to save multiple files:
async with axonProxy.upload() as upfd: for fp in file_paths: # Assumes bytesGenerator yields bytes async for byts in bytsgenerator(fp): upfd.write(byts) upfd.save()
- Returns
An Upload manager object.
- Return type
UpLoadShare
- async wants(sha256s)
Get a list of sha256 values the axon does not have from a input list.
- Parameters
sha256s (list) – A list of sha256 values as bytes.
- Returns
A list of bytes containing the sha256 hashes the Axon does not have.
- Return type
list
- async wget(url, params=None, headers=None, json=None, body=None, method='GET', ssl=True, timeout=None, proxy=None)
Stream a file download directly into the Axon.
- Parameters
url (str) – The URL to retrieve.
params (dict) – Additional parameters to add to the URL.
headers (dict) – Additional HTTP headers to add in the request.
json – A JSON body which is included with the request.
body – The body to be included in the request.
method (str) – The HTTP method to use.
ssl (bool) – Perform SSL verification.
timeout (int) – The timeout of the request, in seconds.
Notes
The response body will be stored, regardless of the response code. The
ok
value in the reponse does not reflect that a status code, such as a 404, was encountered when retrieving the URL.The dictionary returned by this may contain the following values:
{ 'ok': <boolean> - False if there were exceptions retrieving the URL. 'url': <str> - The URL retrieved (which could have been redirected). This is a url-decoded string. 'code': <int> - The response code. 'reason': <str> - The reason phrase for the HTTP status code. 'mesg': <str> - An error message if there was an exception when retrieving the URL. 'err': <tuple> - An error tuple if there was an exception when retrieving the URL. 'headers': <dict> - The response headers as a dictionary. 'size': <int> - The size in bytes of the response body. 'hashes': { 'md5': <str> - The MD5 hash of the response body. 'sha1': <str> - The SHA1 hash of the response body. 'sha256': <str> - The SHA256 hash of the response body. 'sha512': <str> - The SHA512 hash of the response body. }, 'request': { 'url': The request URL. This is a url-decoded string. 'headers': The request headers. 'method': The request method. } 'history': A sequence of response bodies to track any redirects, not including hashes. }
- Returns
An information dictionary containing the results of the request.
- Return type
dict
- async wput(sha256, url, params=None, headers=None, method='PUT', ssl=True, timeout=None, proxy=None)