Large-scale file ingest / upload is best performed using an automated feed / module / API. However,
pushfile can be useful for uploading one-off files.
pushfile is executed from an operating system command shell. The command usage is as follows:
usage: synapse.tools.pushfile [-h] -a AXON [-c CORTEX] [-r] [-t TAGS] filenames [filenames ...]
AXONis the telepath URL to a storage Axon.
CORTEXis the optional path to a Cortex where a corresponding
file:bytesnode should be created.
Note: while this is an optional parameter, it doesn’t make much sense to store a file in an Axon that can’t be referenced from within a Cortex.
TAGSis an optional list of tags to be applied to the
file:bytesnode created in the Cortex.
-ttakes a comma separated list of tags.
The tag should be specified by name only (i.e., without the
-rrecursively finds all files when a glob pattern is used for a file name.
filenamesis one or more names (with optional paths), or glob patterns, to the local file(s) to be uploaded.
If multiple file names are specified, any tag provided with the
-toption will be added to each uploaded file.
Upload the file
myreport.pdf to the specified Axon, create a
file:bytes node in the specified Cortex, and tag the
file:bytes node with the tag
#sometag (replace the Axon and Cortex path below with the path to your Cortex. Note that the command is wrapped for readability):
python -m synapse.tools.pushfile -a tcp://axon.vertex.link:5555/axon00 -c tcp://cortex.vertex.link:4444/cortex00 -t sometag /home/user/reports/myreport.pdf
Executing the command will result in various status messages (lines are wrapped for readability):
2019-07-03 11:46:30,567 [INFO] log level set to DEBUG [common.py:setlogging:MainThread:MainProcess] 2019-07-03 11:46:30,568 [DEBUG] Using selector: EpollSelector [selector_events.py:__init__:MainThread:MainProcess] adding tags: ['sometag'] Uploaded [myreport.pdf] to axon file: myreport.pdf (2606351) added to core (sha256:229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0) as myreport.pdf
adding tags: ['sometag']indicates the tag
#sometagwas applied to the
Uploaded [myreport.pdf] to axonindicates the file was successfully uploaded to the storage Axon.
file: myreport.pdf (2606351) added to core (sha256:229cdde4...5cd40ab0) as myreport.pdfindicates the
file:bytesnode was created in the Cortex.
The message gives the new node’s primary property value (
sha256:229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0) and also notes the
:namesecondary property value assigned to the node (
file:bytes:nameproperty to the base name of the local file being uploaded.
If a given file already exists in the Axon (deconflicted based on the file’s SHA256 hash),
pushfile will not re-upload the file. However, the command will still process any other options, including:
file:bytesnode in the Cortex if it does not already exist.
applying any specified tag.
setting (or overwriting) the
:nameproperty on any existing
file:bytesnode with the base name of the local file specified.
For example (lines wrapped for readability):
python -m synapse.tools.pushfile -a tcp://axon.vertex.link:5555/axon00 -c tcp://cortex.vertex.link:4444/cortex00 -t anothertag,athirdtag /home/user/reports/anotherreport.pdf 2019-07-03 11:59:03,366 [INFO] log level set to DEBUG [common.py:setlogging:MainThread:MainProcess] 2019-07-03 11:59:03,367 [DEBUG] Using selector: EpollSelector [selector_events.py:__init__:MainThread:MainProcess] adding tags: ['anothertag'. 'athirdtag'] Axon already had [anotherreport.pdf] file: anotherreport.pdf (2606351) added to core (sha256:229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0) as anotherreport.pdf
Note the status indicating the Axon already had the specified file. Similarly, the status noting the
file:bytes node was added to the Cortex lists the same SHA256 hash as our first upload (i.e.,
anotherreport.pdf has the same SHA256 hash as
myreport.pdf) and indicates the
:name property has been updated (as
file:bytes node for the uploaded report can now be viewed in the specified Cortex by lifting (see Storm Reference - Lifting) the file using the SHA256 / primary property value from the
pushfile status output:
file:bytes=sha256:229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0 file:bytes=sha256:229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0 .created = 2019/07/03 18:46:40.542 :md5 = 23a14d3a4508628e7e09a4c4868dfb17 :mime = ?? :name = anotherrepport.pdf :sha1 = 99b6b984988581cae681f65b92198ed77609bd11 :sha256 = 229cdde419ba9549023de39c6a0ca8af74b45fade2d7a22cdc4105a75cd40ab0 :size = 2606351 #anothertag #athirdtag #sometag complete. 1 nodes in 3 ms (333/sec).
Viewing the node’s properties, we see that Synapse has set the
:name property and has calculated and set the MD5, SHA1, and SHA256 hash secondary property values, as well as the file’s size in bytes. Similarly the two tags from our two example
pushfile commands have been added to the node.
Alternatively, a glob pattern could be used to upload all PDF files in a given directory:
python -m synapse.tools.pushfile -a tcp://axon.vertex.link:5555/axon00 -c tcp://cortex.vertex.link:4444/cortex00 -t anothertag,athirdtag /home/user/reports/*.pdf