Admin Guide
Configuration
Permissions
Package (synapse-sidepocket) defines the following permissions:
power-ups.sidepocket.admin : Allows a user to add, modify, view, delete a Synapse-Sidepocket source. ( default: false )
power-ups.sidepocket.user : Allows a user to run commands and access internal APIs, but does not grant access to any sources. ( default: false )
power-ups.sidepocket.sources : Used in addition to power-ups.sidepocket.user to allow a user to query sources. ( default: false )
You may add rules to users/roles directly from storm:
> auth.user.addrule visi power-ups.sidepocket.user
Added rule power-ups.sidepocket.user to user visi.
or:
> auth.role.addrule ninjas power-ups.sidepocket.user
Added rule power-ups.sidepocket.user to role ninjas.
In order to query a source, users with the power-ups.sidepocket.user permission
must also have a power-ups.sidepocket.sources permission.
The sources permission is hierarchial, e.g. to grant access to only the mydb source the
power-ups.sidepocket.sources.mydb permission should be used.
Adding a data source
It is recommended to use the Optic Workflow for managing sources, however sources can also be added using Storm commands.
Each data source must have a unique name and a set of arguments to pass when connecting.
The name must also only contain the following characters: a-z, A-Z, 0-9, _, -
The available connectors and their arguments can be displayed using the sidepocket.connectors Storm command.
Assuming you had a local MySQL database you wanted to use, you could add it using
sidepocket.mysql.add:
sidepocket.mysql.add mydb --host localhost --user exampleuser --password examplepass --db exampledb
The Sidepocket service will attempt to make a connection using the information supplied, and if successful the data source will then be available for use.
If the connector supports connection argument overrides the source can also be added without any persistent values for those arguments.
$sidepocket = $lib.import(sidepocket)
// Add a source that has default auth arguments, but can be overriden by a user
$name = "mydb00"
$type = "mysql"
$args = ({
"host": "localhost",
"db": "exampledb",
"user": "exampleuser",
"password": "examplepass"
})
$sidepocket.addSource($name, $type, $args)
// Override exampleuser
$opts = ({"user": "anotheruser", "password": "secret"})
for $mesg in $sidepocket.query("mydb00", "select * from foo", opts=$opts) {
$lib.print($mesg)
}
// Add a source that has no default auth arguments
// Auth arguments must be provided at runtime
$name = "mydb01"
$type = "mysql"
$args = ({
"host": "localhost",
"db": "exampledb",
})
$opts = ({
"user": "exampleuser",
"password": "examplepass"
})
$sidepocket.addSource($name, $type, $args, opts=$opts)
// This will fail
for $mesg in $sidepocket.query("mydb01", "select * from foo") {
$lib.print($mesg)
}
// Provide runtime auth arguments to execute the query
$opts = ({"user": "anotheruser", "password": "secret"})
for $mesg in $sidepocket.query("mydb01", "select * from foo", opts=$opts) {
$lib.print($mesg)
}
The sidepocket.sources Storm command can be used to list all of the currently configured data sources.
Exported Storm APIs
Synapse Sidepocket exports the sidepocket module as a Storm API.
When a results dictionary is specified as the return type it will contain the following keys:
success (bool): Flag indicating whether the operation was successfulmesg (str or None): Optional error message.
When message tuples are specified as the return type, the following types may be emitted:
init: Optional message from connector (e.g. data contains the query id for the Athena connector)data: The structured response dataprint: Debug message from the connectorwarn: Warning/error messagefini: Optional message from the connector
When a function emits this indicates it is a generator which can yield arbitrary values.
addSource(name, type, args, opts=$lib.null)
-------------------------------------------
Add a data source.
Args:
name (str): Name for the data source.
type (str): Type of data source to add.
args (dict): Arguments to provide when connecting to the data source.
opts (dict): Optional runtime connection arguments to use when testing the connection.
Returns:
dict: Results dictionary.
addSnowflakeOAuth2Provider(name, account, client_id, client_secret, redirect_uri)
-------------------------------------------------
Add a new Snowflake OAuth2 provider to the Cortex.
Args:
name (str): Name of the data source.
account (str): Account to log in to.
client_id (str): The OAuth V2 client id.
client_secret (str): The OAuth V2 client secret.
redirect_uri (str): The local URI to redirect OAuth V2 auth requests.
Returns:
str: The iden of the provider in the Cortex.
addKustoOAuth2Provider(name, cluster_id, region, authority_id, client_id, client_secret, redirect_uri)
-------------------------------------------------
Add a new Kusto OAuth2 provider to the Cortex.
Args:
name (str): Name of the data source.
cluster_id (str): The cluster name.
region (str): The cluster region.
authority_id (str): The tenant id.
client_id (str): The OAuth V2 client id.
client_secret (str): The OAuth V2 client secret.
redirect_uri (str): The local URI to redirect OAuth V2 auth requests.
Returns:
str: The iden of the provider in the Cortex.
delSource(name)
---------------
Remove a data source.
Args:
name (str): Name of the data source.
Returns:
dict: Results dictionary.
renameSource(name, newv)
------------------------
Rename a data source.
Args:
name (str): Name of the data source.
newv (str): New name for the source.
Returns:
dict: Results dictionary.
listSources()
-------------
Get a list of the data sources currently configured.
Note:
The info dictionary contains the following keys:
type
The connection type name.
args
Dict of arg name to the (sanitized) stored value.
extra_args
Dict of additional sanitized args stored for the source.
Emits:
tuple: Source name and info dictionary.
listConnectors()
----------------
Get the data source connector types available.
Note:
The info dictionary contains the following keys:
args
Dict of connection argument names and a description.
query_syntax
String containing error details or empty if successful.
query_execution
One of ("sync", "async") representing the query execution pattern.
emits:
tuple: Connector name and info dict.
query(name, query, opts=$lib.null)
----------------------------------
Query a configured data source and emit message tuples.
Args:
name (str): Name of the data source.
query (str): Query string to submit to the data source.
opts (dict or None): Optional dictionary to provide to the connector.
Emits:
tuple: Tuple of (<message type>, <data>, <info dict>)
queryRows(name, query, opts=$lib.null)
--------------------------------------
Query a configured data source and emit data rows only.
Note:
Print/debug and warning/error message will go the console.
Args:
name (str): Name of the data source.
query (str): Query string to submit to the data source.
opts (dict or None): Optional dictionary to provide to the connector.
Emits:
(tuple or dict): Row data (type varies by source)
status(name, query_id=$lib.null, opts=$lib.null)
------------------------------------------------
Retrieve the status for a query or for all queries.
Note:
This method is not implemented for all connectors.
Args:
name (str): Name of the data source.
query_id (str or None): If specified, retrieve status for only the query id.
opts (dict or None): Optional dictionary to provide to the connector.
Emits:
tuple: Tuple of (<message type>, <data>, <info dict>)
cancel(name, query_id, opts=$lib.null)
--------------------------------------
Cancel a query in the given data source.
Note:
This method is not implemented for all connectors.
Args:
name (str): Name of the data source to cancel the query in.
query_id (str): Query id to cancel.
opts (dict or None): Optional dictionary to provide to the connector.
Returns:
dict: Results dictionary.
Workflows
Synapse Sidepocket provides the following workflows in Optic:
Title: Data explorer
Title: Source management
Node Actions
Synapse Sidepocket does not currently provide any node actions in Optic.