Storm Package: synapse-sidepocket
The following Commands are available from this package. This documentation is generated for version 3.27.0 of the package.
Storm Commands
This package implements the following Storm Commands.
sidepocket.add
Add a side pocket data source.
This command configures a side pocket data source to be queried. Requires a
unique name for the data source, a type that matches an available connector
and a dictionary of arguments to pass when connecting to the data source.
A list of the available connector types can be displayed using the
sidepocket.connectors command.
Examples:
// Add a MySQL connection named 'foodata'.
$args = ({
"host": "localhost",
"user": "testuser",
"password": "testpass",
"db": "cooldb"
})
sidepocket.add foodata mysql $args
Usage: sidepocket.add [options] <name> <type> <args>
Options:
--help : Display the command usage.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<type> : Type of data source.
<args> : Args to pass when connecting to the data source.
sidepocket.athena.add
Add an Athena data source.
This command configures an Athena data source for use in sidepocket queries.
Querying Athena:
By default the Athena connector will start the query and poll until it is finished and yield results.
The following query options can be provided to modify behavior:
- timeout: In seconds, the amount of time to wait on each Athena operation (defaults to 30).
- background: If set, start the query, yield the query_id, but don't wait for results.
- debug: When using sidepocket.query this is set by --debug, but it can also be provided directly.
- If waiting for results, the arguments to sidepocket.results can also be provided.
Output is a tuple of (type, data, info), for more details see help in sidepocket.results.
Examples:
// Add an Athena connection named 'foodata', configured with a workgroup
sidepocket.athena.add foodata --workgroup my_workgroup
// Add an Athena connection named 'bardata', configured with a database and catalog
sidepocket.athena.add bardata --database my_db --catalog my_catalog --output s3://my_results
// Add an Athena connection named 'bazdata', using a boto3 configuration
$boto3 = ({"aws_access_key_id": "<id>", "aws_secret_access_key": "<secret>", "region_name": "us-west-2"})
sidepocket.athena.add bazdata --database my_db --catalog my_catalog --output s3://my_results --boto3 $boto3
// Start a query and then parse the results when it completes
$sidepocket = $lib.import(sidepocket)
$queryopts = ({"max_items": 100})
for ($type, $data, $info) in $sidepocket.query(foodata, $query, opts=$queryopts) {
switch $type {
init: { $lib.print("query_id={q}", q=$data.query_id) }
print: { $lib.print($data) }
warn: {
$lib.warn($data)
if $info.excinfo { $lib.warn($info.excinfo) }
}
data: {
($col1, $col2) = $data
}
}
}
// Start a query but do not wait for results
$sidepocket = $lib.import(sidepocket)
$queryopts = ({"max_items": 100, "background": true})
for ($type, $data, $info) in $sidepocket.query(foodata, $query, opts=$queryopts) {
switch $type {
init: { $lib.print("query_id={q}", q=$data.query_id) }
print: { $lib.print($data) }
warn: {
$lib.warn($data)
if $info.excinfo { $lib.warn($info.excinfo) }
}
}
}
Usage: sidepocket.athena.add [options] <name>
Options:
--help : Display the command usage.
--database <database> : The database name to use for Athena queries.
--catalog <catalog> : The catalog name to use for Athena queries.
--workgroup <workgroup> : The workgroup to use for Athena queries.
--output <output> : The S3 output location for Athena to store results.
--boto3 <boto3> : A dict of boto3 configuration key-vals.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.cancel
Cancel a query.
Examples:
sidepocket.cancel foodata "1b7bd133-c766-4bea-b481-8dd5cbb6de00"
Usage: sidepocket.cancel [options] <name> <query-id>
Options:
--help : Display the command usage.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<query_id> : The query_id to cancel
sidepocket.connectors
List the data source connector types available.
Usage: sidepocket.connectors [options]
Options:
--help : Display the command usage.
sidepocket.del
Remove a side pocket data source.
This command deletes a configured side pocket data source.
Examples:
// Delete a connection named 'foodata'.
sidepocket.del foodata
Usage: sidepocket.del [options] <name>
Options:
--help : Display the command usage.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.elasticsearch.add
Add a Elasticsearch data source.
This command configures a Elasticsearch data source for use in sidepocket queries.
Examples:
// Add a ES connection named 'foodata'.
sidepocket.elasticsearch.add foodata --host es01 --index foodata
// Add a ES connection for supporting queries across a index prefix
sidepocket.elasticsearch.add foodata --host es01 --index "foo*"
Usage: sidepocket.elasticsearch.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Elasticsearch host.
--index <index> : Database to use.
--sniff : Enable connection sniffing.
--timeout <timeout> : Connection timeout information.
--basicauth <basicauth> : Username and password tuple used for connecting to the Elasticsearch instance.
--no-ssl-verify : Disable SSL certificate verification.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.impala.add
Add an Apache Impala data source.
This command configures an Apache Impala data source for use in sidepocket queries.
Examples:
// Add an Impala connection named 'foodata'.
sidepocket.impala.add foodata --host localhost --user testuser
--password testpass --db cooldb
Usage: sidepocket.impala.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--user <user> : Username to log in as.
--password <password> : Password to use.
--db <db> : Database to use.
--port <port> : Port to use.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.kusto.add
Add an Azure Kusto data source.
This command configures an Azure Kusto data source for use in sidepocket queries.
The following authentication methods are supported, with "MSI" used as the default:
- "basic": requires username and password with authority-id being optional
- "application_key": requires client-id, client-secret, and authority-id
- "msi": authority-id is optional
- "unsecured": no additional arguments are required
The dbname can be set for the source or provided in the opts dictionary when querying.
Examples:
// Add a Kusto connection named 'foodata' that will use MSI authentication.
sidepocket.kusto.add foodata https://foocluster00.kusto.windows.net
// Add a Kusto connection that uses a configured OAuthV2 provider
sidepocket.kusto.oauth2.add foo mycluster00 eastus tenant00
myclientid myclientsecret https://my.optic.netloc/oauth2
sidepocket.kusto.add foodata https://foocluster00.kusto.windows.net
--authenticator oauth2 --oauth2-provider foo
Usage: sidepocket.kusto.add [options] <name> <cluster>
Options:
--help : Display the command usage.
--authenticator <authenticator>: The authentication method to use. (default: msi)
--username <username> : The AAD username to use with username/password authentication.
--password <password> : The AAD password to use with username/password authentication.
--client-id <client_id> : The AAD application ID to use with key or MSI authentication.
--client-secret <client_secret>: The AAD application key to use with key authentication
--authority-id <authority_id>: The AAD tenant ID to use with key or user/password authentication
--dbname <dbname> : The database name to use for all queries.
--oauth2-provider <oauth2_provider>: The OAuth V2 provider name used in sidepocket.kusto.oauth2.add.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<cluster> : The cluster URL.
sidepocket.kusto.oauth2.add
Add a Kusto OAuth V2 provider to the Cortex.
Examples:
// Add a Kusto OAuth V2 provider named "foo" for use with Sidepocket sources
sidepocket.kusto.oauth2.add foo mycluster00 eastus tenant00
myclientid myclientsecret https://my.optic.netloc/oauth2
sidepocket.kusto.add foodata https://foocluster00.kusto.windows.net
--authenticator oauth2 --oauth2-provider foo
Usage: sidepocket.kusto.oauth2.add [options] <name> <cluster-id> <region> <authority-id> <client-id> <client-secret> <redirect-uri>
Options:
--help : Display the command usage.
Arguments:
<name> : Name for the OAuth V2 provider configuration.
<cluster_id> : The cluster name.
<region> : The cluster region.
<authority_id> : The tenant id.
<client_id> : The OAuth V2 client id.
<client_secret> : The OAuth V2 client secret.
<redirect_uri> : The local URI to redirect OAuth V2 auth requests (e.g. "https://<optic_netloc>/oauth2").
sidepocket.mysql.add
Add a MySQL data source.
This command configures a MySQL data source for use in sidepocket queries.
Examples:
// Add a MySQL connection named 'foodata'.
sidepocket.mysql.add foodata --host localhost --user testuser
--password testpass --db cooldb
Usage: sidepocket.mysql.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--user <user> : Username to log in as.
--password <password> : Password to use.
--db <db> : Database to use.
--port <port> : MySQL port to use.
--timeout <timeout> : Timeout in seconds when establishing a connection.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.neo4j.add
Add a Neo4j data source.
This command configures a Neo4j data source for use in sidepocket queries.
Examples:
// Add a Neo4j connection named 'foodata'.
sidepocket.neo4j.add foodata --uri neo4j://localhost:1234
--auth (testuser, testpass)
Usage: sidepocket.neo4j.add [options] <name>
Options:
--help : Display the command usage.
--uri <uri> : URI for connecting to Neo4j
--auth <auth> : A tuple of username and password information for basic auth.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.orientdb.add
Add an OrientDB data source.
This command configures an OrientDB data source for use in sidepocket queries.
The db_name can be set for the source or provided in the opts dictionary when querying.
Note: Only OrientDB versions prior to 3.0.0 are supported with this connector.
Examples:
// Add an OrientDB connection named 'foodata'.
sidepocket.orientdb.add foodata --host dbhost --user dbuser --password dbpass
Usage: sidepocket.orientdb.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--port <port> : Port to use for connecting to the host. (default: 2424)
--user <user> : The username to use for authentication.
--password <password> : The password to use for authentication.
--db_name <db_name> : Default database to connect to for querying.
--timeout <timeout> : The amount of time to wait for each operation (defaults to 30). (default: 30)
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.postgresql.add
Add a PostgreSQL data source.
This command configures a PostgreSQL data source for use in sidepocket queries.
Examples:
// Add a PostgreSQL connection named 'foodata'.
sidepocket.postgresql.add foodata --host localhost --user testuser
--password testpass --dbname cooldb
Usage: sidepocket.postgresql.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--user <user> : Username to log in as.
--password <password> : Password to use.
--dbname <dbname> : Database to use.
--port <port> : Port to use.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.presto.add
Add a Presto data source.
This command configures a Presto data source for use with
the sidepocket.query command.
Examples:
// Add a Presto connection named 'foodata'.
sidepocket.presto.add foodata --host localhost --username testuser
--catalog coolcat --schema coolschema
Usage: sidepocket.presto.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--username <username> : Username to log in as.
--catalog <catalog> : Catalog to use.
--schema <schema> : Schema to use.
--port <port> : Port to use.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.query
Query a side pocket data source.
This command queries a data source that has been configured using
sidepocket.add and prints out the values returned by the query.
The Sidepocket Storm API can be used to iterate over the results for ingesting data.
Example:
// Query a previously configured MySQL db which has a table named 'dataz'
sidepocket.query foodb "select * from dataz"
// Use the Storm API to iterate over the rows
$sidepocket = $lib.import(sidepocket)
$queryopts = ({"args": ["baz"]})
$query = 'SELECT foo, bar FROM sometable WHERE foo=%s'
for ($mtype, $data, $info) in $sidepocket.query(foodb, $query, opts=$queryopts) {
switch $mtype {
"data": { $lib.print($data) /* data var is the raw row */ }
"warn": { $lib.warn($data) }
}
}
Usage: sidepocket.query [options] <name> <query>
Options:
--help : Display the command usage.
--opts <opts> : Additional options to pass when querying the data source.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<query> : The query for the data source.
sidepocket.rename
Rename a side pocket data source.
Note: Renaming a source does not alter any existing permissions
using the power-ups.sidepocket.sources.<name> pattern.
Examples:
// Rename a connection from 'foodata' to 'bardata'.
sidepocket.rename foodata bardata
Usage: sidepocket.rename [options] <name> <newv>
Options:
--help : Display the command usage.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<newv> : New name for the data source.
sidepocket.results
Get results from a previously executed query.
If the query has not finished, this command will poll until it does.
Output is a tuple of (type, data, info):
- type: The type of message
- init: data equals the query_id
- print: If the debug option is set, data is a message str (e.g. polling status)
- warn: Data is a warning or error message
- data: A tuple of values by column
- fini: Indicates the command has completed
- data: Data to parse as specified by the type
- info: A dict containing additional information (e.g. excinfo for certain warns)
For long-running queries the sidepocket.status command can be used to check
if the query completed instead of waiting on polling to complete.
Example:
// Return results from a previously executed query
sidepocket.results foodata "1b7bd133-c766-4bea-b481-8dd5cbb6de00"
// Use the Storm API query method to loop over data
$sidepocket = $lib.import(sidepocket)
$queryopts = ({"query_id": "1b7bd133-c766-4bea-b481-8dd5cbb6de00"})
// NOTE: The second arg "query" is set to null
for ($type, $data, $info) in $sidepocket.query(foodata, $lib.null, opts=$queryopts) {
switch $type {
init: { $lib.print("query_id={q}", q=$data) }
print: { $lib.print($data) }
warn: {
$lib.warn($data)
if $info.excinfo { $lib.warn($info.excinfo) }
}
data: {
($col1, $col2) = $data
}
}
}
Usage: sidepocket.results [options] <name> <query-id>
Options:
--help : Display the command usage.
--timeout <timeout> : The amount of time to wait for each Athena operation (defaults to 30). (default: 30)
--max-items <max_items> : Return up to max-items results (defaults to no limit).
--page-size <page_size> : Fetch pages of page-size (defaults to 500). (default: 500)
--poll-s <poll_s> : Poll at this interval to wait for the query to finish (defaults to 5). (default: 5)
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
<query_id> : The query_id to retrieve results for.
sidepocket.snowflake.add
Add a Snowflake data source.
This command configures a Snowflake data source for use in sidepocket queries.
Examples:
// Add a Snowflake connection named 'foodata'.
sidepocket.snowflake.add foodata --account xy12345.us-east-2.aws
--user testuser
--password testpass
// Add a Snowflake connection that uses a configured OAuthV2 provider
sidepocket.snowflake.oauth2.add foo xy12345.us-east-2.aws
myclientid
myclientsecret
https://my.optic.netloc/oauth2
sidepocket.snowflake.add foodata --authenticator oauth --account myorg-myaccount --oauth2-provider foo
Usage: sidepocket.snowflake.add [options] <name>
Options:
--help : Display the command usage.
--authenticator <authenticator>: Authenticator to use for Snowflake. Valid values are (snowflake, oauth) (default: snowflake)
--account <account> : Account to log in to.
--user <user> : User to log in as.
--password <password> : Password to use.
--token <token> : OAuth access token to use for authentication.
--oauth2-provider <oauth2_provider>: The OAuth V2 provider name used in sidepocket.snowflake.oauth2.add.
--database <database> : Default database to use.
--schema <schema> : Default schema to use.
--warehouse <warehouse> : Default warehouse to use.
--role <role> : Default role to use.
--login-timeout <login_timeout>: Timeout in seconds for login (defaults to 60). (default: 60)
--network-timeout <network_timeout>: Timeout in seconds for all other operations (defaults to none).
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.snowflake.oauth2.add
Add a Snowflake OAuth V2 provider to the Cortex.
Examples:
// Add a Snowflake OAuth V2 provider named "foo" for use with Sidepocket sources.
sidepocket.snowflake.oauth2.add foo xy12345.us-east-2.aws
myclientid
myclientsecret
https://my.optic.netloc/oauth2
sidepocket.snowflake.add foodata --authenticator oauth --account myorg-myaccount --oauth2-provider foo
Usage: sidepocket.snowflake.oauth2.add [options] <name> <account> <client-id> <client-secret> <redirect-uri>
Options:
--help : Display the command usage.
Arguments:
<name> : Name for the OAuth V2 provider configuration.
<account> : Account to log in to.
<client_id> : The OAuth V2 client id.
<client_secret> : The OAuth V2 client secret.
<redirect_uri> : The local URI to redirect OAuth V2 auth requests (e.g. "https://<optic_netloc>/oauth2").
sidepocket.sources
List the configured data sources.
Usage: sidepocket.sources [options]
Options:
--help : Display the command usage.
--debug : Show verbose debug output.
sidepocket.status
Get the status for a single query or all queries.
Note that only connectors that support async query execution implement this functionality.
Examples:
// Get the status for a single query
sidepocket.status foodata --query-id "1b7bd133-c766-4bea-b481-8dd5cbb6de00"
// Get the status for all queries
sidepocket.status foodata
// Use the Storm API status method to loop over data
$sidepocket = $lib.import(sidepocket)
$opts = ({"max_items": 100})
for $resu in $sidepocket.status(foodata, opts=$opts) {
warn: {
$lib.warn($data)
if $info.excinfo { $lib.warn($info.excinfo) }
}
data: {
$lib.print("query_id={q} state={s}", q=$data.query_id, s=$data.state)
}
}
Usage: sidepocket.status [options] <name>
Options:
--help : Display the command usage.
--query-id <query_id> : The query_id to retrieve results for.
--max-items <max_items> : Return up to max-items results (defaults to 25). (default: 25)
--page-size <page_size> : Fetch pages of page-size (defaults to 25). (default: 25)
--database <database> : Database to pull query information from.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
sidepocket.trino.add
Add a Trino data source.
This command configures a Trino data source for use with
the sidepocket.query command.
Examples:
// Add a Trino connection named 'foodata'.
sidepocket.trino.add foodata --host localhost --username testuser
--catalog coolcat --schema coolschema
Usage: sidepocket.trino.add [options] <name>
Options:
--help : Display the command usage.
--host <host> : Host where the database server is located.
--username <username> : Username to log in as.
--catalog <catalog> : Catalog to use.
--schema <schema> : Schema to use.
--port <port> : Port to use.
--debug : Show verbose debug output.
Arguments:
<name> : Name for the data source.
Storm Modules
This package does not export any Storm APIs.