Synapse Deployment Guide

Introduction

This step-by-step guide will walk you through a production-ready Synapse deployment. Services will be configured to register with AHA for service discovery and to prepare for future devops tasks such as promoting a mirror to leader and provisioning future Synapse Advanced Power-Ups.

This guide will also walk you through deploying all Synapse services using TLS to authenticate both servers and clients using client-certificates to minimize the need for secrets management by eliminating passwords from all telepath URLs.

For the purposes of this guide, we will use docker-compose as a light-weight orchestration mechanism. The steps, configurations, and volume mapping guidance given in this guide apply equally to other container orchestration mechanisms such as Kubernetes but for simplicity’s sake, this guide will only cover docker-compose based deployments.

Note

Due to known networking limitations of docker on Mac we do not support or recommend the use of Docker for Mac for testing or deploying production Synapse instances. Containers run within separate docker-compose commands will not be able to reliably communicate with each other.

Synapse services require persistent storage. Each docker container expects persistent storage to be available within the directory /vertex/storage which should be a persistent mapped volume. Only one container may run from a given volume at a time.

Note

To allow hosts to be provisioned on one system, this guide instructs you to disable HTTP API listening ports on all services other than the main Cortex. You may remove those configuration options if you are running on separate hosts or select alternate ports which do not conflict.

Prepare your Hosts

Ensure that you have an updated install of docker and docker-compose.

In order to help you run the Synapse service containers as a non-root user, Synapse service docker containers have been preconfigured with a user named synuser with UID 999. You may replace 999 in the configs below, but keep in mind that doing so will result in the container not having a name for the user. We recommend that you do not use the Linux user nobody for this purpose.

Default kernel parameters on most Linux distributions are not optimized for database performance. We recommend adding the following lines to /etc/sysctl.conf on all systems being used to host Synapse services:

vm.swappiness=10
vm.dirty_expire_centisecs=20
vm.dirty_writeback_centisecs=20

See Performance Tuning for a list of additional tuning options.

We will use the directory /srv/syn/ on the host systems as the base directory used to deploy the Synapse services. Each service will be deployed in separate /srv/syn/<svcname> directories. This directory can be changed to whatever you would like, and the services may be deployed to any host provided that the hosts can directly connect to each other. It is critical to performance that these storage volumes be low-latency. More latent storage mechanisms such as spinning disks, NFS, or EFS should be avoided!

We highly recommend that hosts used to run Synapse services deploy a log aggregation agent to make it easier to view the logs from the various containers in a single place.

When using AHA, you may run any of the other services on additional hosts as long as they can connect directly to the AHA service. You may also shutdown a service, move it’s volume to a different host, and start it backup without changing anything.

Decide on a Name

Throughout the examples, we will be using <yournetwork> as the AHA network name which is also used as the common-name (CN) for the CA certificate. This should be changed to an appropriate network name used by your synapse deployment such as syn.acmecorp.com. We will use <yournetwork> in the following configs to specify locations which should be replaced with your selected AHA network name. For a test deployment which runs all docker containers on one host, you may use loop.vertex.link.

Note

It is important that you choose a name and stick with it for a given deployment. Once we begin generating host and service account certificates, changing this name will be difficult.

Deploy AHA Service

The AHA service is used for service discovery and acts as a CA to issue host/user certificates used to link Synapse services. Other Synapse services will need to be able to resolve the IP address of the AHA service by name, so it is likely that you need to create a DNS A/AAAA record in your existing resolver. When you are using AHA, the only host that needs DNS or other external name resolution is the AHA service.

Note

It is important to ensure that aha.<yournetwork> is resolvable via DNS or docker container service name resolution from within the container environment! There are configuration options you may use if this is impossible, but the configuration is far simpler if we can make this assumption.

Create the container directory:

mkdir -p /srv/syn/aha/storage

Create the /srv/syn/aha/docker-compose.yaml file with contents:

version: "3.3"
services:
  aha:
    user: "999"
    image: vertexproject/synapse-aha:v2.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        - SYN_AHA_HTTPS_PORT=null
        - SYN_AHA_AHA_NAME=aha
        - SYN_AHA_AHA_NETWORK=<yournetwork>
        - SYN_AHA_DMON_LISTEN=ssl://aha.<yournetwork>?ca=<yournetwork>
        - SYN_AHA_PROVISION_LISTEN=ssl://aha.<yournetwork>:27272

Note

Don’t forget to replace <yournetwork> with your chosen network name!

Change ownership of the storage directory to the user you will use to run the container:

chown -R 999 /srv/syn/aha/storage

Start the container using docker-compose:

docker-compose -f /srv/syn/aha/docker-compose.yaml pull
docker-compose -f /srv/syn/aha/docker-compose.yaml up -d

To view the container logs at any time you may run the following command on the host from the /srv/syn/aha directory:

docker-compose logs -f

You may also execute a shell inside the container using docker-compose from the /srv/syn/aha directory on the host. This will be necessary for some of the additional provisioning steps:

docker-compose exec aha /bin/bash

Deploy Axon Service

In the Synapse service architecture, an Axon provides a place to store arbitrary bytes/files as binary blobs and exposes APIs for streaming files in and out regardless of their size. Given sufficient file system size, an Axon can be used to efficiently store and retrieve very large files as well as a high number (easily billions) of files.

Inside the AHA container

Generate a one-time use provisioning URL:

python -m synapse.tools.aha.provision.service 00.axon

These one-time use URLs are used to connect to the Aha service, retrieve configuration data, and provision SSL certificates for the service. When this is done, the service records that the URL has been used in its persistent storage, and will not attempt to perform the provisioning process again unless the URL changes. If the provisioning URL is reused, services will encounter NoSuchName errors and fail to start up - this indicates a service has attempted to re-use the one-time use URL!

Note

We strongly encourage you to use a numbered hierarchical naming convention for services where the first part of the name is a 0 padded number and the second part is the service type. The above example 00.axon will allow you to deploy mirror instances in the future, such as 01.axon, where the AHA name axon.<yournetwork> will automatically resolve to which ever one is the current leader.

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container directory:

mkdir -p /srv/syn/00.axon/storage
chown -R 999 /srv/syn/00.axon/storage

Create the /srv/syn/00.axon/docker-compose.yaml file with contents:

version: "3.3"
services:
  00.axon:
    user: "999"
    image: vertexproject/synapse-axon:v2.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        # disable HTTPS API for now to prevent port collisions
        - SYN_AXON_HTTPS_PORT=null
        - SYN_AXON_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/00.axon/docker-compose.yaml pull
docker-compose --file /srv/syn/00.axon/docker-compose.yaml up -d

Deploy JSONStor Service

Inside the AHA container

Generate a one-time use provisioning URL:

python -m synapse.tools.aha.provision.service 00.jsonstor

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container directory:

mkdir -p /srv/syn/00.jsonstor/storage
chown -R 999 /srv/syn/00.jsonstor/storage

Create the /srv/syn/00.jsonstor/docker-compose.yaml file with contents:

version: "3.3"
services:
  00.jsonstor:
    user: "999"
    image: vertexproject/synapse-jsonstor:v2.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        # disable HTTPS API for now to prevent port collisions
        - SYN_JSONSTOR_HTTPS_PORT=null
        - SYN_JSONSTOR_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/00.jsonstor/docker-compose.yaml pull
docker-compose --file /srv/syn/00.jsonstor/docker-compose.yaml up -d

Deploy Cortex Service

Inside the AHA container

Generate a one-time use provisioning URL:

python -m synapse.tools.aha.provision.service 00.cortex

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container directory:

mkdir -p /srv/syn/00.cortex/storage
chown -R 999 /srv/syn/00.cortex/storage

Create the /srv/syn/00.cortex/docker-compose.yaml file with contents:

version: "3.3"
services:
  00.cortex:
    user: "999"
    image: vertexproject/synapse-cortex:v2.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        - SYN_CORTEX_AXON=aha://axon...
        - SYN_CORTEX_JSONSTOR=aha://jsonstor...
        - SYN_CORTEX_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Note

The values aha://axon... and aha://jsonstor... can be used as-is without changing them because the AHA network (provided by the provisioning server) is automatically subtituted in any aha:// scheme URL ending with ...

Start the container:

docker-compose --file /srv/syn/00.cortex/docker-compose.yaml pull
docker-compose --file /srv/syn/00.cortex/docker-compose.yaml up -d

Remember, you can view the container logs in real-time using:

docker-compose --file /srv/syn/00.cortex/docker-compose.yaml logs -f

Deploy Cortex Mirror (optional)

Inside the AHA container

Generate a one-time use URL for provisioning from inside the AHA container:

python -m synapse.tools.aha.provision.service 01.cortex --mirror cortex

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container storage directory:

mkdir -p /srv/syn/01.cortex/storage
chown -R 999 /srv/syn/01.cortex/storage

Create the /srv/syn/01.cortex/docker-compose.yaml file with contents:

version: "3.3"
services:
  01.cortex:
    user: "999"
    image: vertexproject/synapse-cortex:v2.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        - SYN_CORTEX_AXON=aha://axon...
        - SYN_CORTEX_JSONSTOR=aha://jsonstor...
        # disable HTTPS API for now to prevent port collisions
        - SYN_CORTEX_HTTPS_PORT=null
        - SYN_CORTEX_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/01.cortex/docker-compose.yaml pull
docker-compose --file /srv/syn/01.cortex/docker-compose.yaml up -d

Note

If you are deploying a mirror from an existing large Cortex, this startup may take a while to complete initialization.

Enroll CLI Users

A Synapse user is generally synonymous with a user account on the Cortex. To bootstrap CLI users who will have Cortex access using the Telepath API, we will need to add them to the Cortex and generate user certificates for them. To add a new admin user to the Cortex, run the following command from inside the Cortex container:

python -m synapse.tools.moduser --add --admin true visi

Note

If you are a Synapse Enterprise customer, using the Synapse UI with SSO, the admin may now login to the Synapse UI. You may skip the following steps if the admin will not be using CLI tools to access the Cortex.

Then we will need to generate a one-time use URL they may use to generate a user certificate. Run the following command from inside the AHA container to generate a one-time use URL for the user:

python -m synapse.tools.aha.provision.user visi

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Then the user may run:

python -m synapse.tools.aha.enroll ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Once they are enrolled, they will have a user certificate located in ~/.syn/certs/users and their telepath configuration located in ~/.syn/telepath.yaml will be updated to reflect the use of the AHA server. From there the user should be able to use standard Synapse CLI tools using the aha:// URL such as:

python -m synapse.tools.storm aha://visi@cortex.<yournetwork>

Configure a Storm Query Pool (optional)

A Cortex may be configured to use a pool of mirrors in order to offload Storm query execution and distribute query load among a configurable group of mirrors. We will assume you have configured two additional mirrors named 01.cortex... and 02.cortex... using the process described in the previous Deploy Cortex Mirror (optional) step. In our example, we will also assume that the mirrors will be used for both query parallelism and for graceful promotions to minimize downtime during upgrades and optimization.

The following commands are run using the Storm CLI tool discussed in the Enroll CLI Users section. First, use the Storm CLI to run the aha.pool.add command to create a new AHA pool:

aha.pool.add pool00.cortex...

Then add the Cortex leader as well as the two mirrors to the pool:

aha.pool.svc.add pool00.cortex... 00.cortex...
aha.pool.svc.add pool00.cortex... 01.cortex...
aha.pool.svc.add pool00.cortex... 02.cortex...

Then configure the Cortex to use the newly created AHA service pool:

cortex.storm.pool.set aha://pool00.cortex...

Now your Cortex will distribute Storm queries across the available mirrors. You may add or remove mirrors from the pool at any time using the aha.pool.svc.add and aha.pool.svc.del commands and the pool topology updates will be automatically sent. You may want to review some of the command options to adjust timeouts for your environment.

If you wish to remove the pool configuration from the Cortex you may use the cortex.storm.pool.del command.

What’s next?

See the Synapse Admin Guide for instructions on performing application administrator tasks. See the Synapse Devops Guide for instructions on performing various maintenance tasks on your deployment!