DevOps Guide

Overview

For a general overview of common devops tasks for Synapse services see Synapse Devops Guide - Overview.

Common DevOps Tasks

Use a Custom Bucket

The s3:bucket option can be used to override the default axon bucket name. If the bucket does not exist when Synapse S3 Axon starts up, the service will attempt to create it using the provided credentials.

For example, to set using an environment variable in the docker-compose.yaml file:

environment:
    # ...
    SYN_S3AXON_S3_BUCKET=mycustombucket

Provide Custom Boto3 Configuration Options

To configure the S3 options, you can set the following configuration values:

s3:boto3
This is a dictionary containing configuration options for the Boto3 SDK, which is used to access S3. For example, if you are using an access key ID and a secret access key to interact with S3, those values should be set in aws_access_key_id and aws_secret_access_key values of the dictionary.

If this configuration value is not provided, Boto3 will attempt to resolve its configuration information via its documented methods, including AWS IAM based roles. This allows for Synapse S3 Axon to be deployed in AWS environments (EC2, ECS, EKS) which may be configured to allow hosts or containers to access specific S3 buckets without the need to specify access key IDs and secret access keys to the service directly.

More information about the Boto3 options can be found at boto3 configuration options and boto3 credentials options.

s3:upload:multipart:chunksize
The partition size of each part for a multipart upload. Only files above this size will be uploaded as multipart uploads. By default, the service uses a value of 100MB.

s3:upload:io:queue:max
The maximum amount of read parts that can be queued in memory for multipart uploads. By default, the service uses a value of 10.

An example docker-compose.yaml file can be seen below. This example omits the s3:boto3 configuration. This would cause Boto3 to resolve the service credentials itself, possibly through AWS IAM roles:

version: "3.3"
services:
  00.axon:
    user: "999"
    image: vertexproject/synapse-s3:v3.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        SYN_S3AXON_HTTPS_PORT: null
        SYN_S3AXON_AHA_PROVISION: "ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>"

Deploy a Mirror

A Synapse S3 Axon mirror can independently read and write to the S3 storage bucket, with all changes to the local index data written back to the upstream.

Inside the AHA container

Generate a one-time use URL for provisioning from inside the AHA container:

python -m synapse.tools.aha.provision.service 01.axon --mirror axon

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container storage directory:

mkdir -p /srv/syn/01.axon/storage
chown -R 999 /srv/syn/01.axon/storage

Create the /srv/syn/01.s3/docker-compose.yaml file with contents:

version: "3.3"
services:
  01.axon:
    user: "999"
    image: vertexproject/synapse-s3:v3.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        # disable HTTPS API for now to prevent port collisions
        - SYN_S3AXON_HTTPS_PORT=null
        - SYN_S3AXON_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/01.s3/docker-compose.yaml pull
docker-compose --file /srv/syn/01.s3/docker-compose.yaml up -d

Devops Details

Docker Images

The Synapse-S3 service is available as a Docker container from Docker Hub. The repository can be found at:

https://hub.docker.com/repository/docker/vertexproject/synapse-s3/

Note

There are tagged images available on Docker Hub which correspond to software releases seen in the changelog. The docker tag master is the latest development release. A generic major version tag is available, representing the latest release on a given major version. For example, the v2.x.x tag represents the most current release for the v2.x.x release line. You can utilize specific tagged versions, or a major version specifier, depending on your chosen deployment strategy.

Required Permissions

The provided credentials must have the following permissions (when used in AWS):

s3:ListBucket: This is used to ensure that the S3 bucket used to store files in exists.
s3:GetObject: This is used to retrieve files uploaded the S3 bucket.
s3:PutObject: This is used to put files in the S3 bucket.
s3:DeleteObject: This is used to delete files uploaded to the S3 bucket. In the case where a bulk delete is requested the delete operation is run on each object. If an S3 delete operation fails (e.g. a role permission error), the SHA-256 will still be removed from the Axon index and the object would need to be manually deleted in the bucket.
s3:CreateBucket: This permission is only needed to make the bucket if the bucket does not exist. It is best practice when integrating with AWS S3 to create the bucket separately, since the Synapse S3 service does not specify any sort of access policies when it creates the bucket.

Since Synapse S3 Axon implements the Synapse Axon API, permissions are also inherited. For example, the axon.get rule is required to retrieve a file and the axon.upload rule is required to upload a file.

Configuration Options

The following is a list of available configuration options.

aha:admin

An AHA client certificate CN to register as a local admin user.

Type: string
Environment Variable: SYN_S3AXON_AHA_ADMIN

aha:leader

The AHA service name to claim as the active instance of a storm service.

Type: string
Environment Variable: SYN_S3AXON_AHA_LEADER

aha:name

The name of the cell service in the aha service registry.

Type: string
Environment Variable: SYN_S3AXON_AHA_NAME

aha:network

The AHA service network.

Type: string
Environment Variable: SYN_S3AXON_AHA_NETWORK

aha:provision

The telepath URL of the aha provisioning service.

Type: ['string', 'array']
Environment Variable: SYN_S3AXON_AHA_PROVISION

aha:registry

The telepath URL of the aha service registry.

Type: ['string', 'array']
Environment Variable: SYN_S3AXON_AHA_REGISTRY

aha:user

The username of this service when connecting to others.

Type: string
Environment Variable: SYN_S3AXON_AHA_USER

auth:anon

Allow anonymous telepath access by mapping to the given user name.

Type: string
Environment Variable: SYN_S3AXON_AUTH_ANON

auth:passwd

Set to <passwd> (local only) to bootstrap the root user password.

Type: string
Environment Variable: SYN_S3AXON_AUTH_PASSWD

auth:passwd:policy

Specify password policy/complexity requirements.

Type: object
Environment Variable: SYN_S3AXON_AUTH_PASSWD_POLICY

backup:dir

A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.

Type: string
Environment Variable: SYN_S3AXON_BACKUP_DIR

dmon:listen

A config-driven way to specify the telepath bind URL.

Type: ['string', 'null']
Environment Variable: SYN_S3AXON_DMON_LISTEN

health:sysctl:checks

Enable sysctl parameter checks and warn if values are not optimal.

Type: boolean
Default Value: True
Environment Variable: SYN_S3AXON_HEALTH_SYSCTL_CHECKS

http:proxy

An aiohttp-socks compatible proxy URL to use in the wget API.

Type: string
Environment Variable: SYN_S3AXON_HTTP_PROXY

https:headers

Headers to add to all HTTPS server responses.

Type: object
Environment Variable: SYN_S3AXON_HTTPS_HEADERS

https:parse:proxy:remoteip

Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.

Type: boolean
Default Value: False
Environment Variable: SYN_S3AXON_HTTPS_PARSE_PROXY_REMOTEIP

https:port

A config-driven way to specify the HTTPS port.

Type: ['integer', 'null']
Environment Variable: SYN_S3AXON_HTTPS_PORT

limit:disk:free

Minimum disk free space percentage before setting the cell read-only.

Type: ['integer', 'null']
Default Value: 5
Environment Variable: SYN_S3AXON_LIMIT_DISK_FREE

max:bytes

The maximum number of bytes that can be stored in the Axon.

Type: integer
Environment Variable: SYN_S3AXON_MAX_BYTES

max:count

The maximum number of files that can be stored in the Axon.

Type: integer
Environment Variable: SYN_S3AXON_MAX_COUNT

max:users

Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).

Type: integer
Default Value: 0
Environment Variable: SYN_S3AXON_MAX_USERS

nexslog:en

Record all changes to a stream file on disk. Required for mirroring (on both sides).

Type: boolean
Default Value: False
Environment Variable: SYN_S3AXON_NEXSLOG_EN

onboot:optimize

Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.

Type: boolean
Default Value: False
Environment Variable: SYN_S3AXON_ONBOOT_OPTIMIZE

s3:boto3

Boto3 configuration options.

Type: object
Default Value: None
Environment Variable: SYN_S3AXON_S3_BOTO3

s3:bucket

The name of the bucket to use.

Type: string
Default Value: 'axon'
Environment Variable: SYN_S3AXON_S3_BUCKET

s3:log:level

Log level for S3 related logging. Enabling this at the DEBUG level may contain sensitive information such as private key materials depending on the deployment configuration

Type: string
Default Value: 'INFO'
Environment Variable: SYN_S3AXON_S3_LOG_LEVEL

s3:upload:io:queue:max

The maximum amount of read parts that can be queued in memory.

Type: integer
Default Value: 10
Environment Variable: SYN_S3AXON_S3_UPLOAD_IO_QUEUE_MAX

s3:upload:multipart:chunksize

The partition size of each part for a multipart upload. Only files above this size will be uploaded as multipart uploads.

Type: integer
Default Value: 100000000
Environment Variable: SYN_S3AXON_S3_UPLOAD_MULTIPART_CHUNKSIZE

tls:ca:dir

An optional directory of CAs which are added to the TLS CA chain for wget and wput APIs.

Type: string
Environment Variable: SYN_S3AXON_TLS_CA_DIR