Devops Guide

Overview

For a general overview of common devops tasks for Synapse services see Synapse Devops Guide - Overview.

Common DevOps Tasks

Set the Maximum Number of Workers

Parsing tasks are distributed to subprocess workers. By default Synapse FileParser will set a maxmium number of parallel workers equal to the number of CPUs detected on the host, or 1 if the detection failed. The workers configuration option can be used to set this value directly.

Deploy a Mirror

Inside the AHA container

Generate a one-time use URL for provisioning from inside the AHA container:

python -m synapse.tools.aha.provision.service 01.fileparser --mirror fileparser

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container storage directory:

mkdir -p /srv/syn/01.fileparser/storage
chown -R 999 /srv/syn/01.fileparser/storage

Create the /srv/syn/01.fileparser/docker-compose.yaml file with contents:

version: "3.3"
services:
  01.fileparser:
    user: "999"
    image: vertexproject/synapse-fileparser:v4.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        # disable HTTPS API for now to prevent port collisions
        - SYN_FILEPARSER_HTTPS_PORT=null
        - SYN_FILEPARSER_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/01.fileparser/docker-compose.yaml pull
docker-compose --file /srv/syn/01.fileparser/docker-compose.yaml up -d

Devops Details

Docker Images

The Synapse FileParser service is available as a Docker container from Docker Hub. The repository can be found at:

https://hub.docker.com/repository/docker/vertexproject/synapse-fileparser/

Note

There are tagged images available on Docker Hub which correspond to software releases seen in the changelog. The docker tag master is the latest development release. A generic major version tag is available, representing the latest release on a given major version. For example, the v2.x.x tag represents the most current release for the v2.x.x release line. You can utilize specific tagged versions, or a major version specifier, depending on your chosen deployment strategy.

Configuration Options

The following is a list of available configuration options.

aha:admin

An AHA client certificate CN to register as a local admin user.

Type: string
Environment Variable: SYN_FILEPARSER_AHA_ADMIN

aha:leader

The AHA service name to claim as the active instance of a storm service.

Type: string
Environment Variable: SYN_FILEPARSER_AHA_LEADER

aha:name

The name of the cell service in the aha service registry.

Type: string
Environment Variable: SYN_FILEPARSER_AHA_NAME

aha:network

The AHA service network.

Type: string
Environment Variable: SYN_FILEPARSER_AHA_NETWORK

aha:provision

The telepath URL of the aha provisioning service.

Type: ['string', 'array']
Environment Variable: SYN_FILEPARSER_AHA_PROVISION

aha:registry

The telepath URL of the aha service registry.

Type: ['string', 'array']
Environment Variable: SYN_FILEPARSER_AHA_REGISTRY

aha:user

The username of this service when connecting to others.

Type: string
Environment Variable: SYN_FILEPARSER_AHA_USER

auth:anon

Allow anonymous telepath access by mapping to the given user name.

Type: string
Environment Variable: SYN_FILEPARSER_AUTH_ANON

auth:passwd

Set to <passwd> (local only) to bootstrap the root user password.

Type: string
Environment Variable: SYN_FILEPARSER_AUTH_PASSWD

auth:passwd:policy

Specify password policy/complexity requirements.

Type: object
Environment Variable: SYN_FILEPARSER_AUTH_PASSWD_POLICY

axon

Telepath url to axon.

Type: string
Environment Variable: SYN_FILEPARSER_AXON

backup:dir

A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.

Type: string
Environment Variable: SYN_FILEPARSER_BACKUP_DIR

dmon:listen

A config-driven way to specify the telepath bind URL.

Type: ['string', 'null']
Environment Variable: SYN_FILEPARSER_DMON_LISTEN

health:sysctl:checks

Enable sysctl parameter checks and warn if values are not optimal.

Type: boolean
Default Value: True
Environment Variable: SYN_FILEPARSER_HEALTH_SYSCTL_CHECKS

https:headers

Headers to add to all HTTPS server responses.

Type: object
Environment Variable: SYN_FILEPARSER_HTTPS_HEADERS

https:parse:proxy:remoteip

Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.

Type: boolean
Default Value: False
Environment Variable: SYN_FILEPARSER_HTTPS_PARSE_PROXY_REMOTEIP

https:port

A config-driven way to specify the HTTPS port.

Type: ['integer', 'null']
Environment Variable: SYN_FILEPARSER_HTTPS_PORT

limit:disk:free

Minimum disk free space percentage before setting the cell read-only.

Type: ['integer', 'null']
Default Value: 5
Environment Variable: SYN_FILEPARSER_LIMIT_DISK_FREE

max:users

Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).

Type: integer
Default Value: 0
Environment Variable: SYN_FILEPARSER_MAX_USERS

nexslog:en

Record all changes to a stream file on disk. Required for mirroring (on both sides).

Type: boolean
Default Value: False
Environment Variable: SYN_FILEPARSER_NEXSLOG_EN

onboot:optimize

Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.

Type: boolean
Default Value: False
Environment Variable: SYN_FILEPARSER_ONBOOT_OPTIMIZE

workers

Max parallel workers (by default will be set to number of CPUs in the system).

Type: integer
Environment Variable: SYN_FILEPARSER_WORKERS