DevOps Guide

Overview

For a general overview of common devops tasks for Synapse services see Synapse Devops Guide - Overview.

Common DevOps Tasks

Modify the Number of Workers

Synapse Yara Grid leverages Synapse Swarm and multiprocessing workers to achieve efficient concurrency for matching a given file against a large number of rules.

The workers configuration option can be used to control the number of Yara match workers to spin up.

For example, to set this in the docker-compose.yaml file as an environment variable:

environment:
    # ...
    - SYN_YARA_WORKERS=2

Distribute Work Across Hosts

Optionally, Yara rule processing may be spread across more than one host. Simply instantiate more instances of Yara Grid on the new hosts with the same axon and swarm settings, and jobs will be executed on the new instances. These additional Yara Grid instances should not be added to the Cortex as a service.

Deploying mirrors of Synapse Yara Grid can also be used to distribute across hosts if the workers configuration option is intended to be identical.

Deploy a Mirror

Inside the AHA container

Generate a one-time use URL for provisioning from inside the AHA container:

python -m synapse.tools.aha.provision.service 01.yara --mirror yara

You should see output that looks similar to this:

one-time use URL: ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

On the Host

Create the container storage directory:

mkdir -p /srv/syn/01.yara/storage
chown -R 999 /srv/syn/01.yara/storage

Create the /srv/syn/01.yara/docker-compose.yaml file with contents:

version: "3.3"
services:
  01.yara:
    user: "999"
    image: vertexproject/synapse-yara:v4.x.x
    network_mode: host
    restart: unless-stopped
    volumes:
        - ./storage:/vertex/storage
    environment:
        # disable HTTPS API for now to prevent port collisions
        - SYN_YARA_HTTPS_PORT=null
        - SYN_YARA_AHA_PROVISION=ssl://aha.<yournetwork>:27272/<guid>?certhash=<sha256>

Note

Don’t forget to replace your one-time use provisioning URL!

Start the container:

docker-compose --file /srv/syn/01.yara/docker-compose.yaml pull
docker-compose --file /srv/syn/01.yara/docker-compose.yaml up -d

Devops Details

Data Processing and Caching

A Yara Grid work unit consists of all the yara rules which are going to be matched against a given file. The Yara Grid worker processes cache the compiled Yara rules based on their full text, in order to speed up the pre-computation time for matching the same ruleset against multiple files. In order to do rule evaluation, the file has to be retrieved from the Axon which is backing the Yara Grid service. This is retrieved once per work unit, and the file contents are not cached. For example, a job with ten rules against 5 files will cause the 5 files to each be retrieved once; but ten jobs with one rule apiece against 5 files will cause each file to be retrieved ten times.

Docker Images

The Synapse-Yara service is available as a Docker container from Docker Hub. The repository can be found at:

https://hub.docker.com/repository/docker/vertexproject/synapse-yara/

Note

There are tagged images available on Docker Hub which correspond to software releases seen in the changelog. The docker tag master is the latest development release. A generic major version tag is available, representing the latest release on a given major version. For example, the v4.x.x tag represents the most current release for the v4.x.x release line. You can utilize specific tagged versions, or a major version specifier, depending on your chosen deployment strategy.

Configuration Options

The following is a list of available configuration options.

aha:admin

An AHA client certificate CN to register as a local admin user.

Type: string
Environment Variable: SYN_YARA_AHA_ADMIN

aha:leader

The AHA service name to claim as the active instance of a storm service.

Type: string
Environment Variable: SYN_YARA_AHA_LEADER

aha:name

The name of the cell service in the aha service registry.

Type: string
Environment Variable: SYN_YARA_AHA_NAME

aha:network

The AHA service network. This makes aha:name/aha:leader relative names.

Type: string
Environment Variable: SYN_YARA_AHA_NETWORK

aha:provision

The telepath URL of the aha provisioning service.

Type: ['string', 'array']
Environment Variable: SYN_YARA_AHA_PROVISION

aha:registry

The telepath URL of the aha service registry.

Type: ['string', 'array']
Environment Variable: SYN_YARA_AHA_REGISTRY

aha:user

The username of this service when connecting to others.

Type: string
Environment Variable: SYN_YARA_AHA_USER

auth:anon

Allow anonymous telepath access by mapping to the given user name.

Type: string
Environment Variable: SYN_YARA_AUTH_ANON

auth:passwd

Set to <passwd> (local only) to bootstrap the root user password.

Type: string
Environment Variable: SYN_YARA_AUTH_PASSWD

axon

The telepath URL for an axon blob store.

Type: string
Environment Variable: SYN_YARA_AXON

backup:dir

A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.

Type: string
Environment Variable: SYN_YARA_BACKUP_DIR

dmon:listen

A config-driven way to specify the telepath bind URL.

Type: ['string', 'null']
Environment Variable: SYN_YARA_DMON_LISTEN

https:headers

Headers to add to all HTTPS server responses.

Type: object
Environment Variable: SYN_YARA_HTTPS_HEADERS

https:parse:proxy:remoteip

Enable the HTTPS server to parse X-Forwarded-For and X-Real-IP headers to determine requester IP addresses.

Type: boolean
Default Value: False
Environment Variable: SYN_YARA_HTTPS_PARSE_PROXY_REMOTEIP

https:port

A config-driven way to specify the HTTPS port.

Type: ['integer', 'null']
Environment Variable: SYN_YARA_HTTPS_PORT

limit:disk:free

Minimum disk free space percentage before setting the cell read-only.

Type: ['integer', 'null']
Default Value: 5
Environment Variable: SYN_YARA_LIMIT_DISK_FREE

max:users

Maximum number of users allowed on system, not including root or locked/archived users (0 is no limit).

Type: integer
Default Value: 0
Environment Variable: SYN_YARA_MAX_USERS

nexslog:en

Record all changes to a stream file on disk. Required for mirroring (on both sides).

Type: boolean
Default Value: False
Environment Variable: SYN_YARA_NEXSLOG_EN

onboot:optimize

Delay startup to optimize LMDB databases during boot to recover free space and increase performance. This may take a while.

Type: boolean
Default Value: False
Environment Variable: SYN_YARA_ONBOOT_OPTIMIZE

swarm

A telepath URL of a swarm system to consume work from.

Type: string
Environment Variable: SYN_YARA_SWARM

workers

The number of workers to spin up.

Type: number
Default Value: 8
Environment Variable: SYN_YARA_WORKERS