Skip to main content

Docker Executor

The Docker executor is the default executor for the yardrail-worker. It runs CWL container tools as Docker containers on the same Docker daemon the worker has access to, using S3 for artifact storage and tar-streaming for container I/O.

When to use it​

The Docker executor is designed for local development and non-Kubernetes deployments where the worker runs on a host with Docker installed. For production Kubernetes deployments, use the Kubernetes executor instead.

ScenarioExecutor
Local development (native or Docker Compose with socket mount)docker
Non-Kubernetes server with Docker installeddocker
Production Kubernetes deploymentkubernetes
info

The Docker executor requires access to a Docker daemon socket. When the worker runs inside a container (Docker Compose or Kubernetes), the host's Docker socket must be bind-mounted into the worker container. The default Docker Compose and Helm configurations do not do this automatically — see deployment notes below.

Service architecture​

Execution flow​

How tar-streaming works​

Unlike the Kubernetes executor which uses CSI volumes for direct S3 access, the Docker executor transfers files between S3 and the container using Docker's CopyToContainer and CopyFromContainer APIs, which speak the tar archive format.

Input injection​

Before the container starts, the executor:

  1. Reads all objects from the three S3 working directories (out, tmp, staging)
  2. Builds a single tar archive with entries rooted at their container mount target paths
  3. Streams the tar into the stopped container at / via CopyToContainer

The container's filesystem then has the staged files at the correct paths when it starts.

Output extraction​

After the container exits:

  1. CopyFromContainer retrieves the output directory as a tar stream
  2. Each tar entry is parsed and written to the S3 output filesystem
  3. The leading directory component is stripped (Docker prepends the source directory name)

Stdout and stderr​

The executor attaches to the container before starting it. Docker's multiplexed stream protocol interleaves stdout and stderr in a single connection with 8-byte frame headers ([stream_type:1][padding:3][size:4 big-endian][payload]). The executor demuxes the frames and buffers stdout/stderr separately in memory, then uploads both to the S3 output filesystem after the container exits.

Configuration​

FlagEnv varDefaultDescription
--executorEXECUTORdockerSet to docker (or omit — it's the default)
--s3-endpointS3_ENDPOINT(required)S3-compatible endpoint
--s3-access-keyS3_ACCESS_KEY—S3 access key
--s3-secret-keyS3_SECRET_KEY—S3 secret key
--s3-bucketS3_BUCKETyardrail-workflowsBucket for workflow artifacts
--s3-use-sslS3_USE_SSLfalseTLS for S3 connection

The Docker daemon address is read from the DOCKER_HOST environment variable, defaulting to unix:///var/run/docker.sock.

Deployment considerations​

Docker socket access​

The Docker executor uses the Docker Engine SDK, which connects to the daemon via a Unix socket (or TCP if DOCKER_HOST is set). The worker process must have access to this socket.

  • Running natively on the host: works out of the box — the socket is at /var/run/docker.sock.
  • Running in Docker Compose: add a volume mount for the socket in your override file:
    services:
    yardrail-worker:
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  • Running in Kubernetes: requires mounting the host's Docker socket into the worker Pod, which is a security concern and not recommended. Use the Kubernetes executor instead.

Network isolation​

When a CWL tool specifies NetworkAccess: false, the Docker executor creates the container with NetworkMode: "none", fully isolating it from all networks.

Image management​

The Docker executor actively manages images — EnsureImage checks whether the image is present on the daemon and pulls it if not. It caches image presence in memory to avoid repeated API calls. DockerFile, DockerLoad, and DockerImport are not supported.