Docker Executor
The Docker executor is the default executor for the yardrail-worker. It runs CWL container tools as Docker containers on the same Docker daemon the worker has access to, using S3 for artifact storage and tar-streaming for container I/O.
When to use it
The Docker executor is designed for local development and non-Kubernetes deployments where the worker runs on a host with Docker installed. For production Kubernetes deployments, use the Kubernetes executor instead.
| Scenario | Executor |
|---|---|
| Local development (native or Docker Compose with socket mount) | docker |
| Non-Kubernetes server with Docker installed | docker |
| Production Kubernetes deployment | kubernetes |
The Docker executor requires access to a Docker daemon socket. When the worker runs inside a container (Docker Compose or Kubernetes), the host's Docker socket must be bind-mounted into the worker container. The default Docker Compose and Helm configurations do not do this automatically — see deployment notes below.
Service architecture
Execution flow
How tar-streaming works
Unlike the Kubernetes executor which uses CSI volumes for direct S3 access, the Docker executor
transfers files between S3 and the container using Docker's CopyToContainer and
CopyFromContainer APIs, which speak the tar archive format.
Input injection
Before the container starts, the executor:
- Reads all objects from the three S3 working directories (out, tmp, staging)
- Builds a single tar archive with entries rooted at their container mount target paths
- Streams the tar into the stopped container at
/viaCopyToContainer
The container's filesystem then has the staged files at the correct paths when it starts.
Output extraction
After the container exits:
CopyFromContainerretrieves the output directory as a tar stream- Each tar entry is parsed and written to the S3 output filesystem
- The leading directory component is stripped (Docker prepends the source directory name)
Stdout and stderr
The executor attaches to the container before starting it. Docker's multiplexed stream protocol
interleaves stdout and stderr in a single connection with 8-byte frame headers
([stream_type:1][padding:3][size:4 big-endian][payload]). The executor demuxes the frames and
buffers stdout/stderr separately in memory, then uploads both to the S3 output filesystem after
the container exits.
Configuration
| Flag | Env var | Default | Description |
|---|---|---|---|
--executor | EXECUTOR | docker | Set to docker (or omit — it's the default) |
--s3-endpoint | S3_ENDPOINT | (required) | S3-compatible endpoint |
--s3-access-key | S3_ACCESS_KEY | — | S3 access key |
--s3-secret-key | S3_SECRET_KEY | — | S3 secret key |
--s3-bucket | S3_BUCKET | yardrail-workflows | Bucket for workflow artifacts |
--s3-use-ssl | S3_USE_SSL | false | TLS for S3 connection |
The Docker daemon address is read from the DOCKER_HOST environment variable, defaulting to
unix:///var/run/docker.sock.
Deployment considerations
Docker socket access
The Docker executor uses the Docker Engine SDK, which connects to the daemon via a Unix socket
(or TCP if DOCKER_HOST is set). The worker process must have access to this socket.
- Running natively on the host: works out of the box — the socket is at
/var/run/docker.sock. - Running in Docker Compose: add a volume mount for the socket in your override file:
services:yardrail-worker:volumes:- /var/run/docker.sock:/var/run/docker.sock
- Running in Kubernetes: requires mounting the host's Docker socket into the worker Pod, which is a security concern and not recommended. Use the Kubernetes executor instead.
Network isolation
When a CWL tool specifies NetworkAccess: false, the Docker executor creates the container with
NetworkMode: "none", fully isolating it from all networks.
Image management
The Docker executor actively manages images — EnsureImage checks whether the image is present
on the daemon and pulls it if not. It caches image presence in memory to avoid repeated API
calls. DockerFile, DockerLoad, and DockerImport are not supported.