Skip to main content

Kubernetes Executor

The Kubernetes executor runs CWL container tools as Kubernetes Pods instead of local Docker containers. It is designed for production deployments where the yardrail-worker runs inside a Kubernetes cluster and workflow tool execution needs to be isolated, scalable, and backed by S3-compatible storage.

When to use it​

ScenarioExecutor
Local development, Docker Composedocker (default)
Production Kubernetes deploymentkubernetes

Set --executor=kubernetes on the worker (or EXECUTOR=kubernetes env var, or worker.executor: kubernetes in the Helm values).

Service architecture​

Execution flow​

A single CWL tool invocation follows this sequence:

CSI volume mapping​

Each invocation generates a UUID and creates three S3 prefixes. The CSI driver mounts them into the Pod as inline volumes:

S3 prefixContainer mountPurpose
workflows/{uuid}/out/Output directoryTool writes output files here
workflows/{uuid}/tmp//tmpScratch space
workflows/{uuid}/stg//var/lib/cwlStaged input files

The worker writes staged inputs to the stg/ prefix before creating the Pod. The CSI driver makes them visible inside the container at the mount path. Outputs written by the tool land directly in S3 — no extraction step is needed after the container exits.

Stdout and stderr capture​

The worker attaches to the running Pod via the Kubernetes SPDY attach API, which provides separate stdout and stderr streams (unlike Docker's multiplexed stream protocol). Captured bytes are buffered in memory and uploaded to the S3 output prefix after the container exits.

Stdin, when specified by the CWL tool, is read from the appropriate S3 filesystem (staging, output, or tmp — whichever the path resolves to) and piped through the same SPDY connection.

RBAC requirements​

The worker's ServiceAccount needs permission to manage Pods in the workflow namespace:

rules:
- apiGroups: [""]
resources: [pods]
verbs: [create, get, list, watch, delete]
- apiGroups: [""]
resources: [pods/attach]
verbs: [create, get]

The Helm chart creates this Role and RoleBinding automatically when worker.executor is set to kubernetes. If the workflow namespace differs from the worker's namespace, cross-namespace RBAC must be configured by the operator.

Configuration​

Worker flags​

FlagEnv varDefaultDescription
--executorEXECUTORdockerdocker or kubernetes
--workflow-namespaceWORKFLOW_NAMESPACEworker's namespaceK8s namespace for workflow Pods
--csi-driver-nameCSI_DRIVER_NAME(required)CSI driver for S3 volumes
--csi-volume-attrsCSI_VOLUME_ATTRS—Extra CSI volumeAttributes (JSON)
--image-pull-secretsIMAGE_PULL_SECRETS—Comma-separated Secret names

Helm values​

worker:
executor: kubernetes
workflowNamespace: "" # empty = worker's own namespace
csiDriverName: s3.csi.aws.amazon.com
workflowImagePullSecrets: []
s3:
bucket: yardrail-workflows

Network isolation​

The executor does not manage network policies. If CWL tools should not have network access (the CWL spec's NetworkAccess requirement), enforce this with a cluster-level NetworkPolicy that targets Pods with the app.kubernetes.io/component: cwl-tool label.

Image pulling​

EnsureImage is a no-op — the kubelet handles image pulls when the Pod is scheduled. For private registries, configure --image-pull-secrets with the names of Kubernetes Secrets containing registry credentials. These are set as imagePullSecrets on the Pod spec.