Node Status Offloading

Why offloading exists

Argo Workflows stores the status of every node in a workflow inside the Workflow custom resource. For workflows with thousands of nodes, or workflows whose nodes carry large outputs, that status payload can exceed the 1 MiB per-object size limit. Argo enforces this limit to stay within etcd's request limit. When the payload exceeds it, Argo first compresses node status into the CR. If the compressed payload still does not fit and you have not configured offloading, the workflow may be stuck its previously 1 MiB compliant state.

Argo's solution is node status offloading. When the node status would grow too large, Argo writes it to an external SQL database (PostgreSQL or MySQL) and keeps only a pointer in the Workflow CR. This lifts the etcd ceiling and is required for very large workflows.

For the upstream rationale and Argo-side configuration, see Offloading large workflows in the Argo Workflows documentation.

Why the Pipekit Agent needs to know

The Pipekit Agent watches Argo workflows and forwards their state to Pipekit so runs are visible in the UI. When offloading is enabled, the Workflow CR no longer contains full node status. The node status field holds only a pointer to the offloading database. An agent that does not know about offloading reads only the pointer, not the actual node-level data. Your Pipekit UI then shows incomplete or empty workflow graphs for any offloaded workflow.

Configure features.workflows.nodeOffloading on the agent with credentials for the same database Argo offloads to. The agent then reads node status directly when the CR is incomplete.

Prerequisites

Configure Argo Workflows for offloading first, then point the agent at the same database. The agent reads from the offloading database; it does not create the schema or turn offloading on for you.

Follow the upstream Argo offloading guide to enable offloading on your Argo Workflows installation.

Enabling offloading on the agent

Set features.workflows.nodeOffloading.enabled=true and provide the database connection details. A minimal Helm install for PostgreSQL with inline credentials:

helm upgrade -i -n argo \
  pipekit-agent pipekit/pipekit-agent \
  --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
  --set secrets.pipekitClusterId="[provided Cluster ID]" \
  --set features.workflows.nodeOffloading.enabled=true \
  --set features.workflows.nodeOffloading.argoDB.driver=postgres \
  --set features.workflows.nodeOffloading.argoDB.host=argo-db.example.com \
  --set features.workflows.nodeOffloading.argoDB.port=5432 \
  --set features.workflows.nodeOffloading.argoDB.database=argo \
  --set features.workflows.nodeOffloading.argoDB.secret.argoDBUsername=argo \
  --set features.workflows.nodeOffloading.argoDB.secret.argoDBPassword='[password]'

For real deployments, use a values file rather than a long chain of --set flags.

Table names must match Argo

The agent reads from two tables: argo_workflows (offloaded node status) and argo_archived_workflows (archived workflows). These names match Argo Workflows' defaults, and you should not need to change them. If you have customized the table names in your Argo Workflows configuration, set the same values on the agent:

features:
  workflows:
    nodeOffloading:
      enabled: true
      argoDB:
        offloadedNodesTableName: argo_workflows
        archiveTableName: argo_archived_workflows

A mismatch here causes the agent to log database errors and serve workflows with empty node status until you correct the names.

Database driver

The agent supports both PostgreSQL and MySQL. Set features.workflows.nodeOffloading.argoDB.driver to postgres or mysql. The default port is 5432. If you set driver: mysql, set port explicitly to your MySQL port (typically 3306). The chart does not adjust the default port when the driver changes. The driver must match the database Argo Workflows is offloading to.

Secrets: inline vs. existing

Inline credentials are suitable for development:

features:
  workflows:
    nodeOffloading:
      enabled: true
      argoDB:
        host: argo-db.example.com
        database: argo
        secret:
          argoDBUsername: argo
          argoDBPassword: '[password]'

For production, reference an existing Secret:

features:
  workflows:
    nodeOffloading:
      enabled: true
      argoDB:
        host: argo-db.example.com
        database: argo
        secret:
          existingSecret: argo-db-credentials

The referenced Secret must contain the data keys ARGO_DB_USERNAME and ARGO_DB_PASSWORD:

kubectl -n argo create secret generic argo-db-credentials \
    --from-literal=ARGO_DB_USERNAME='argo' \
    --from-literal=ARGO_DB_PASSWORD='[password]'

Use single quotes to escape special characters such as $, \, *, and ! in your password. Without them, your shell interprets these characters before kubectl creates the secret.

TLS

For PostgreSQL connections, set features.workflows.nodeOffloading.argoDB.sslMode to one of disable, allow, prefer, require, verify-ca, or verify-full. The default is disable, which is only appropriate for development. For production, use require at minimum, and verify-full if you have CA certificates available.

The agent ignores this setting for MySQL.

Connection pool tuning

The defaults are reasonable for moderate load:

Setting	Default	When to change
maxOpenConnections	30	Increase if the agent serves many concurrent UI requests against offloaded workflows.
maxIdleConnections	15	Keep at roughly half of maxOpenConnections.
maxConnectionLifetime	5 (minutes)	Shorter values help when sitting behind connection-pool middleware such as PgBouncer.
maxRetries	3	Raise if your database has frequent transient failures.

Verifying it's working

1. Check the agent logs at startup for a successful database connection. Errors at this stage usually indicate wrong credentials, wrong host or port, or sslMode mismatch with the database server.

2. Run a workflow large enough to trigger offloading on Argo's side, or pick an existing offloaded workflow.

3. Open the run in the Pipekit UI and confirm the run graph is fully populated rather than showing empty or stub nodes.

Reference

For the full set of features.workflows.nodeOffloading values, see the values table in Pipekit Agent Helm Chart.

Upstream documentation: Argo Workflows — Offloading large workflows.