> For the complete documentation index, see [llms.txt](https://docs.pipekit.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pipekit.io/scaling/node-status-offloading.md). # Node Status Offloading ## Why offloading exists Argo Workflows stores the status of every node in a workflow inside the Workflow custom resource. For workflows with thousands of nodes, or workflows whose nodes carry large outputs, that status payload can exceed the 1 MiB per-object size limit. Argo enforces this limit to stay within etcd's request limit. When the payload exceeds it, Argo first compresses node status into the CR. If the compressed payload still does not fit and you have not configured offloading, the workflow may be stuck its previously 1 MiB compliant state. Argo's solution is *node status offloading*. When the node status would grow too large, Argo writes it to an external SQL database (PostgreSQL or MySQL) and keeps only a pointer in the Workflow CR. This lifts the etcd ceiling and is required for very large workflows. For the upstream rationale and Argo-side configuration, see [Offloading large workflows](https://argo-workflows.readthedocs.io/en/latest/offloading-large-workflows/) in the Argo Workflows documentation. ## Why the Pipekit Agent needs to know The Pipekit Agent watches Argo workflows and forwards their state to Pipekit so runs are visible in the UI. When offloading is enabled, the Workflow CR no longer contains full node status. The node status field holds only a pointer to the offloading database. An agent that does not know about offloading reads only the pointer, not the actual node-level data. Your Pipekit UI then shows incomplete or empty workflow graphs for any offloaded workflow. Configure `features.workflows.nodeOffloading` on the agent with credentials for the same database Argo offloads to. The agent then reads node status directly when the CR is incomplete. ## Prerequisites {% hint style="info" %} Configure Argo Workflows for offloading first, then point the agent at the same database. The agent reads from the offloading database; it does not create the schema or turn offloading on for you. {% endhint %} Follow the upstream [Argo offloading guide](https://argo-workflows.readthedocs.io/en/latest/offloading-large-workflows/) to enable offloading on your Argo Workflows installation. ## Enabling offloading on the agent Set `features.workflows.nodeOffloading.enabled=true` and provide the database connection details. A minimal Helm install for PostgreSQL with inline credentials: ```bash helm upgrade -i -n argo \ pipekit-agent pipekit/pipekit-agent \ --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \ --set secrets.pipekitClusterId="[provided Cluster ID]" \ --set features.workflows.nodeOffloading.enabled=true \ --set features.workflows.nodeOffloading.argoDB.driver=postgres \ --set features.workflows.nodeOffloading.argoDB.host=argo-db.example.com \ --set features.workflows.nodeOffloading.argoDB.port=5432 \ --set features.workflows.nodeOffloading.argoDB.database=argo \ --set features.workflows.nodeOffloading.argoDB.secret.argoDBUsername=argo \ --set features.workflows.nodeOffloading.argoDB.secret.argoDBPassword='[password]' ``` For real deployments, use a values file rather than a long chain of `--set` flags. ## Table names must match Argo The agent reads from two tables: `argo_workflows` (offloaded node status) and `argo_archived_workflows` (archived workflows). These names match Argo Workflows' defaults, and you should not need to change them. If you have customized the table names in your Argo Workflows configuration, set the same values on the agent: ```yaml features: workflows: nodeOffloading: enabled: true argoDB: offloadedNodesTableName: argo_workflows archiveTableName: argo_archived_workflows ``` {% hint style="info" %} A mismatch here causes the agent to log database errors and serve workflows with empty node status until you correct the names. {% endhint %} ## Database driver The agent supports both PostgreSQL and MySQL. Set `features.workflows.nodeOffloading.argoDB.driver` to `postgres` or `mysql`. The default port is 5432. If you set `driver: mysql`, set `port` explicitly to your MySQL port (typically 3306). The chart does not adjust the default port when the driver changes. The driver must match the database Argo Workflows is offloading to. ## Secrets: inline vs. existing Inline credentials are suitable for development: ```yaml features: workflows: nodeOffloading: enabled: true argoDB: host: argo-db.example.com database: argo secret: argoDBUsername: argo argoDBPassword: '[password]' ``` For production, reference an existing Secret: ```yaml features: workflows: nodeOffloading: enabled: true argoDB: host: argo-db.example.com database: argo secret: existingSecret: argo-db-credentials ``` The referenced Secret must contain the data keys `ARGO_DB_USERNAME` and `ARGO_DB_PASSWORD`: ```bash kubectl -n argo create secret generic argo-db-credentials \ --from-literal=ARGO_DB_USERNAME='argo' \ --from-literal=ARGO_DB_PASSWORD='[password]' ``` Use single quotes to escape special characters such as `$`, `\`, `*`, and `!` in your password. Without them, your shell interprets these characters before kubectl creates the secret. ## TLS For PostgreSQL connections, set `features.workflows.nodeOffloading.argoDB.sslMode` to one of `disable`, `allow`, `prefer`, `require`, `verify-ca`, or `verify-full`. The default is `disable`, which is only appropriate for development. For production, use `require` at minimum, and `verify-full` if you have CA certificates available. The agent ignores this setting for MySQL. ## Connection pool tuning The defaults are reasonable for moderate load: | Setting | Default | When to change | | ----------------------- | ----------- | ------------------------------------------------------------------------------------- | | `maxOpenConnections` | 30 | Increase if the agent serves many concurrent UI requests against offloaded workflows. | | `maxIdleConnections` | 15 | Keep at roughly half of `maxOpenConnections`. | | `maxConnectionLifetime` | 5 (minutes) | Shorter values help when sitting behind connection-pool middleware such as PgBouncer. | | `maxRetries` | 3 | Raise if your database has frequent transient failures. | ## Verifying it's working 1. Check the agent logs at startup for a successful database connection. Errors at this stage usually indicate wrong credentials, wrong host or port, or `sslMode` mismatch with the database server. 2. Run a workflow large enough to trigger offloading on Argo's side, or pick an existing offloaded workflow. 3. Open the run in the Pipekit UI and confirm the run graph is fully populated rather than showing empty or stub nodes. ## Reference For the full set of `features.workflows.nodeOffloading` values, see the values table in [Pipekit Agent Helm Chart](/pipekit-agent/helm-chart.md). Upstream documentation: [Argo Workflows — Offloading large workflows](https://argo-workflows.readthedocs.io/en/latest/offloading-large-workflows/). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.pipekit.io/scaling/node-status-offloading.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.