Scale
Multi-cluster routing, log scale-out, node-status offloading, cross-cluster failover.
Last updated
Multi-cluster routing, log scale-out, node-status offloading, cross-cluster failover.
Argo Workflows scales fine inside one cluster until you hit one of the usual ceilings: the etcd size limit for large workflow statuses, log volume that overwhelms in-cluster storage, or the operational cost of running many independent Argo deployments side by side. Pipekit handles the multi-cluster shape and the in-cluster ceilings so platform teams stop building these in-house.
Register as many Clusters as you want with one Pipekit Organization. The Pipekit Agent runs in each cluster and reports back to a single control plane, so your Pipes, Runs, and Templates are visible across clusters from one UI.
Pipekit lets you set per-cluster workflow priorities and queue caps from the UI. This is useful when you want a dev cluster to back off during a prod traffic spike. See Clusters > Queuing.
In-cluster log capture works for small workflows; at high volume, Argo's wait-container log path drops lines and overwhelms the API server. Pipekit's Vector integration intercepts logs at the node level and ships them to your storage of choice (S3, GCS, etc.) before they reach the wait container. See Operating at Scale > Vector Log Collection.
Argo stores each Workflow's full node status inside the Workflow resource in etcd. Workflows with many nodes (large fan-outs, deep DAGs) eventually exceed etcd's 1 MB object size and fail to update. Pipekit's node status offloading moves the status off etcd into an external store so very large workflows stop hitting the ceiling.
If a cluster goes down, you can fail Pipes over to another registered Cluster without rewriting Pipe definitions. The DR runbook covers the toggle, the data-plane considerations, and the rollback path. See Clusters > Disaster Recovery.
A Template lives in Git and Pipekit propagates it to every cluster it's registered on. You define the workflow component once and have it usable from any Pipe on any cluster, without copying YAML between clusters.
The Infrastructure as Code with OpenTofu use case demonstrates this shape across three Terraform paths in parallel, with mutual exclusion at the engine level and an apply Template that propagates across environments.
Last updated