Ctrlk
PipekitPricingBlogPipekit StatusRelease Notes
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Upgrading to the Operator-Managed Agent

Upgrade the Pipekit Agent from the 6.x chart to the 7.x operator-managed model, including prerequisites, RBAC review, validation, and rollback.

Chart 7.0.0 changed how the Pipekit Agent is deployed. The chart no longer creates the agent's workloads directly. Instead it installs a small operator, a custom resource definition, and an Installation custom resource, and the operator creates and continuously reconciles everything else.

This page covers the upgrade from a 6.x chart to 7.x. If you are installing the agent for the first time, follow Pipekit Agent Helm Chart instead: a fresh 7.x install needs no migration steps.

What changes

Under 6.x, the chart templated the agent Deployment, ServiceAccount, RBAC, ConfigMap, Service, and ServiceMonitor, and pulled in the upstream Vector chart as a subchart.

Under 7.x, the chart installs:

  • the installations.agent.pipekit.io CRD, from the chart's crds/ directory

  • the agent operator, as a Deployment named agent-controller-manager

  • one Installation custom resource, named after your Helm release. A release named pipekit-agent produces Installation/pipekit-agent; any other release name produces Installation/<release>-pipekit-agent

The operator reads that custom resource and creates the agent Deployment, ServiceAccount, RBAC, ConfigMap, Service, and ServiceMonitor, plus the metrics collector and, when you enable it, the Vector DaemonSet. It keeps reconciling them, so manual edits to those resources are reverted. From chart 7.5.0 the metrics collector always runs, so expect a <release-name>-collector Deployment even if you never turned metrics on.

Your values file drives the same behavior it always did. The chart translates it into the Installation resource rather than into workloads. The agent's resource names are derived from the Installation name, which is your release's full name, so a standard install keeps the names it had under 6.x.

Labels do change. The operator labels what it creates with app.kubernetes.io/managed-by: pipekit-agent-operator instead of Helm, and the agent's pods carry app.kubernetes.io/component: agent and app.kubernetes.io/instance: <installation-name> in place of app.kubernetes.io/name: pipekit-agent. The Installation name is your release's full name, so for a release named pipekit-agent the instance label is pipekit-agent. The Deployment keeps its app.kubernetes.io/name label, but its pods do not. Update anything that selects the agent's pods by label, such as a NetworkPolicy, a dashboard query, or a kubectl logs -l command.

The Installation custom resource is cluster-scoped, and the operator reconciles every Installation in the cluster, so one operator can manage more than one agent. Refer to Namespace-Scoped Agents.

Chart 7.x deploys one agent per release, and a second release of the chart in the same cluster fails because the operator's ClusterRoles use fixed names. If you run more than one pipekit-agent release on a cluster, contact us before upgrading.

Before you begin

Work through these before you schedule the upgrade.

Check how the agent was installed. This page covers a Helm release of the pipekit-agent chart. If you deployed the agent with pipekit connect cluster, that is a normal Helm release and these steps apply, but rerunning the command is not a shortcut past them. The CLI upgrades an existing release through Helm's upgrade path, so it does not install the CRD either.

Plan to install the CRD yourself. Helm applies the chart's crds/ directory on helm install only. It does not apply CRDs when upgrading an existing release, even with --install, so a 6.x to 7.x upgrade never installs the Installation CRD for you. This is Helm's own behavior, not a Pipekit design decision: Helm has no support for upgrading CRDs. Apply it before you upgrade, as described in step 2, or step 3 fails with no matches for kind "Installation" in version "agent.pipekit.io/v1alpha1".

From chart 7.6.0, the chart closes that gap for later upgrades with a Job that applies its own copy of the CRD before every install and upgrade, described in Updating the Installation CRD. That Job does not cover the 6.x to 7.x move. Helm resolves every kind in the release manifest before it runs any hook, so with the CRD missing the upgrade fails before the Job starts.

GitOps tooling behaves differently, and the two tools differ from each other. Argo CD applies the chart's CRDs on every sync, so step 2 is already covered unless the Application's Helm source sets skipCrds: true. Flux defaults spec.upgrade.crds to Skip, so an existing HelmRelease does not pick up the CRD when you move it to 7.x: set spec.upgrade.crds to CreateReplace, or apply the CRD yourself. With the CRD missing, the sync succeeds but nothing reconciles, because the Installation resource has no definition.

Clear the operator's RBAC with your security team. The operator creates the agent's Role, RoleBinding, ClusterRole, and ClusterRoleBinding on your behalf, so its own ClusterRole includes the bind and escalate verbs on all four of those resource types, cluster-scoped ones included. Policy engines such as Kyverno, OPA Gatekeeper, and Azure Policy commonly flag escalate. Confirm your policies permit it, or add an exception, before the upgrade window.

From chart 7.6.0, the CRD apply Job also needs a short-lived ServiceAccount, ClusterRole, and ClusterRoleBinding with create on customresourcedefinitions, which Helm deletes once the Job succeeds. Set crds.applyJob.enabled to false if your cluster does not allow the chart to create cluster-scoped roles, and apply the CRD yourself on each upgrade. Refer to Updating the Installation CRD for the full rule set.

Check your Vector configuration. Vector is no longer an upstream subchart, so any vector.* key that belonged to that subchart is ignored. Refer to Vector users below.

Pick a maintenance window. Expect a brief agent restart. Refer to Expected disruption.

Values that changed

Most keys carry over untouched, and no secrets.* key was removed or renamed. Four keys were removed and are silently ignored if left in your values file:

Removed key
What to do

configMap.name

Nothing. The operator names the ConfigMap <release-name>-config

vector.existingConfigMaps

Nothing. The operator builds and owns Vector's ConfigMap

vector.role

Nothing. Vector always runs as a DaemonSet

vector.service

Nothing. No Vector Service is created

Any other vector.* key you set for the upstream Vector chart, such as vector.customConfig or vector.podLabels, is no longer read. The chart honors only the vector.* keys documented in Pipekit Agent Helm Chart.

One key was renamed. From chart 7.5.0, features.metrics.enabled is replaced by features.metrics.scrapeCluster, which defaults to true. The metrics collector now always runs, and scrapeCluster controls whether it reads cluster-wide telemetry from nodes, pods, and the Kubernetes API server. An explicit features.metrics.enabled: false in your values file still maps to scrapeCluster: false, so an existing opt-out survives the upgrade. If you never set features.metrics.enabled, the upgrade turns cluster scraping on, because the 6.x default was off and the 7.x default is on. Set features.metrics.scrapeCluster to false if you do not want that. Setting features.workflows.namespace forces it off, because a namespace-scoped agent asked not to hold cluster-wide access.

New keys you can now set: deployment.affinity, deployment.tolerations, deployment.extraEnvFrom, vector.nodeSelector, vector.tolerations, vector.affinity, features.metrics.pipekitApiUrl, features.metrics.metricsCollectorUrl, features.metrics.scrapeCluster, features.metrics.insecureK8s, scheduling.*, operator.*, and crds.applyJob.*.

One place to set where the pods run

The chart has a top-level scheduling block, with nodeSelector, tolerations, and affinity. Set any of these values in scheduling and it will be applied to all appropriate pods controlled by the chart and operator. Those pods are the operator, the CRD apply Job, the Pipekit Agent, the metrics collector, and Vector.

Each pod also keeps a section of its own, and the value there replaces the scheduling one for that pod:

Pod
Keys

Operator

operator.nodeSelector, operator.tolerations, operator.affinity

CRD apply Job

crds.applyJob.nodeSelector, crds.applyJob.tolerations, crds.applyJob.affinity

Pipekit Agent

deployment.nodeSelector, deployment.tolerations, deployment.affinity

Metrics collector

features.metrics.collectorNodeSelector, features.metrics.collectorTolerations, features.metrics.collectorAffinity

Vector

vector.nodeSelector, vector.tolerations, vector.affinity

Vector uses scheduling.tolerations only. A node selector or an affinity rule would stop the DaemonSet collecting logs on the nodes it skips. Set vector.nodeSelector or vector.affinity when you do want to restrict Vector, and expect to lose the logs of every workflow pod on the nodes you exclude.

The kubernetes.io/os: linux default moved from deployment.nodeSelector to scheduling.nodeSelector, so every pod except Vector now gets it. A per-pod key replaces the whole scheduling value rather than merging with it. Setting deployment.nodeSelector to nodegroup: my-nodes gives the agent pod that one label, where chart 8.0 and earlier gave it nodegroup: my-nodes and kubernetes.io/os: linux together. Add kubernetes.io/os: linux to your own block if you still want it.

Your existing values file keeps working. While you leave both scheduling and the pod's own section unset, the operator pod and the CRD apply Job still read deployment.nodeSelector, deployment.tolerations, and deployment.affinity, so an upgrade needs no change from you. A later release removes that one fallback. Move those settings into scheduling, operator, or crds.applyJob when you next edit your values file. The per-pod sections themselves are permanent.

scheduling is a new field on the Installation custom resource, so the CRD apply Job has to run before the operator can read it. The Job runs by default, and Argo CD applies the CRD on every sync. If you set crds.applyJob.enabled to false and do not use Argo CD, apply the new CRD yourself first. Otherwise the API server drops spec.scheduling from the resource, helm upgrade still reports success, and the agent, the collector, and Vector run without your settings. The operator pod and the CRD apply Job read their values from Helm, so those two are placed either way.

Upgrade steps

These commands assume a release named pipekit-agent in the argo namespace. Substitute your own release name and namespace throughout.

1. Record your current state

Capture your live values and a record of the resources the 6.x chart owns:

Keep pipekit-agent-6x-values.yaml, because you need it to roll back. Use -o yaml, because the default output adds a USER-SUPPLIED VALUES: heading that does not belong in a values file.

After the upgrade, list the same resources by the operator's label instead, since the agent's ConfigMap no longer carries app.kubernetes.io/name: pipekit-agent:

2. Install the CRD

Do this if you upgrade with Helm directly or with Flux. Argo CD applies the chart's CRDs on sync, so you can go straight to step 3 unless the Application sets skipCrds: true. Helm does not apply crds/ when upgrading an existing release, so the CRD has to be in place before step 3. Check the latest 7.x version with helm search repo pipekit/pipekit-agent --versions and use it in place of 7.6.0 below:

The CRD is cluster-scoped, so this is safe to run again if it already exists.

3. Upgrade the release

Confirm step 2 is done before you run this. Pin the version explicitly, because the 6.x line is still published and an unpinned upgrade may not move you where you expect:

Helm removes the workloads the old chart owned, and the operator recreates them from the Installation resource under its own ownership. No manual intervention is required between those two phases. From chart 7.6.0, Helm also runs the CRD apply Job before the upgrade. It re-applies the CRD you installed in step 2, which is harmless, and pulls the operator image at the chart's appVersion to do so.

4. Validate

Work through this in order. The Installation resource has no status conditions, so verify the operator and the resources it produced rather than the custom resource itself.

Check
Command
Expect

CRD present

kubectl get crd installations.agent.pipekit.io

The CRD exists

Installation created

kubectl get installation

One resource, named after your release

Operator running

kubectl -n argo get deploy agent-controller-manager

1/1 available

Operator reconciled cleanly

kubectl -n argo logs deploy/agent-controller-manager

No repeated errors, no forbidden

Agent running

kubectl -n argo get deploy pipekit-agent

Available, same name as before

Agent healthy

kubectl -n argo logs deploy/pipekit-agent

No forbidden errors listing workflows

Cluster connected

pipekit list clusters, or the Clusters view in the Pipekit UI

[ACTIVE] is true and [AGENT VERSION] shows the new version

Metrics collector

kubectl -n argo get deploy pipekit-agent-collector

Available

Vector, if enabled

kubectl -n argo get ds pipekit-vector

Ready on every node

Then submit a workflow and confirm the run and its logs appear in Pipekit.

Vector users

Two behaviors changed.

Vector is built by the operator. The upstream Vector chart is no longer a dependency. The operator builds the DaemonSet, its ConfigMap, and its RBAC, and names the DaemonSet pipekit-vector. Under 6.x the DaemonSet was named after your release, so any automation or dashboard that targets the old name needs updating.

Credentials are inherited automatically. The Vector ConfigMap now reads ${PIPEKIT_SECRET_ACCESS_KEY} from the agent's Secret instead of having the key inlined at template time. When you leave vector.envFrom empty, the operator mounts the agent's Secret for you, whether that Secret came from secrets.existingSecret or was created by the chart. You no longer need to repeat the secret reference under vector.envFrom, though an explicit entry still overrides the default if you want Vector to read a different Secret.

Refer to Vector Log Collection for the full picture.

Expected disruption

Component
What happens

Agent pod

Brief gap. Helm removes the old Deployment and the operator recreates it

Metrics collector

Brief gap if it was running. Created if it was not, from chart 7.5.0

Vector

Brief gap per node, if Vector is enabled

Running Argo Workflows are not interrupted by the agent restarting, but Pipekit does not track state while the agent is down. Upgrade outside a critical workflow window.

Troubleshooting

Symptom
Cause
Fix

no matches for kind "Installation" in version "agent.pipekit.io/v1alpha1"

The CRD is not installed. Helm does not apply crds/ on upgrade

Run step 2, then repeat step 3

exists and cannot be imported into the current release

Helm was asked to take over a resource the operator owns

Delete the Installation resource, wait for its resources to go, then rerun the release

no Deployment with the name "pipekit-agent" found

Helm was asked to take over a resource the operator owns

Delete the Installation resource, wait for its resources to go, then rerun the release

Operator logs show attempting to grant RBAC permissions not currently held

The operator's escalate permission is missing or blocked by admission policy

Confirm the operator's ClusterRole applied cleanly and that your policy engine permits escalate

Operator logs show forbidden

Admission policy is blocking a resource the operator creates

Review the policy that matches the named resource

Operator is available but no agent Deployment appears

features.workflows.enabled is false in your values

Set it to true and upgrade again

Agent runs but the cluster shows as disconnected

The agent's Secret did not carry over

Check secrets.existingSecret still names a Secret in the namespace

Collect the operator's logs before changing anything:

Rolling back

Remove the operator's resources before you reinstall the 6.x chart. Helm refuses to adopt a resource that exists in the cluster but was absent from the release it is upgrading from, so rolling straight back fails with Deployment "pipekit-agent" in namespace "argo" exists and cannot be imported into the current release. Deleting the Installation resource lets Kubernetes garbage-collect everything the operator owns:

Wait until that second command returns nothing before you continue.

Then reinstall the 6.x chart with the values you captured in step 1:

The chart takes ownership of the agent workloads again and recreates them under its own names and labels. The CRD is cluster-scoped and is not removed by the rollback. Leave it in place if you plan to retry, or remove it once no installations remain:

Deleting the CRD deletes every Installation resource in the cluster, and Kubernetes garbage-collects the resources those installations own. Confirm the 6.x chart has recreated the agent before you remove the CRD.

Getting help

If the operator does not reconcile, collect kubectl -n argo logs deploy/agent-controller-manager and kubectl get installation -o yaml, then contact us over Slack, or by email at support@pipekit.io.

Last updated