> For the complete documentation index, see [llms.txt](https://docs.pipekit.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pipekit.io/using-pipekit/clusters/agent-upgrade.md).

# Upgrading to the Operator-Managed Agent

Upgrade the Pipekit Agent from the 6.x chart to the 7.x operator-managed model, including prerequisites, RBAC review, validation, and rollback.

Chart `7.0.0` changed how the Pipekit Agent is deployed. The chart no longer creates the agent's workloads directly. Instead it installs a small operator, a custom resource definition, and an `Installation` custom resource, and the operator creates and continuously reconciles everything else.

This page covers the upgrade from a `6.x` chart to `7.x`. If you are installing the agent for the first time, follow [Pipekit Agent Helm Chart](/reference/helm-values/agent.md) instead: a fresh `7.x` install needs no migration steps.

## What changes

Under `6.x`, the chart templated the agent `Deployment`, `ServiceAccount`, RBAC, `ConfigMap`, `Service`, and `ServiceMonitor`, and pulled in the upstream Vector chart as a subchart.

Under `7.x`, the chart installs:

* the `installations.agent.pipekit.io` CRD, from the chart's `crds/` directory
* the agent operator, as a `Deployment` named `agent-controller-manager`
* one `Installation` custom resource, named after your Helm release. A release named `pipekit-agent` produces `Installation/pipekit-agent`; any other release name produces `Installation/<release>-pipekit-agent`

The operator reads that custom resource and creates the agent `Deployment`, `ServiceAccount`, RBAC, `ConfigMap`, `Service`, and `ServiceMonitor`, plus the metrics collector and, when you enable it, the Vector DaemonSet. It keeps reconciling them, so manual edits to those resources are reverted. From chart `7.5.0` the metrics collector always runs, so expect a `<release-name>-collector` Deployment even if you never turned metrics on.

Your values file drives the same behavior it always did. The chart translates it into the `Installation` resource rather than into workloads. The agent's resource names are derived from the `Installation` name, which is your release's full name, so a standard install keeps the names it had under `6.x`.

Labels do change. The operator labels what it creates with `app.kubernetes.io/managed-by: pipekit-agent-operator` instead of `Helm`, and the agent's pods carry `app.kubernetes.io/component: agent` and `app.kubernetes.io/instance: <installation-name>` in place of `app.kubernetes.io/name: pipekit-agent`. The `Installation` name is your release's full name, so for a release named `pipekit-agent` the instance label is `pipekit-agent`. The Deployment keeps its `app.kubernetes.io/name` label, but its pods do not. Update anything that selects the agent's pods by label, such as a `NetworkPolicy`, a dashboard query, or a `kubectl logs -l` command.

The `Installation` custom resource is cluster-scoped, and the operator reconciles every `Installation` in the cluster, so one operator can manage more than one agent. Refer to [Namespace-Scoped Agents](/using-pipekit/clusters/namespaced-agent.md).

{% hint style="warning" %}
Chart `7.x` deploys one agent per release, and a second release of the chart in the same cluster fails because the operator's ClusterRoles use fixed names. If you run more than one `pipekit-agent` release on a cluster, contact us before upgrading.
{% endhint %}

## Before you begin

Work through these before you schedule the upgrade.

**Check how the agent was installed.** This page covers a Helm release of the `pipekit-agent` chart. If you deployed the agent with `pipekit connect cluster`, that is a normal Helm release and these steps apply, but rerunning the command is not a shortcut past them. The CLI upgrades an existing release through Helm's upgrade path, so it does not install the CRD either.

**Plan to install the CRD yourself.** Helm applies the chart's `crds/` directory on `helm install` only. It does not apply CRDs when upgrading an existing release, even with `--install`, so a `6.x` to `7.x` upgrade never installs the `Installation` CRD for you. This is [Helm's own behavior](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations), not a Pipekit design decision: Helm has no support for upgrading CRDs. Apply it before you upgrade, as described in step 2, or step 3 fails with `no matches for kind "Installation" in version "agent.pipekit.io/v1alpha1"`.

From chart `7.6.0`, the chart closes that gap for later upgrades with a Job that applies its own copy of the CRD before every install and upgrade, described in [Updating the Installation CRD](/reference/helm-values/agent.md#updating-the-installation-crd). That Job does not cover the `6.x` to `7.x` move. Helm resolves every kind in the release manifest before it runs any hook, so with the CRD missing the upgrade fails before the Job starts.

GitOps tooling behaves differently, and the two tools differ from each other. Argo CD applies the chart's CRDs on every sync, so step 2 is already covered unless the Application's Helm source sets `skipCrds: true`. Flux defaults `spec.upgrade.crds` to `Skip`, so an existing `HelmRelease` does not pick up the CRD when you move it to `7.x`: set `spec.upgrade.crds` to `CreateReplace`, or apply the CRD yourself. With the CRD missing, the sync succeeds but nothing reconciles, because the `Installation` resource has no definition.

**Clear the operator's RBAC with your security team.** The operator creates the agent's `Role`, `RoleBinding`, `ClusterRole`, and `ClusterRoleBinding` on your behalf, so its own `ClusterRole` includes the `bind` and `escalate` verbs on all four of those resource types, cluster-scoped ones included. Policy engines such as Kyverno, OPA Gatekeeper, and Azure Policy commonly flag `escalate`. Confirm your policies permit it, or add an exception, before the upgrade window.

From chart `7.6.0`, the CRD apply Job also needs a short-lived `ServiceAccount`, `ClusterRole`, and `ClusterRoleBinding` with `create` on `customresourcedefinitions`, which Helm deletes once the Job succeeds. Set `crds.applyJob.enabled` to `false` if your cluster does not allow the chart to create cluster-scoped roles, and apply the CRD yourself on each upgrade. Refer to [Updating the Installation CRD](/reference/helm-values/agent.md#updating-the-installation-crd) for the full rule set.

**Check your Vector configuration.** Vector is no longer an upstream subchart, so any `vector.*` key that belonged to that subchart is ignored. Refer to [Vector users](#vector-users) below.

**Pick a maintenance window.** Expect a brief agent restart. Refer to [Expected disruption](#expected-disruption).

## Values that changed

Most keys carry over untouched, and no `secrets.*` key was removed or renamed. Four keys were removed and are silently ignored if left in your values file:

| Removed key                 | What to do                                                        |
| --------------------------- | ----------------------------------------------------------------- |
| `configMap.name`            | Nothing. The operator names the ConfigMap `<release-name>-config` |
| `vector.existingConfigMaps` | Nothing. The operator builds and owns Vector's ConfigMap          |
| `vector.role`               | Nothing. Vector always runs as a DaemonSet                        |
| `vector.service`            | Nothing. No Vector Service is created                             |

Any other `vector.*` key you set for the upstream Vector chart, such as `vector.customConfig` or `vector.podLabels`, is no longer read. The chart honors only the `vector.*` keys documented in [Pipekit Agent Helm Chart](/reference/helm-values/agent.md).

One key was renamed. From chart `7.5.0`, `features.metrics.enabled` is replaced by `features.metrics.scrapeCluster`, which defaults to `true`. The metrics collector now always runs, and `scrapeCluster` controls whether it reads cluster-wide telemetry from nodes, pods, and the Kubernetes API server. An explicit `features.metrics.enabled: false` in your values file still maps to `scrapeCluster: false`, so an existing opt-out survives the upgrade. If you never set `features.metrics.enabled`, the upgrade turns cluster scraping on, because the `6.x` default was off and the `7.x` default is on. Set `features.metrics.scrapeCluster` to `false` if you do not want that. Setting `features.workflows.namespace` forces it off, because a namespace-scoped agent asked not to hold cluster-wide access.

New keys you can now set: `deployment.affinity`, `deployment.tolerations`, `deployment.extraEnvFrom`, `vector.nodeSelector`, `vector.tolerations`, `vector.affinity`, `features.metrics.pipekitApiUrl`, `features.metrics.metricsCollectorUrl`, `features.metrics.scrapeCluster`, `features.metrics.insecureK8s`, `scheduling.*`, `operator.*`, and `crds.applyJob.*`.

### One place to set where the pods run

The chart has a top-level `scheduling` block, with `nodeSelector`, `tolerations`, and `affinity`. Set any of these values in `scheduling` and it will be applied to all appropriate pods controlled by the chart and operator. Those pods are the operator, the CRD apply Job, the Pipekit Agent, the metrics collector, and Vector.

Each pod also keeps a section of its own, and the value there replaces the `scheduling` one for that pod:

| Pod               | Keys                                                                                                                    |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Operator          | `operator.nodeSelector`, `operator.tolerations`, `operator.affinity`                                                    |
| CRD apply Job     | `crds.applyJob.nodeSelector`, `crds.applyJob.tolerations`, `crds.applyJob.affinity`                                     |
| Pipekit Agent     | `deployment.nodeSelector`, `deployment.tolerations`, `deployment.affinity`                                              |
| Metrics collector | `features.metrics.collectorNodeSelector`, `features.metrics.collectorTolerations`, `features.metrics.collectorAffinity` |
| Vector            | `vector.nodeSelector`, `vector.tolerations`, `vector.affinity`                                                          |

Vector uses `scheduling.tolerations` only. A node selector or an affinity rule would stop the DaemonSet collecting logs on the nodes it skips. Set `vector.nodeSelector` or `vector.affinity` when you do want to restrict Vector, and expect to lose the logs of every workflow pod on the nodes you exclude.

The `kubernetes.io/os: linux` default moved from `deployment.nodeSelector` to `scheduling.nodeSelector`, so every pod except Vector now gets it. A per-pod key replaces the whole `scheduling` value rather than merging with it. Setting `deployment.nodeSelector` to `nodegroup: my-nodes` gives the agent pod that one label, where chart `8.0` and earlier gave it `nodegroup: my-nodes` and `kubernetes.io/os: linux` together. Add `kubernetes.io/os: linux` to your own block if you still want it.

{% hint style="info" %}
Your existing values file keeps working. While you leave both `scheduling` and the pod's own section unset, the operator pod and the CRD apply Job still read `deployment.nodeSelector`, `deployment.tolerations`, and `deployment.affinity`, so an upgrade needs no change from you. A later release removes that one fallback. Move those settings into `scheduling`, `operator`, or `crds.applyJob` when you next edit your values file. The per-pod sections themselves are permanent.
{% endhint %}

{% hint style="info" %}
`scheduling` is a new field on the `Installation` custom resource, so the CRD apply Job has to run before the operator can read it. The Job runs by default, and Argo CD applies the CRD on every sync. If you set `crds.applyJob.enabled` to `false` and do not use Argo CD, apply the new CRD yourself first. Otherwise the API server drops `spec.scheduling` from the resource, `helm upgrade` still reports success, and the agent, the collector, and Vector run without your settings. The operator pod and the CRD apply Job read their values from Helm, so those two are placed either way.
{% endhint %}

## Upgrade steps

These commands assume a release named `pipekit-agent` in the `argo` namespace. Substitute your own release name and namespace throughout.

### 1. Record your current state

Capture your live values and a record of the resources the `6.x` chart owns:

```bash
helm get values pipekit-agent -n argo -o yaml > pipekit-agent-6x-values.yaml
kubectl -n argo get deploy,sa,cm,svc -l app.kubernetes.io/name=pipekit-agent
```

Keep `pipekit-agent-6x-values.yaml`, because you need it to roll back. Use `-o yaml`, because the default output adds a `USER-SUPPLIED VALUES:` heading that does not belong in a values file.

After the upgrade, list the same resources by the operator's label instead, since the agent's ConfigMap no longer carries `app.kubernetes.io/name: pipekit-agent`:

```bash
kubectl -n argo get deploy,sa,cm,svc -l app.kubernetes.io/managed-by=pipekit-agent-operator
```

### 2. Install the CRD

Do this if you upgrade with Helm directly or with Flux. Argo CD applies the chart's CRDs on sync, so you can go straight to step 3 unless the Application sets `skipCrds: true`. Helm does not apply `crds/` when upgrading an existing release, so the CRD has to be in place before step 3. Check the latest `7.x` version with `helm search repo pipekit/pipekit-agent --versions` and use it in place of `7.6.0` below:

```bash
helm repo update
helm pull pipekit/pipekit-agent --version 7.6.0 --untar
kubectl apply -f pipekit-agent/crds/
kubectl get crd installations.agent.pipekit.io
```

The CRD is cluster-scoped, so this is safe to run again if it already exists.

### 3. Upgrade the release

Confirm step 2 is done before you run this. Pin the version explicitly, because the `6.x` line is still published and an unpinned upgrade may not move you where you expect:

```bash
helm repo update
helm upgrade pipekit-agent pipekit/pipekit-agent \
  --version 7.6.0 \
  --values pipekit-agent-6x-values.yaml \
  -n argo
```

Helm removes the workloads the old chart owned, and the operator recreates them from the `Installation` resource under its own ownership. No manual intervention is required between those two phases. From chart `7.6.0`, Helm also runs the CRD apply Job before the upgrade. It re-applies the CRD you installed in step 2, which is harmless, and pulls the operator image at the chart's `appVersion` to do so.

### 4. Validate

Work through this in order. The `Installation` resource has no status conditions, so verify the operator and the resources it produced rather than the custom resource itself.

| Check                       | Command                                                           | Expect                                                           |
| --------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------- |
| CRD present                 | `kubectl get crd installations.agent.pipekit.io`                  | The CRD exists                                                   |
| Installation created        | `kubectl get installation`                                        | One resource, named after your release                           |
| Operator running            | `kubectl -n argo get deploy agent-controller-manager`             | `1/1` available                                                  |
| Operator reconciled cleanly | `kubectl -n argo logs deploy/agent-controller-manager`            | No repeated errors, no `forbidden`                               |
| Agent running               | `kubectl -n argo get deploy pipekit-agent`                        | Available, same name as before                                   |
| Agent healthy               | `kubectl -n argo logs deploy/pipekit-agent`                       | No `forbidden` errors listing workflows                          |
| Cluster connected           | `pipekit list clusters`, or the `Clusters` view in the Pipekit UI | `[ACTIVE]` is `true` and `[AGENT VERSION]` shows the new version |
| Metrics collector           | `kubectl -n argo get deploy pipekit-agent-collector`              | Available                                                        |
| Vector, if enabled          | `kubectl -n argo get ds pipekit-vector`                           | Ready on every node                                              |

Then submit a workflow and confirm the run and its logs appear in Pipekit.

## Vector users

Two behaviors changed.

**Vector is built by the operator.** The upstream Vector chart is no longer a dependency. The operator builds the DaemonSet, its ConfigMap, and its RBAC, and names the DaemonSet `pipekit-vector`. Under `6.x` the DaemonSet was named after your release, so any automation or dashboard that targets the old name needs updating.

**Credentials are inherited automatically.** The Vector ConfigMap now reads `${PIPEKIT_SECRET_ACCESS_KEY}` from the agent's Secret instead of having the key inlined at template time. When you leave `vector.envFrom` empty, the operator mounts the agent's Secret for you, whether that Secret came from `secrets.existingSecret` or was created by the chart. You no longer need to repeat the secret reference under `vector.envFrom`, though an explicit entry still overrides the default if you want Vector to read a different Secret.

Refer to [Vector Log Collection](/using-pipekit/operating-at-scale/vector-log-collection.md) for the full picture.

## Expected disruption

| Component         | What happens                                                             |
| ----------------- | ------------------------------------------------------------------------ |
| Agent pod         | Brief gap. Helm removes the old Deployment and the operator recreates it |
| Metrics collector | Brief gap if it was running. Created if it was not, from chart `7.5.0`   |
| Vector            | Brief gap per node, if Vector is enabled                                 |

{% hint style="info" %}
Running Argo Workflows are not interrupted by the agent restarting, but Pipekit does not track state while the agent is down. Upgrade outside a critical workflow window.
{% endhint %}

## Troubleshooting

| Symptom                                                                      | Cause                                                                          | Fix                                                                                                 |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `no matches for kind "Installation" in version "agent.pipekit.io/v1alpha1"`  | The CRD is not installed. Helm does not apply `crds/` on upgrade               | Run step 2, then repeat step 3                                                                      |
| `exists and cannot be imported into the current release`                     | Helm was asked to take over a resource the operator owns                       | Delete the `Installation` resource, wait for its resources to go, then rerun the release            |
| `no Deployment with the name "pipekit-agent" found`                          | Helm was asked to take over a resource the operator owns                       | Delete the `Installation` resource, wait for its resources to go, then rerun the release            |
| Operator logs show `attempting to grant RBAC permissions not currently held` | The operator's `escalate` permission is missing or blocked by admission policy | Confirm the operator's `ClusterRole` applied cleanly and that your policy engine permits `escalate` |
| Operator logs show `forbidden`                                               | Admission policy is blocking a resource the operator creates                   | Review the policy that matches the named resource                                                   |
| Operator is available but no agent `Deployment` appears                      | `features.workflows.enabled` is `false` in your values                         | Set it to `true` and upgrade again                                                                  |
| Agent runs but the cluster shows as disconnected                             | The agent's Secret did not carry over                                          | Check `secrets.existingSecret` still names a Secret in the namespace                                |

Collect the operator's logs before changing anything:

```bash
kubectl -n argo logs deploy/agent-controller-manager --tail=100
kubectl get installation -o yaml
```

## Rolling back

Remove the operator's resources before you reinstall the `6.x` chart. Helm refuses to adopt a resource that exists in the cluster but was absent from the release it is upgrading from, so rolling straight back fails with `Deployment "pipekit-agent" in namespace "argo" exists and cannot be imported into the current release`. Deleting the `Installation` resource lets Kubernetes garbage-collect everything the operator owns:

```bash
kubectl delete installation pipekit-agent
kubectl -n argo get deploy,sa,cm,svc -l app.kubernetes.io/managed-by=pipekit-agent-operator
```

Wait until that second command returns nothing before you continue.

Then reinstall the `6.x` chart with the values you captured in step 1:

```bash
helm upgrade pipekit-agent pipekit/pipekit-agent \
  --version 6.67.2 \
  --values pipekit-agent-6x-values.yaml \
  -n argo
```

The chart takes ownership of the agent workloads again and recreates them under its own names and labels. The CRD is cluster-scoped and is not removed by the rollback. Leave it in place if you plan to retry, or remove it once no installations remain:

```bash
kubectl get installation
kubectl delete crd installations.agent.pipekit.io
```

{% hint style="warning" %}
Deleting the CRD deletes every `Installation` resource in the cluster, and Kubernetes garbage-collects the resources those installations own. Confirm the `6.x` chart has recreated the agent before you remove the CRD.
{% endhint %}

## Getting help

If the operator does not reconcile, collect `kubectl -n argo logs deploy/agent-controller-manager` and `kubectl get installation -o yaml`, then contact us over Slack, or by email at <support@pipekit.io>.