# Helm Install ## Installation The Pipekit Agent must be installed onto each cluster where your Argo Workflow controller runs. ### Helm Install When you choose to [connect a cluster in Pipekit](https://docs.pipekit.io/readme), you will be provided with a Secret Access Key for the cluster and a unique Cluster ID. These need to be passed to the helm chart when installing the Pipekit Agent. You should store the Secret Access Key securely in a password manager or similar. ```bash kubectl create ns argo helm repo add pipekit https://helm.pipekit.io helm upgrade -i -n argo \ pipekit-agent pipekit/pipekit-agent \ --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \ --set secrets.pipekitClusterId="[provided Cluster ID]" ``` ### Argo CD If you are using Argo CD, you can install the Pipekit Agent using the following steps: 1. Create and push [a secret](#secrets) to your cluster containing your pipekitSecretAccessKey and pipekitClusterId 2. Create a new Argo CD Application Manifest, referencing that secret: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: pipekit-agent namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: destination: namespace: argo server: 'https://kubernetes.default.svc' source: repoURL: 'https://helm.pipekit.io' chart: pipekit-agent targetRevision: HEAD helm: parameters: - name: secrets.existingSecret value: "pipekit-agent" project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - PrunePropagationPolicy=background - CreateNamespace=true ``` Argo CD offers other secrets management alternatives to ensure that your secrets are not stored in plaintext in your git repository. You can read more about them [in the Argo CD documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/#secret-management). ### Release Notes Release notes are published on the [Pipekit Releases site](https://releases.pipekit.io). ## Pipekit Agent Helm Chart Values There are other optional, configurable options within the Helm Chart. These are listed below. | Key | Type | Default | Description | | ---------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | configMap.additionalConfig | object | `{}` | Add any additional configMap entries to the Pipekit Agent ConfigMap. | | configMap.allowExternalWorkflows | bool | `false` | Allow Pipekit Agent to monitor workflows created without Pipekit. This is disabled by default. | | configMap.annotations | object | `{}` | Add any additional annotations to the Pipekit Agent ConfigMap. | | configMap.enableTelemetryUsersHTTPClient | bool | `false` | Should telemetry be enabled for users http client. | | configMap.enableTelemetryUsersHTTPClientEndpointInfo | bool | `false` | Should endpoint info be included in the telemetry for users http client. | | configMap.enableTelemetryWorkqueue | bool | `false` | Should telemetry be enabled for workqueue. | | configMap.externalWorkflowsMaxWorkers | int | 10 | The maximum number of workers that the Pipekit Agent can use to execute external workflows. | | configMap.logLevel | string | `"INFO"` | Set the logLevel for the Pipekit Agent. | | configMap.logSendWorkers | int | 64 | The number of workers that the Pipekit Agent can use to send logs. | | configMap.maxLogBatchSize | int | 1024 | The maximum number of logs to send in a single batch. | | configMap.messengerBaseUri | string | `https://api.pipekit.io` | Set the base URI for the Pipekit Messenger Service. Only set this if you are running a self-hosted Pipekit instance. | | configMap.name | string | `""` | Specifies the name of the ConfigMap to create. Leave blank to auto-generate a name. | | configMap.podInformerWorkers | int | 8 | The number of workers that the Pipekit Agent can use to process pod consuming queue. | | configMap.sendLogsToPipekit | bool | `true` | Send Workflow logs to Pipekit. Pipekit will store and index logs for you to view at pipekit.io Refer to the Vector section to enable performant log collection when running at massive scale. | | configMap.shouldDeleteExternalWorkflows | bool | `false` | Should the Pipekit Agent delete external workflows after execution. This is set to false by default. | | configMap.telemetryProtocols | string | `"prometheus"` | set to 'none' to disable all telemetry | | configMap.usersBaseUri | string | `https://api.pipekit.io` | Set the base URI for the Pipekit Users Service. Only set this if you are running a self-hosted Pipekit instance. | | configMap.workflowInformerWorkers | int | 8 | The number of workers that the Pipekit Agent can use to process workflow consuming queue. | | customCABundle.configMapKey | string | `"root-ca.pem"` | Key in the ConfigMap that contains the CA certificate. | | customCABundle.configMapName | string | `""` | Name of the ConfigMap containing the CA certificate. | | customCABundle.enabled | bool | `false` | Enable mounting a custom CA certificate bundle. | | customCABundle.mountPath | string | `"/etc/ssl/certs/root-ca.pem"` | Path where the CA certificate will be mounted. | | deployment.image.imagePullSecrets | list | `[]` | Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent image. | | deployment.image.pullPolicy | string | `"IfNotPresent"` | The imagePullPolicy for the Pipekit Agent image. | | deployment.image.repository | string | `"pipekit13/pipekit-agent"` | The Pipekit Agent image name and repository. Change this if you wish to host the container yourself. | | deployment.image.tag | string | `""` | Allows you to pin to a specific image tag. The Chart.yaml contains a default value. | | deployment.nodeSelector | object | `{"kubernetes.io/os":"linux"}` | The Pipekit Agent pod's node selector. | | deployment.podAnnotations | object | `{}` | Add any additional annotations to the Pipekit Agent pod. | | deployment.podLabels | object | `{}` | Add any additional labels to the Pipekit Agent pod. | | deployment.resources | object | `{}` | Set the Pipekit Agent pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage. | | features | object | `{"metrics":{"argoWfControllerLabels":{},"argoWfMetricsPath":"/metrics","argoWfMetricsPort":9090,"argoWfNamespace":"argo","collectorLogLevel":"info","collectorNodeSelector":{"kubernetes.io/os":"linux"},"collectorResources":{"requests":{"cpu":"80m","memory":"256Mi"}},"deployment":{"image":{"imagePullSecrets":[],"pullPolicy":"IfNotPresent","repository":"pipekit13/agent","tag":""},"replicas":1,"resources":{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}},"enabled":false,"k8sAuthType":"","otelCollectorImage":"","otelCollectorTag":""},"workflows":{"enabled":true,"namespace":"","nodeOffloading":{"argoDB":{"archiveTableName":"argo_archived_workflows","database":"","driver":"postgres","host":"","maxConnectionLifetime":5,"maxIdleConnections":15,"maxOpenConnections":30,"maxRetries":3,"offloadedNodesTableName":"argo_workflows","port":5432,"secret":{"annotations":{},"argoDBPassword":"","argoDBUsername":"","existingSecret":"","name":""},"sslMode":"disable"},"enabled":false},"stepPodsDelete":{"enabled":true}}}` | Enable and Disable features of the Pipekit Agent. | | features.metrics.argoWfControllerLabels | object | `{}` | the labels we should use to select the workflow controller. Defaults to `app: workflow-controller` if not set (suitable for the official Workflows release manifest) | | features.metrics.argoWfMetricsPath | string | `"/metrics"` | The path where your Argo Workflows controller exposes metrics. | | features.metrics.argoWfMetricsPort | int | `9090` | The port number of your Argo Workflows controller that exposes metrics. | | features.metrics.argoWfNamespace | string | `"argo"` | The namespace where your Argo Workflows controller is running. This tells Pipekit Agent where to find the Workflow metrics. | | features.metrics.collectorLogLevel | string | `"info"` | The Pipekit Agent Metrics Collector will log at this level. | | features.metrics.collectorNodeSelector | object | `{"kubernetes.io/os":"linux"}` | A nodeselector for the opentelemetry collector | | features.metrics.collectorResources | object | `{"requests":{"cpu":"80m","memory":"256Mi"}}` | Resource requests and limits for the collector | | features.metrics.deployment.image.imagePullSecrets | list | `[]` | Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent Operator image. | | features.metrics.deployment.image.pullPolicy | string | `"IfNotPresent"` | The imagePullPolicy for the Pipekit Agent Operator image. | | features.metrics.deployment.image.repository | string | `"pipekit13/agent"` | The Pipekit Agent Operator image name and repository. Change this if you wish to host the container yourself. | | features.metrics.deployment.image.tag | string | `""` | Allows you to pin to a specific image tag. The Chart.yaml contains a default value. | | features.metrics.deployment.replicas | int | `1` | The number of replicas for the Pipekit Agent Metrics Collector. | | features.metrics.deployment.resources | object | `{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Set the Pipekit Agent Controller Manager pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage. | | features.metrics.enabled | bool | `false` | Enable and Disable Pipekit Workflow metrics. This passes Workflow metrics from your cluster to Pipekit in order to view them in the Workflow Metrics dashboard. # [You must sign up here to use this feature](https://pipekit.io/metrics-signup) or email . # This feature requires feature.workflows.namespace to be "", for cluster wide access | | features.metrics.k8sAuthType | string | `""` | kubernetes authentication method, defaults to serviceAccount alternative setting is none | | features.metrics.otelCollectorImage | string | `""` | The docker name of the opentelemetry contrib collector image to deploy | | features.metrics.otelCollectorTag | string | `""` | The tag of the collector image to use | | features.workflows | object | `{"enabled":true,"namespace":"","nodeOffloading":{"argoDB":{"archiveTableName":"argo_archived_workflows","database":"","driver":"postgres","host":"","maxConnectionLifetime":5,"maxIdleConnections":15,"maxOpenConnections":30,"maxRetries":3,"offloadedNodesTableName":"argo_workflows","port":5432,"secret":{"annotations":{},"argoDBPassword":"","argoDBUsername":"","existingSecret":"","name":""},"sslMode":"disable"},"enabled":false},"stepPodsDelete":{"enabled":true}}` | Enable and Disable the Pipekit Agent. This communicates with the Pipekit control plane and allows you to run and manage your workflows. | | features.workflows.namespace | string | "" | Specify a single namespace where this Pipekit Agent instance will manage workflows. If empty this will work across all namespaces as a cluster wide agent | | features.workflows.nodeOffloading | object | false | Enable reading offloaded node status from Argo's database when node status offloading is configured. | | features.workflows.nodeOffloading.argoDB | object | `{"archiveTableName":"argo_archived_workflows","database":"","driver":"postgres","host":"","maxConnectionLifetime":5,"maxIdleConnections":15,"maxOpenConnections":30,"maxRetries":3,"offloadedNodesTableName":"argo_workflows","port":5432,"secret":{"annotations":{},"argoDBPassword":"","argoDBUsername":"","existingSecret":"","name":""},"sslMode":"disable"}` | Argo Workflows Database Configuration. Only required when nodeOffloading is enabled. | | features.workflows.nodeOffloading.argoDB.archiveTableName | string | "argo\_archived\_workflows" | Table name for archived workflows (must match Argo Workflows config) | | features.workflows.nodeOffloading.argoDB.database | string | `""` | Database name | | features.workflows.nodeOffloading.argoDB.driver | string | "postgres" | Database driver: "postgres" or "mysql" | | features.workflows.nodeOffloading.argoDB.host | string | `""` | Database host address | | features.workflows.nodeOffloading.argoDB.maxConnectionLifetime | int | 5 | Max db connection lifetime in minutes | | features.workflows.nodeOffloading.argoDB.maxIdleConnections | int | 15 | Max number of idle db connections | | features.workflows.nodeOffloading.argoDB.maxOpenConnections | int | 30 | Max number of open db connections | | features.workflows.nodeOffloading.argoDB.maxRetries | int | `3` | Maximum number of connection retries | | features.workflows.nodeOffloading.argoDB.offloadedNodesTableName | string | "argo\_workflows" | Table name for offloaded node status (must match Argo Workflows config) | | features.workflows.nodeOffloading.argoDB.port | int | 5432 | Database port (default: 5432 for postgres, 3306 for mysql) | | features.workflows.nodeOffloading.argoDB.secret.annotations | object | `{}` | Add any additional annotations to the ArgoWF DB Secret. Requires features.workflows.nodeOffloading.argoDB.secret.existingSecret to be blank. | | features.workflows.nodeOffloading.argoDB.secret.argoDBPassword | string | `""` | Enter the argoDBPassword for the database used by Argo Workflows. Requires features.workflows.nodeOffloading.argoDB.secret.existingSecret to be blank. | | features.workflows.nodeOffloading.argoDB.secret.argoDBUsername | string | `""` | Enter the argoDBUsername for the database used by Argo Workflows. Requires features.workflows.nodeOffloading.argoDB.secret.existingSecret to be blank. | | features.workflows.nodeOffloading.argoDB.secret.existingSecret | string | `""` | The name of an [existing secret](#secrets) containing your argoDBUsername and argoDBPassword. | | features.workflows.nodeOffloading.argoDB.secret.name | string | `""` | Specifies the name of the Secret to create. Leave blank to auto-generate a name. Requires features.workflows.nodeOffloading.argoDB.secret.existingSecret to be blank. | | features.workflows.nodeOffloading.argoDB.sslMode | string | `"disable"` | SSL mode for PostgreSQL connections (e.g., "disable", "require", "verify-full") This is ignored for MySQL connections. | | features.workflows.stepPodsDelete | object | `{"enabled":true}` | Allow Pipekit Agent to delete the Pod associated with a running Workflow Step when triggered through the Pipekit UI. If a retryStrategy is enabled in the running Workflow, this will cause the Step to be retried. | | fullnameOverride | string | `""` | Completely replace the generated name with the provided name. | | metrics.enabled | bool | `false` | Enables a Service and ServiceMonitor for Prometheus metrics. | | nameOverride | string | `""` | Replaces the name of the chart in Chart.yaml. | | secrets.annotations | object | `{}` | Add any additional annotations to the Pipekit Agent Secret. Requires secrets.existingSecret to be blank. | | secrets.existingSecret | string | `""` | The name of an [existing secret](#secrets) containing your pipekitSecretAccessKey and pipekitClusterId. | | secrets.name | string | `""` | Specifies the name of the Secret to create. Leave blank to auto-generate a name. Requires secrets.existingSecret to be blank. | | secrets.pipekitClusterId | string | `""` | Enter the pipekitClusterId provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank. | | secrets.pipekitSecretAccessKey | string | `""` | Enter the pipekitSecretAccessKey provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank. | | serviceAccount.annotations | object | `{}` | Add any additional annotations to the Pipekit Agent ServiceAccount and ServiceAccount Token Secret. | | serviceAccount.create | bool | `true` | Specifies whether a ServiceAccount should be created. If false, you must provide an [existing ServiceAccount name](#pipekit-agent-cluster-permissions). | | serviceAccount.name | string | `""` | Specifies the name of the ServiceAccount to create if serviceAccount.create is true. Otherwise, specifies the name of an existing ServiceAccount to use. | | vector.dataDir | string | `"/tmp"` | Specify the path for Vector's data, only used when existingConfigMaps are used. | | vector.enabled | bool | `false` | Enable Vector for node-level log collection. When enabled, Vector runs as a DaemonSet and collects logs directly from the node filesystem for better performance. `configMap.sendLogsToPipekit` should be `false` when setting `vector.enabled` to `true` | | vector.envFrom | list | `[]` | Environment variables loaded from Secret. This allows Vector to authenticate with Pipekit if using secrets.existingSecret. Set secretRef.name to your existing secret name. If using secrets.pipekitSecretAccessKey, this is not needed. | | vector.existingConfigMaps | list | `["pipekit-vector"]` | Reference the configuration ConfigMap that is generated by this Helm chart | | vector.image.pullSecrets | list | `[]` | The [imagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) to reference for the Vector Pods. | | vector.resources | object | `{"requests":{"cpu":"30m","memory":"130Mi"}}` | Set Vector resource requests and limits. | | vector.role | string | `"Agent"` | Run Vector as a DaemonSet (Agent role) to collect logs from each node | | vector.service.enabled | bool | `false` | | ## Secrets If you wish to provide your own secret you need to populate it with the pipekitSecretAccessKey and pipekitClusterId values, using the defined data keys below: | Chart var | .data. in Secret | | ------------------------------ | ---------------------------- | | secrets.pipekitSecretAccessKey | PIPEKIT\_SECRET\_ACCESS\_KEY | | secrets.pipekitClusterId | PIPEKIT\_CLUSTER\_ID | eg: ```bash kubectl -n argo create secret generic pipekit-agent \ --from-literal=PIPEKIT_SECRET_ACCESS_KEY='abc123' \ --from-literal=PIPEKIT_CLUSTER_ID='123abc' ``` You must use single quotes '' to escape special characters such as $, , \*, =, and ! in your strings. If you don't, your shell will interpret these characters. ## Enabling Externally Triggered Workflows If you wish to enable [externally triggered workflows](https://docs.pipekit.io/pipekit/pipes/externally-triggered-workflows), you can set the following values in your Helm install command: ```bash kubectl create ns argo helm repo add pipekit https://helm.pipekit.io helm upgrade -i -n argo \ pipekit-agent pipekit/pipekit-agent \ --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \ --set secrets.pipekitClusterId="[provided Cluster ID]" \ --set configMap.allowExternalWorkflows=true ``` You can additionally set `configMap.shouldDeleteExternalWorkflows`. Refer to the values table above for more information. ## Pipekit Agent Cluster Permissions The Pipekit Agent needs to interact with Argo Workflows on your cluster. In order to do this, we create a ServiceAccount and ClusterRole for the Pipekit Agent. If you wish to manually manage this ServiceAccount and ClusterRole, you can set `serviceAccount.create` to false and provide the name of an existing ServiceAccount to use. The required minimum permissions are: ```yaml - apiGroups: - argoproj.io resources: - clusterworkflowtemplates - workflows - cronworkflows - workflowtemplates verbs: - create - delete - get - list - patch - update - watch ``` ## Upgrading Pipekit Agent ### Checking the latest version Pipekit Agent is automatically published to [Artifact Hub](https://artifacthub.io/packages/helm/pipekit-helm/pipekit-agent). You can use this service to configure automatic notifications of new versions, either via an RSS feed or a webhook. Alternatively, you can search the helm repo for the latest version of the Pipekit Agent using the following command: ```bash # Add the repo if you don't already have it. helm repo add pipekit https://helm.pipekit.io helm repo update helm search repo pipekit/pipekit-agent --versions ``` For further information on the `helm search repo` command, please refer to the [official Helm documentation](https://helm.sh/docs/helm/helm_search_repo/). ### Changes to the default values.yaml Prior to upgrading, you should ensure that you understand any changes to the default values.yaml and the impact those changes may have on your installation. This page is always updated with the latest available helm chart values for the Pipekit Agent. If you wish to upgrade to an older version of the Pipekit Agent helm chart, you can extract the default values for that version using the following command: ```bash helm show values pipekit/pipekit-agent --version [version] ``` The Pipekit-Agent values file is commented so you can see what each value does. You can extract your existing values from your current installation using the following command: ```bash helm get values pipekit-agent -n argo ``` For further information the helm commands used above, please refer to the [official Helm documentation](https://helm.sh/docs/helm/). ### Upgrading the Pipekit Agent To upgrade the Pipekit Agent, you can use the following command: ```bash helm upgrade pipekit-agent pipekit/pipekit-agent \ --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \ --set secrets.pipekitClusterId="[provided Cluster ID]" \ -n argo ``` For more information on using Helm to perform upgrades, please check [the official Helm documentation](https://helm.sh/docs/helm/helm_upgrade/). ### Automating the upgrades using Gitops If you use a Gitops tool such as Argo CD, you can simply commit your changes to your git repository and the tool will handle the Helm upgrade for you. If you wish to automate the upgrade process, we recommend a third party tool called [Renovate Bot](https://docs.renovatebot.com/) that can be configured to automatically raise pull requests for you when a new version of the Pipekit Agent is released. ### Upgrade Support If you have an issue upgrading your Pipekit Agent that isn't addressed here, please contact us over Slack, or by email at .