Helm Install

Installation

The Pipekit Agent must be installed onto each cluster where your Argo Workflow controller runs.

Helm Install

When you choose to connect a cluster in Pipekit, you will be provided with a Secret Access Key for the cluster and a unique Cluster ID. These need to be passed to the helm chart when installing the Pipekit Agent. You should store the Secret Access Key securely in a password manager or similar.

kubectl create ns argo
helm repo add pipekit https://helm.pipekit.io

helm upgrade -i -n argo \
pipekit-agent pipekit/pipekit-agent \
--set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
--set secrets.pipekitClusterId="[provided Cluster ID]"

Argo CD

If you are using Argo CD, you can install the Pipekit Agent using the following steps:

Create and push a secret to your cluster containing your pipekitSecretAccessKey and pipekitClusterId
Create a new Argo CD Application Manifest, referencing that secret:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: pipekit-agent
namespace: argocd
finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
destination:
    namespace: argo
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: 'https://helm.pipekit.io'
    chart: pipekit-agent
    targetRevision: HEAD
    helm:
      parameters:
        - name: secrets.existingSecret
          value: "pipekit-agent"
project: default
syncPolicy:
    automated:
    prune: true
    selfHeal: true
    syncOptions:
    - PrunePropagationPolicy=background
    - CreateNamespace=true

Argo CD offers other secrets management alternatives to ensure that your secrets are not stored in plaintext in your git repository. You can read more about them here.

Release Notes

Release notes are published on the Pipekit Releases site.

Pipekit Agent Helm Chart Values

There are other optional, configurable options within the Helm Chart. These are listed below.

Key Type Default Description

Key	Type	Default	Description
configMap.additionalConfig	object	`{}`
configMap.allowExternalWorkflows	bool	`false`	Allow Pipekit Agent to monitor workflows created without Pipekit. This is disabled by default.
configMap.annotations	object	`{}`	Add any additional annotations to the Pipekit Agent ConfigMap.
configMap.externalWorkflowsCheckPeriod	string	`"10s"`	The period at which the Pipekit Agent checks for external workflows. This is set to 10s by default.
configMap.externalWorkflowsMaxWorkers	string	10	The maximum number of workers that the Pipekit Agent can use to execute external workflows.
configMap.externalWorkflowsNamespaces	string	`All namespaces`	The list of comma-separated namespaces where the Pipekit Agent should look for external workflows. Notes: Pipekit Agent needs appropriate permissions to get, list, patch and delete workflows in the defined namespaces.
configMap.logLevel	string	`"INFO"`	Set the logLevel for the Pipekit Agent.
configMap.name	string	`""`	Specifies the name of the ConfigMap to create. Leave blank to auto-generate a name.
configMap.sendLogsToPipekit	bool	`true`	Send Workflow logs to Pipekit. Pipekit will store and index logs for you to view at pipekit.io
configMap.shouldDeleteExternalWorkflows	bool	`false`	Should the Pipekit Agent delete external workflows after execution. This is set to false by default.
configMap.telemetryProtocols	string	`"prometheus"`	set to 'none' to disable all telemetry
deployment.image.imagePullSecrets	list	`[]`	Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent image.
deployment.image.pullPolicy	string	`"IfNotPresent"`	The imagePullPolicy for the Pipekit Agent image.
deployment.image.repository	string	`"pipekit13/pipekit-agent"`	The Pipekit Agent image name and repository. Change this if you wish to host the container yourself.
deployment.image.tag	string	`""`	Allows you to pin to a specific image tag. The Chart.yaml contains a default value.
deployment.nodeSelector	object	`{"kubernetes.io/os":"linux"}`	The Pipekit Agent pod's node selector.
deployment.podAnnotations	object	`{}`	Add any additional annotations to the Pipekit Agent pod.
deployment.podLabels	object	`{}`	Add any additional labels to the Pipekit Agent pod.
deployment.resources	object	`{}`	Set the Pipekit Agent pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage.
features	object	{"metrics":{"argoWfControllerLabels":{},"argoWfMetricsPath":"/metrics","argoWfMetricsPort":9090,"argoWfNamespace":"argo","collectorLogLevel":"info","collectorNodeSelector":{"kubernetes.io/os":"linux"},"collectorResources":{"requests":{"cpu":"80m","memory":"256Mi"}},"deployment":{"image":{"imagePullSecrets":[],"pullPolicy":"IfNotPresent","repository":"pipekit13/agent","tag":""},"replicas":1,"resources":{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}},"enabled":false,"k8sAuthType":"","otelCollectorImage":"","otelCollectorTag":""},"workflows":{"enabled":true}}	Enable and Disable features of the Pipekit Agent.
features.metrics.argoWfControllerLabels	object	`{}`	the labels we should use to select the workflow controller. Defaults to `app: workflow-controller` if not set (suitable for the official Workflows release manifest)
features.metrics.argoWfMetricsPath	string	`"/metrics"`	The path where your Argo Workflows controller exposes metrics.
features.metrics.argoWfMetricsPort	int	`9090`	The port number of your Argo Workflows controller that exposes metrics.
features.metrics.argoWfNamespace	string	`"argo"`	The namespace where your Argo Workflows controller is running. This tells Pipekit Agent where to find the Workflow metrics.
features.metrics.collectorLogLevel	string	`"info"`	The Pipekit Agent Metrics Collector will log at this level.
features.metrics.collectorNodeSelector	object	`{"kubernetes.io/os":"linux"}`	A nodeselector for the opentelemetry collector
features.metrics.collectorResources	object	`{"requests":{"cpu":"80m","memory":"256Mi"}}`	Resource requests and limits for the collector
features.metrics.deployment.image.imagePullSecrets	list	`[]`	Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent Operator image.
features.metrics.deployment.image.pullPolicy	string	`"IfNotPresent"`	The imagePullPolicy for the Pipekit Agent Operator image.
features.metrics.deployment.image.repository	string	`"pipekit13/agent"`	The Pipekit Agent Operator image name and repository. Change this if you wish to host the container yourself.
features.metrics.deployment.image.tag	string	`""`	Allows you to pin to a specific image tag. The Chart.yaml contains a default value.
features.metrics.deployment.replicas	int	`1`	The number of replicas for the Pipekit Agent Metrics Collector.
features.metrics.deployment.resources	object	`{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}`	Set the Pipekit Agent Controller Manager pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage.
features.metrics.enabled	bool	`false`	Enable and Disable Pipekit Workflow metrics. This passes Workflow metrics from your cluster to Pipekit in order to view them in the Workflow Metrics dashboard. # You must sign up here to use this feature or email hello@pipekit.io.
features.metrics.k8sAuthType	string	`""`	kubernetes authentication method, defaults to serviceAccount alternative setting is none
features.metrics.otelCollectorImage	string	`""`	The docker name of the opentelemetry contrib collector image to deploy
features.metrics.otelCollectorTag	string	`""`	The tag of the collector image to use
features.workflows	object	`{"enabled":true}`	Enable and Disable the Pipekit Agent. This communicates with Pipekit and allows you to run and manage your workflows.
fullnameOverride	string	`""`	Completely replace the generated name with the provided name.
metrics.enabled	bool	`false`	Enables a Service and ServiceMonitor for Prometheus metrics.
nameOverride	string	`""`	Replaces the name of the chart in Chart.yaml.
secrets.annotations	object	`{}`	Add any additional annotations to the Pipekit Agent Secret. Requires secrets.existingSecret to be blank.
secrets.existingSecret	string	`""`	The name of an existing secret containing your pipekitSecretAccessKey and pipekitClusterId.
secrets.name	string	`""`	Specifies the name of the Secret to create. Leave blank to auto-generate a name. Requires secrets.existingSecret to be blank.
secrets.pipekitClusterId	string	`""`	Enter the pipekitClusterId provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank.
secrets.pipekitSecretAccessKey	string	`""`	Enter the pipekitSecretAccessKey provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank.
serviceAccount.annotations	object	`{}`	Add any additional annotations to the Pipekit Agent ServiceAccount and ServiceAccount Token Secret.
serviceAccount.create	bool	`true`	Specifies whether a ServiceAccount should be created. If false, you must provide an existing ServiceAccount name.
serviceAccount.name	string	`""`	Specifies the name of the ServiceAccount to create if serviceAccount.create is true. Otherwise, specifies the name of an existing ServiceAccount to use.

configMap.additionalConfig

object

{}

configMap.allowExternalWorkflows

bool

false

Allow Pipekit Agent to monitor workflows created without Pipekit. This is disabled by default.

configMap.annotations

object

{}

Add any additional annotations to the Pipekit Agent ConfigMap.

configMap.externalWorkflowsCheckPeriod

string

"10s"

The period at which the Pipekit Agent checks for external workflows. This is set to 10s by default.

configMap.externalWorkflowsMaxWorkers

string

The maximum number of workers that the Pipekit Agent can use to execute external workflows.

configMap.externalWorkflowsNamespaces

string

All namespaces

The list of comma-separated namespaces where the Pipekit Agent should look for external workflows. Notes: Pipekit Agent needs appropriate permissions to get, list, patch and delete workflows in the defined namespaces.

configMap.logLevel

string

"INFO"

Set the logLevel for the Pipekit Agent.

configMap.name

string

""

Specifies the name of the ConfigMap to create. Leave blank to auto-generate a name.

configMap.sendLogsToPipekit

bool

true

Send Workflow logs to Pipekit. Pipekit will store and index logs for you to view at pipekit.io

configMap.shouldDeleteExternalWorkflows

bool

false

Should the Pipekit Agent delete external workflows after execution. This is set to false by default.

configMap.telemetryProtocols

string

"prometheus"

set to 'none' to disable all telemetry

deployment.image.imagePullSecrets

list

[]

Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent image.

deployment.image.pullPolicy

string

"IfNotPresent"

The imagePullPolicy for the Pipekit Agent image.

deployment.image.repository

string

"pipekit13/pipekit-agent"

The Pipekit Agent image name and repository. Change this if you wish to host the container yourself.

deployment.image.tag

string

""

Allows you to pin to a specific image tag. The Chart.yaml contains a default value.

deployment.nodeSelector

object

{"kubernetes.io/os":"linux"}

The Pipekit Agent pod's node selector.

deployment.podAnnotations

object

{}

Add any additional annotations to the Pipekit Agent pod.

deployment.podLabels

object

{}

Add any additional labels to the Pipekit Agent pod.

deployment.resources

object

{}

Set the Pipekit Agent pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage.

features

object

{"metrics":{"argoWfControllerLabels":{},"argoWfMetricsPath":"/metrics","argoWfMetricsPort":9090,"argoWfNamespace":"argo","collectorLogLevel":"info","collectorNodeSelector":{"kubernetes.io/os":"linux"},"collectorResources":{"requests":{"cpu":"80m","memory":"256Mi"}},"deployment":{"image":{"imagePullSecrets":[],"pullPolicy":"IfNotPresent","repository":"pipekit13/agent","tag":""},"replicas":1,"resources":{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}},"enabled":false,"k8sAuthType":"","otelCollectorImage":"","otelCollectorTag":""},"workflows":{"enabled":true}}

Enable and Disable features of the Pipekit Agent.

features.metrics.argoWfControllerLabels

object

{}

the labels we should use to select the workflow controller. Defaults to app: workflow-controller if not set (suitable for the official Workflows release manifest)

features.metrics.argoWfMetricsPath

string

"/metrics"

The path where your Argo Workflows controller exposes metrics.

features.metrics.argoWfMetricsPort

int

9090

The port number of your Argo Workflows controller that exposes metrics.

features.metrics.argoWfNamespace

string

"argo"

The namespace where your Argo Workflows controller is running. This tells Pipekit Agent where to find the Workflow metrics.

features.metrics.collectorLogLevel

string

"info"

The Pipekit Agent Metrics Collector will log at this level.

features.metrics.collectorNodeSelector

object

{"kubernetes.io/os":"linux"}

A nodeselector for the opentelemetry collector

features.metrics.collectorResources

object

{"requests":{"cpu":"80m","memory":"256Mi"}}

Resource requests and limits for the collector

features.metrics.deployment.image.imagePullSecrets

list

[]

Allows you to define the name of existing imagePullSecrets to use for pulling the Pipekit Agent Operator image.

features.metrics.deployment.image.pullPolicy

string

"IfNotPresent"

The imagePullPolicy for the Pipekit Agent Operator image.

features.metrics.deployment.image.repository

string

"pipekit13/agent"

The Pipekit Agent Operator image name and repository. Change this if you wish to host the container yourself.

features.metrics.deployment.image.tag

string

""

Allows you to pin to a specific image tag. The Chart.yaml contains a default value.

features.metrics.deployment.replicas

int

1

The number of replicas for the Pipekit Agent Metrics Collector.

features.metrics.deployment.resources

object

{"limits":{"cpu":"200m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}

Set the Pipekit Agent Controller Manager pod's resource requests and limits. We suggest a minimal amount of resources below, but you should increase these if needed as actual resource usage will depend on usage.

features.metrics.enabled

bool

false

Enable and Disable Pipekit Workflow metrics. This passes Workflow metrics from your cluster to Pipekit in order to view them in the Workflow Metrics dashboard. # You must sign up here to use this feature or email hello@pipekit.io.

features.metrics.k8sAuthType

string

""

kubernetes authentication method, defaults to serviceAccount alternative setting is none

features.metrics.otelCollectorImage

string

""

The docker name of the opentelemetry contrib collector image to deploy

features.metrics.otelCollectorTag

string

""

The tag of the collector image to use

features.workflows

object

{"enabled":true}

Enable and Disable the Pipekit Agent. This communicates with Pipekit and allows you to run and manage your workflows.

fullnameOverride

string

""

Completely replace the generated name with the provided name.

metrics.enabled

bool

false

Enables a Service and ServiceMonitor for Prometheus metrics.

nameOverride

string

""

Replaces the name of the chart in Chart.yaml.

secrets.annotations

object

{}

Add any additional annotations to the Pipekit Agent Secret. Requires secrets.existingSecret to be blank.

secrets.existingSecret

string

""

The name of an existing secret containing your pipekitSecretAccessKey and pipekitClusterId.

secrets.name

string

""

Specifies the name of the Secret to create. Leave blank to auto-generate a name. Requires secrets.existingSecret to be blank.

secrets.pipekitClusterId

string

""

Enter the pipekitClusterId provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank.

secrets.pipekitSecretAccessKey

string

""

Enter the pipekitSecretAccessKey provided by Pipekit when you added the cluster. Requires secrets.existingSecret to be blank.

serviceAccount.annotations

object

{}

Add any additional annotations to the Pipekit Agent ServiceAccount and ServiceAccount Token Secret.

serviceAccount.create

bool

true

Specifies whether a ServiceAccount should be created. If false, you must provide an existing ServiceAccount name.

serviceAccount.name

string

""

Specifies the name of the ServiceAccount to create if serviceAccount.create is true. Otherwise, specifies the name of an existing ServiceAccount to use.

Secrets

If you wish to provide your own secret you need to populate it with the pipekitSecretAccessKey and pipekitClusterId values, using the defined data keys below:

Chart var

.data. in Secret

secrets.pipekitSecretAccessKey

PIPEKIT_SECRET_ACCESS_KEY

secrets.pipekitClusterId

PIPEKIT_CLUSTER_ID

eg:

kubectl -n argo create secret generic pipekit-agent \
    --from-literal=PIPEKIT_SECRET_ACCESS_KEY='abc123' \
    --from-literal=PIPEKIT_CLUSTER_ID='123abc'

You must use single quotes '' to escape special characters such as $, , *, =, and ! in your strings. If you don't, your shell will interpret these characters.

Enabling Externally Triggered Workflows

If you wish to enable externally triggered workflows, you can set the following values in your Helm install command:

kubectl create ns argo
helm repo add pipekit https://helm.pipekit.io

helm upgrade -i -n argo \
pipekit-agent pipekit/pipekit-agent \
--set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
--set secrets.pipekitClusterId="[provided Cluster ID]" \
--set configMap.allowExternalWorkflows=true

You can additionally set configMap.externalWorkflowsCheckPeriod, configMap.externalWorkflowsNamespace and configMap.shouldDeleteExternalWorkflows. Refer to the values table above for more information.

Pipekit Agent Cluster Permissions

The Pipekit Agent needs to interact with Argo Workflows on your cluster. In order to do this, we create a ServiceAccount and ClusterRole for the Pipekit Agent. If you wish to manually manage this ServiceAccount and ClusterRole, you can set serviceAccount.create to false and provide the name of an existing ServiceAccount to use.

The required minimum permissions are:

  - apiGroups:
      - argoproj.io
    resources:
      - clusterworkflowtemplates
      - workflows
      - cronworkflows
      - workflowtemplates
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch

Upgrading Pipekit Agent

Checking the latest version

Pipekit Agent is automatically published to Artifact Hub. You can use this service to configure automatic notifications of new versions, either via an RSS feed or a webhook.

Alternatively, you can search the helm repo for the latest version of the Pipekit Agent using the following command:

# Add the repo if you don't already have it.
helm repo add pipekit https://helm.pipekit.io
helm repo update
helm search repo pipekit/pipekit-agent --versions

For further information on the helm search repo command, please refer to the official Helm documentation.

Changes to the default values.yaml

Prior to upgrading, you should ensure that you understand any changes to the default values.yaml and the impact those changes may have on your installation. This page is always updated with the latest available helm chart values for the Pipekit Agent.

If you wish to upgrade to an older version of the Pipekit Agent helm chart, you can extract the default values for that version using the following command:

helm show values pipekit/pipekit-agent --version [version]

The Pipekit-Agent values file is commented so you can see what each value does.

You can extract your existing values from your current installation using the following command:

helm get values pipekit-agent -n argo

For further information the helm commands used above, please refer to the official Helm documentation.

Upgrading the Pipekit Agent

To upgrade the Pipekit Agent, you can use the following command:

helm upgrade pipekit-agent pipekit/pipekit-agent \
--set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
--set secrets.pipekitClusterId="[provided Cluster ID]" \
-n argo

For more information on using Helm to perform upgrades, please check the official Helm documentation.

Automating the upgrades using Gitops

If you use a Gitops tool such as Argo CD, you can simply commit your changes to your git repository and the tool will handle the Helm upgrade for you.

If you wish to automate the upgrade process, we recommend a third party tool called Renovate Bot that can be configured to automatically raise pull requests for you when a new version of the Pipekit Agent is released.

Upgrade Support

If you have an issue upgrading your Pipekit Agent that isn't addressed here, please contact us over Slack, or by email at support@pipekit.io.

Last updated 1 month ago