PipekitPricingBlogPipekit StatusRelease Notes

Dependencies and Pre-requisites

Pre-requisites for self-hosting Pipekit

Pipekit depends on the following tools being available in your environment:

Kubernetes

  • We recommend Kubernetes 1.33 or greater.

Setup

  • The installation makes use of ingresses to route traffic to the various services. You will need to have an ingress controller installed in your cluster. We recommend using the ingress-nginx controller.

  • OTEL or Prometheus will allow you to monitor the pipekit services on their defined metrics ports. This is not a strict requirement but is recommended.

  • An in-cluster log collection tool such as Fluentd and an aggregation tool (e.g. Loki) is recommended to collect and display logs from the various services. This is not a strict requirement but is recommended.

  • The Pipekit services make use of HPAs to scale under load. You will need to have the metrics server installed in your cluster in order to make use of this.

Postgres

  • We recommend you use a managed Postgres service such as AWS RDS or Azure Postgres.

  • We recommend Postgres 17 or greater.

Setup

  • The Pipekit services require a Postgres database to store their data. You will need to create a database and user for the id and users services to use:

    CREATE DATABASE users;
CREATE DATABASE id;

Redis

  • Can be inside the cluster or a managed service such as AWS Elasticache. Internally, we use Valkey as a Redis compatible database.

  • We recommend Redis or Valkey 8 or greater running in a high availability setup with sentinel.

  • We recommend configuring disk persistence for Redis. However, there is no need to back up this disk. If the data is lost, Pipekit will recreate it, but in-flight workflows may not have the correct state in Pipekit if Redis' data is lost at the time the workflow is being processed.

Setup

No special configuration is required for Redis. We would recommend that it is configured to be highly available. It would also be advisable to configure Redis to use a password which can be passed to the Pipekit Helm Chart.

Grafana Loki

  • Can be hosted within the Kubernetes cluster, or externally.

  • We recommend Loki 3.5 or greater.

Setup

No special configuration is required for Loki. We would recommend that it is configured to be highly available.

Git Provider (GitHub, GitLab)

You can optionally connect your Git provider to Pipekit in order to make use of the GitOps features. While this is optional, it is highly recommended.

Setup

Follow the instructions in the Git Provider documentation to connect your Git provider to Pipekit.

Last updated