Pipekit Docs
Search…
Pipekit API
The Pipekit API allows you to manage your Argo Workflows deployments across multiple clusters.

Overview

The Pipekit API provides a series of endpoints to enable the following:
  • triggering and terminating Argo Workflows across multiple clusters
  • fetching logs for workflow runs and pods in a workflow run
  • adding, updating, or removing tags from clusters managed by Pipekit

Flow of events

  1. 1.
    User makes sends request to Pipekit API.
    1. 1.
      Payload contains i) JSON representation of an Argo workflow YAML, ii) Pipekit configurations
  2. 2.
    Pipekit reads request body and routes workflow definition to a message queue for the cluster specified in the Pipekit block.
  3. 3.
    Target cluster reads from message queue and executes the workflow in the target namespace.
  4. 4.
    Target cluster sends logs and workflow state back to Pipekit.
  5. 5.
    Target cluster waits for workflow to complete and cleans up all resources created during the workflow run.

Definitions

  • Pipe - We refer to workflows as "pipes" in Pipekit. Pipes can be shared across clusters, users, and teams. i.e. pipe-id represents to the unique ID of each workflow created in Pipekit.

Workflows Endpoints

Use the following endpoints to trigger or terminate workflow runs across the clusters you

Run workflow

This method runs a workflow on a given cluster and with a given secrets environment. It takes a workflow specification object that defines the workflow to be run.
1
POST https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs
Copied!
1
{
2
"pipekit": {
3
"pipeName": "ETL Pipeline",
4
"cluster": "Customer A",
5
"secretsEnvironment": "Builder",
6
"runName": "ETL run uuid",
7
"namespace": "target namespace",
8
"tags": ["customerX", "encoderA", "urgent"]
9
},
10
"argo": {
11
"apiVersion": "argoproj.io/v1alpha1",
12
"kind": "Workflow",
13
"metadata": {
14
"generateName": "init-container-"
15
},
16
"spec": {
17
"entrypoint": "init-container-example",
18
"templates": [{
19
"name": "init-container-example",
20
"container": {
21
"image": "alpine:latest",
22
"command": [
23
"echo",
24
"bye"
25
],
26
"volumeMounts": [{
27
"name": "foo",
28
"mountPath": "/foo"
29
}]
30
},
31
"initContainers": [{
32
"name": "hello",
33
"image": "alpine:latest",
34
"command": [
35
"echo",
36
"hello"
37
],
38
"mirrorVolumeMounts": true
39
}]
40
}],
41
"volumes": [{
42
"name": "foo",
43
"emptyDir": {}
44
}]
45
}
46
}
47
}
Copied!
post
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs
Run workflow
The argo object houses the JSON representation of an Argo workflow YAML. It follows the official specification declared by argo workflows here and is converted into JSON rather than YAML.

Stop workflow

Stop a running workflow. This is useful if there is a workflow that is hanging.
delete
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Stop workflow
1
DELETE https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Copied!

Logs and Analytics Endpoints

Get workflow logs

Logs can be fetched at both the global workflow level and then also for the pods in a workflow run.
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=workflow
Get workflow logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=pods
Get pod logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/runs/?tags={parameterA},{parameterB}
Get runs by tags
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Get run status
get
https://pipekit.io/api/users
/v1/users/{user-id}/usage?tags=customerA,urgent&start-date=YYYY-MM-DD&end-date=YYYY-MM-DD
Get resources usage

Cluster Endpoints

Clusters will be managed by a Pipekit daemon that runs as a deployment on each cluster. It will listen to a dedicated SQS instance for that cluster that Pipekit will send messages to in order to schedule runs. This prevents the cluster from having to be exposed to the internet.
get
https://pipekit.io/api/users
/v1/users/{user-id}/clusters
Get cluster list
put
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Update cluster
delete
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Disconnect cluster
Last modified 4mo ago