Pipekit API

Pipekit Docs

Search…

Pipekit Docs

Pipekit Quickstart

Pipekit API

The Pipekit API allows you to manage your Argo Workflows deployments across multiple clusters.

Overview
The Pipekit API provides a series of endpoints to enable the following: 
triggering and terminating Argo Workflows across multiple clusters
fetching logs for workflow runs and pods in a workflow run
adding, updating, or removing tags from clusters managed by Pipekit 
Flow of events
1.User makes sends request to Pipekit API.
1.Payload contains i) JSON representation of an Argo workflow YAML, ii) Pipekit configurations
2.Pipekit reads request body and routes workflow definition to a message queue for the cluster specified in the Pipekit block.
3.Target cluster reads from message queue and executes the workflow in the target namespace.
4.Target cluster sends logs and workflow state back to Pipekit.
5.Target cluster waits for workflow to complete and cleans up all resources created during the workflow run.
Definitions
Pipe - We refer to workflows as "pipes" in Pipekit. Pipes can be shared across clusters, users, and teams. i.e. pipe-id represents to the unique ID of each workflow created in Pipekit.
Workflows Endpoints
Use the following endpoints to trigger or terminate workflow runs across the clusters you 
Run workflow
This method runs a workflow on a given cluster and with a given secrets environment. It takes a workflow specification object that defines the workflow to be run.
1
POST https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs
Copied!
1
{
2
    "pipekit": {
3
   	 "pipeName": "ETL Pipeline",
4
   	 "cluster": "Customer A",
5
   	 "secretsEnvironment": "Builder",
6
   	 "runName": "ETL run uuid",
7
   	 "namespace": "target namespace",
8
     "tags": ["customerX", "encoderA", "urgent"]
9
    },
10
    "argo": {
11
   	 "apiVersion": "argoproj.io/v1alpha1",
12
   	 "kind": "Workflow",
13
   	 "metadata": {
14
   		 "generateName": "init-container-"
15
   	 },
16
   	 "spec": {
17
   		 "entrypoint": "init-container-example",
18
   		 "templates": [{
19
   			 "name": "init-container-example",
20
   			 "container": {
21
   				 "image": "alpine:latest",
22
   				 "command": [
23
   					 "echo",
24
   					 "bye"
25
   				 ],
26
   				 "volumeMounts": [{
27
   					 "name": "foo",
28
   					 "mountPath": "/foo"
29
   				 }]
30
   			 },
31
   			 "initContainers": [{
32
   				 "name": "hello",
33
   				 "image": "alpine:latest",
34
   				 "command": [
35
   					 "echo",
36
   					 "hello"
37
   				 ],
38
   				 "mirrorVolumeMounts": true
39
   			 }]
40
   		 }],
41
   		 "volumes": [{
42
   			 "name": "foo",
43
   			 "emptyDir": {}
44
   		 }]
45
   	 }
46
    }
47
}
Copied!
post
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs
Run workflow
The argo object houses the JSON representation of an Argo workflow YAML. It follows the official specification declared by argo workflows here and is converted into JSON rather than YAML.
Stop workflow
Stop a running workflow. This is useful if there is a workflow that is hanging.
delete
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Stop workflow
1
DELETE https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Copied!
Logs and Analytics Endpoints
Get workflow logs
Logs can be fetched at both the global workflow level and then also for the pods in a workflow run.
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=workflow
Get workflow logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=pods
Get pod logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/runs/?tags={parameterA},{parameterB}
Get runs by tags
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Get run status
get
https://pipekit.io/api/users
/v1/users/{user-id}/usage?tags=customerA,urgent&start-date=YYYY-MM-DD&end-date=YYYY-MM-DD
Get resources usage
Cluster Endpoints
Clusters will be managed by a Pipekit daemon that runs as a deployment on each cluster. It will listen to a dedicated SQS instance for that cluster that Pipekit will send messages to in order to schedule runs. This prevents the cluster from having to be exposed to the internet.
get
https://pipekit.io/api/users
/v1/users/{user-id}/clusters
Get cluster list
put
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Update cluster
delete
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Disconnect cluster