Pipekit API
The Pipekit API allows you to manage your Argo Workflows deployments across multiple clusters.
The Pipekit API provides a series of endpoints to enable the following:
- triggering and terminating Argo Workflows across multiple clusters
- fetching logs for workflow runs and pods in a workflow run
- adding, updating, or removing tags from clusters managed by Pipekit
- 1.User makes sends request to Pipekit API.
- 1.Payload contains i) JSON representation of an Argo workflow YAML, ii) Pipekit configurations
- 2.Pipekit reads request body and routes workflow definition to a message queue for the cluster specified in the Pipekit block.
- 3.Target cluster reads from message queue and executes the workflow in the target namespace.
- 4.Target cluster sends logs and workflow state back to Pipekit.
- 5.Target cluster waits for workflow to complete and cleans up all resources created during the workflow run.
- Pipe - We refer to workflows as "pipes" in Pipekit. Pipes can be shared across clusters, users, and teams. i.e.
pipe-idrepresents to the unique ID of each workflow created in Pipekit.
Use the following endpoints to trigger or terminate workflow runs across the clusters you
This method runs a workflow on a given cluster and with a given secrets environment. It takes a workflow specification object that defines the workflow to be run.
POST https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs
{
"pipekit": {
"pipeName": "ETL Pipeline",
"cluster": "Customer A",
"secretsEnvironment": "Builder",
"runName": "ETL run uuid",
"namespace": "target namespace",
"tags": ["customerX", "encoderA", "urgent"]
},
"argo": {
"apiVersion": "argoproj.io/v1alpha1",
"kind": "Workflow",
"metadata": {
"generateName": "init-container-"
},
"spec": {
"entrypoint": "init-container-example",
"templates": [{
"name": "init-container-example",
"container": {
"image": "alpine:latest",
"command": [
"echo",
"bye"
],
"volumeMounts": [{
"name": "foo",
"mountPath": "/foo"
}]
},
"initContainers": [{
"name": "hello",
"image": "alpine:latest",
"command": [
"echo",
"hello"
],
"mirrorVolumeMounts": true
}]
}],
"volumes": [{
"name": "foo",
"emptyDir": {}
}]
}
}
}
post
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs
Run workflow
The
argo object houses the JSON representation of an Argo workflow YAML. It follows the official specification declared by argo workflows here and is converted into JSON rather than YAML.Stop a running workflow. This is useful if there is a workflow that is hanging.
delete
https://pipekit.io/api/events-handler
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Stop workflow
DELETE https://pipekit.io/api/events-handler/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Logs can be fetched at both the global workflow level and then also for the pods in a workflow run.
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=workflow
Get workflow logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}/logs?type=pods
Get pod logs
get
https://pipekit.io/api/users
/v1/users/{user-id}/runs/?tags={parameterA},{parameterB}
Get runs by tags
get
https://pipekit.io/api/users
/v1/users/{user-id}/pipes/{pipe-id}/runs/{run-id}
Get run status
get
https://pipekit.io/api/users
/v1/users/{user-id}/usage?tags=customerA,urgent&start-date=YYYY-MM-DD&end-date=YYYY-MM-DD
Get resources usage
Clusters will be managed by a Pipekit daemon that runs as a deployment on each cluster. It will listen to a dedicated SQS instance for that cluster that Pipekit will send messages to in order to schedule runs. This prevents the cluster from having to be exposed to the internet.
get
https://pipekit.io/api/users
/v1/users/{user-id}/clusters
Get cluster list
put
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Update cluster
delete
https://pipekit.io/api/users
/v1/users/{user-id}/clusters/{cluster_id}
Disconnect cluster
Last modified 1yr ago