Getting Started

How to run a pipeline on Pipekit

Prerequisites

To get started with Pipekit, you will need the following items available:

• A Windows, Linux or Mac computer to run the Pipekit CLI.

Optional Prerequisites

• A fork of our Pipekit Examples Github repository.

• Python and Hera installed locally.

Create an account

1. Go to https://pipekit.io and click, "Get started free"

2. Complete the form or use SSO (Single Sign On) to create an account.

3. Check your email for a welcome email containing a link to confirm your account.

4. Click the link to confirm your account and log in.

Create an Organization

1. You will be taken to the "Org" page. Click "Create an Org".

2. When prompted, give a name and description for the organization, and click, "Submit".

Connect a Cluster

Now that you've created an organization, we will connect Pipekit to the Kubernetes cluster(s) where we want to run our Workflows.

Free Trial Cluster

You will need a Kubernetes cluster with Argo Workflows already installed. During your free trial period, we can provide you with a small cluster to use. This cluster will have Argo Workflows pre-installed and the Pipekit Agent will be set up for you.

1. When prompted, confirm that you wish to use a Pipekit-hosted free trial cluster. This process will take 2-3 minutes to complete.

2. When the cluster has successfully provisioned, you will be redirected to the Pipes page to configure a pipeline.

Bring your own Cluster

1. Have a Kubernetes cluster with Argo Workflows already installed.
2. Decline the offer for a Pipekit-hosted cluster and click the "Connect A Cluster" button to be taken to the connect cluster page.

3. Select the organization you want to use to connect the cluster.

4. Add the name for the cluster. For ease of use, we recommend keeping the name all lower case with no whitespace.

5. Add a description.

6. Click, "Submit".

7. Follow the on-screen commands to install Pipekit Agent into your cluster.

8. Upon successful installation, you will be redirected to the Pipes page to configure a pipeline.

Configure and run a Pipe

In Pipekit, we use the term Pipe to refer to an Argo Workflow and all of the runs associated with that Workflow. We can create and run Pipes using four methods:

1. The Pipekit CLI;

2. Directly in a Hera Workflow;
3. By connecting to a GitHub/Gitlab account and triggering Workflow runs based on changes to the repository (gitops);

4. By configuring the Pipekit Agent to consume externally-triggered Workflows from your cluster(s).

Click "Create a Pipe" and then choose to use either the CLI or the web interface:

CLI

If you don't have a Workflow available, fork our Pipekit Examples Github repository.

The CLI is the quickest method to run a Pipe. Take the following steps:

1. Install the Pipekit CLI on your machine. Instructions are found here.
2. Navigate to an Argo Workflows file, such as one from the Pipekit Examples Github repository.
3. Login through the CLI using pipekit login

4. Submit the Argo Workflow to the cluster you connected above using the command pipekit submit --cluster-name=your-cluster-name examples/examples/dag-diamond/workflow.yaml

CLI - Hera

More information on running Hera Workflows can be found here.

Web

If you select to create a Pipe using the web interface, you will be prompted to connect to your organization's GitHub or Gitlab account.

GitHub

1. Click the button for GitHub. It will lead you through the process to authorize Pipekit's GitHub App.

2. When prompted by GitHub, select either, "All repositories" or select the list of repositories you would like to grant Pipekit access to using the dropdown.

3. Click, "Authorize".

Gitlab

For Gitlab, you will need to create a Personal Access Token. We recommend creating a service account user in your organization on Gitlab that has access to any repositories that you want Pipekit to read from.

To connect Pipekit, click the button for Gitlab, input the Personal Access Token that you created, optionally change the gitlab URL and click, "Submit".

Creating a Pipe

After connecting to GitHub/Gitlab, you can now create a Pipe using the Create Pipe page. Configure the pipeline by filling out the form using these steps:

1. Add a name for your pipeline. For now, let's call our pipeline "DAG Diamond"

2. Add a description. We'll use this description: "A directed, acyclic graph"

3. Select the repo where your workflow yaml file lives on GitHub/Gitlab

4. Enter the location of the Workflow file. We'll be using workflow.yaml for this example.
5. In the "Run Conditions" section, specify when a pipeline should run. Right now, set the condition to be Branch and main. You can also run pipelines off Git tags and use regex matching.

6. Click submit to create your pipeline

Run your pipeline

On your computer, clone to the repo you used in the section above. Modify the examples/examples/dag-diamond/workflow.yaml file, commit the change, and push it to your main branch.

Then click on the "Dag Diamond" pipeline in the pipes page. You will then see a list of pipeline runs that now has one entry. Click where it says "BRANCH: main" to see the pipeline result and the log output.

Best Practice

It is best practice to set a Namespace and Service Account for your Argo Workflow. Depending on how your organization has the cluster set up, it may be mandatory. If you submit a workflow without a namespace or service account, the Pipekit Agent will use the default namespace and service account for the cluster, which may not have the permissions needed to run the workflow.

The workflows provided in the Pipekit Examples Github repository have the namespace set to argo and the Service Account Name set to argo-workflow for your convenience.