Jupyter Notebooks

Using the Pipekit Python SDK in Jupyter Notebooks

Get a token

Use the Pipekit CLI to generate a token using pipekit hera (ensure you are logged in first).

You can then pass this token to a PipekitService:

from pipekit_sdk.service import PipekitService

# Create a Pipekit service that is used to talk to the Pipekit API
pipekit = PipekitService(token="<token>")

Submission

Create a Workflow to submit:

from hera.workflows import DAG, Workflow, script

@script(image="python:3.12")
def echo(message):
    print(message)

with Workflow(
    generate_name="dag-diamond-",
    entrypoint="diamond",
    namespace="argo",
    service_account_name="argo-workflow",
) as w:
    with DAG(name="diamond"):
        A = echo(name="A", arguments={"message": "A"})
        B = echo(name="B", arguments={"message": "B"})
        C = echo(name="C", arguments={"message": "C"})
        D = echo(name="D", arguments={"message": "D"})
        A >> [B, C] >> D

Then use submit to run the Workflow on Pipekit, remember to pass in your cluster name.

# Submit the Workflow to Pipekit
pipe_run = pipekit.submit(w, "<cluster-name>")
print("pipe_run", pipe_run, "\n")

Log Streaming

After you have submitted a Workflow as above, you will be holding a "pipe_run" - you can stream the logs to your Notebook using print_logs on the uuid:

pipekit.print_logs(pipe_run["uuid"])