Bring Your Own Logging

Pipekit BYO Logging Backend lets you collect and view workflow logs in a self-hosted Loki backend. This allows you to have the full logging experience managed by Pipekit without having Pipekit ever ac

Pipekit allows you to connect your self-hosted Loki instance to the Pipekit control plane. Once you add your Loki configuration to Pipekit, you will have to use a log collector on your Kubernetes clusters in order to send the logs from your cluster to your Loki instance.

Connect a Loki Instance

Go to the Org tab in Pipekit. Then navigate to the BYO Logs tab. Enter the address of your Loki instance along with a username and password that you use for authenticating at the gateway/ingress layer. Then submit the information. You'll have to make sure that the address is accessible to the Internet and that the following endpoints are reachable:

/loki/api/v1/query
/loki/api/v1/query_range
/loki/api/v1/tail
/loki/api/v1/series

Configuring a Log Collector

In order to get the logs from your Kubernetes clusters running Argo Workflows into your Loki instance, you'll have to configure a log collector. We use and test with Fluent Bit.

Labels

Pipekit appends three labels to pods run by workflows on clusters where the Pipekit agent is present and configured to either submit workflows or read externally submitted workflows. This works for both Pipekit-submitted and externally submitted workflows. The labels are the following:

workflows.pipekit.io/org_uuid
workflows.pipekit.io/pipe_uuid
workflows.pipekit.io/run_uuid

Map these to orgUUID, pipeUUID, and runUUID in Loki.

Pipekit also reads three more labels that your log collector must set on each log stream: nodeId, podName, and containerName. podName is the name of the pod and containerName is the name of the container. nodeId is the Argo workflow node ID. Argo sets this ID on each pod as the workflows.argoproj.io/node-id annotation, so read the value from that annotation.

nodeId must hold the Argo workflow node ID, not the name of the Kubernetes cluster node that the pod ran on. Pipekit matches logs to a workflow step by nodeId. If you map it to the cluster node name, the logs for that step do not appear in Pipekit.

Fluent Bit Config

Here's a sample fluent bit config that we'd recommend. Note that this configuration reads logs directly from the container file system:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-config
  namespace: <<REPLACE_WITH_NAMESPACE>>
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
    @INCLUDE input.conf
    @INCLUDE output.conf

  input.conf: |
    [INPUT]
        Name                tail
        Parser              json
        Path                /tmp/*.log # set to whatever path your logs will be stored in the filesystem
        Mem_Buf_Limit       100MB
        Skip_Long_Lines     On
        Refresh_Interval    5
        Rotate_Wait         15

  output.conf: |
    [OUTPUT]
        name                    loki
        match                   *
        host                    <<REPLACE_WITH_LOKI_HOST>>
        port                    <<REPLACE_WITH_LOKI_PORT>>
        labels                  orgUUID=$orgUUID, pipeUUID=$pipeUUID, runUUID=$runUUID, podName=$podName, containerName=$containerName, nodeId=$nodeId
        tenant_id_key           orgUUID # this line is only needed if you're doing multi-tenancy
        remove_keys             orgUUID, pipeUUID, runUUID, podName, containerName, createdAt, nodeId
        line_format             key_value # must be set to key_value, otherwise character escapes will appear
        drop_single_key         on

  parsers.conf: |
    [PARSER]
        Name        json
        Format      json