Managing Bring Your Own Logging Backend

Pipekit BYO Logging Backend lets you collect and view workflow logs in a self-hosted Loki backend. This allows you to have the full logging experience managed by Pipekit without having Pipekit ever ac

Pipekit allows you to connect your self-hosted Loki instance to the Pipekit control plane. Once you add your Loki configuration to Pipekit, you will have to use a log collector on your Kubernetes clusters in order to send the logs from your cluster to your Loki instance.

Connect a Loki Instance

Go to the Org tab in Pipekit. Then navigate to the BYO Logs tab. Enter the address of your Loki instance along with a username and password that you use for authenticating at the gateway/ingress layer. Then submit the information. You'll have to make sure that the address is accessible to the Internet and that the following endpoints are reachable:

/loki/api/v1/query
/loki/api/v1/query_range
/loki/api/v1/tail
/loki/api/v1/series

Configuring a Log Collector

In order to get the logs from your Kubernetes clusters running Argo Workflows into your Loki instance, you'll have to configure a log collector. We use and test with Fluent Bit.

Labels

Pipekit will append the following labels to pods ran by workflows on clusters where the Pipekit agent is present and configured to either submit workflows or read externally submitted workflows. The labels are the following:

workflows.pipekit.io/org_uuid
workflows.pipekit.io/pipe_uuid
workflows.pipekit.io/run_uuid

These labels will need to be mapped to orgUUID, pipeUUID, and runUUID in Loki.

Fluent Bit Config

Here's a sample fluent bit config that we'd recommend. Note that this configuration reads logs directly from the container file system:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-config
  namespace: <<REPLACE_WITH_NAMESPACE>>
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
    @INCLUDE input.conf
    @INCLUDE output.conf

  input.conf: |
    [INPUT]
        Name                tail
        Parser              json
        Path                /tmp/*.log # set to whatever path your logs will be stored in the filesystem
        Mem_Buf_Limit       100MB
        Skip_Long_Lines     On
        Refresh_Interval    5
        Rotate_Wait         15

  output.conf: |
    [OUTPUT]
        name                    loki
        match                   *
        host                    <<REPLACE_WITH_LOKI_HOST>>
        port                    <<REPLACE_WITH_LOKI_PORT>>
        labels                  orgUUID=$orgUUID, pipeUUID=$pipeUUID, runUUID=$runUUID, podName=$podName, containerName=$containerName, nodeId=$nodeId
        tenant_id_key           orgUUID # this line is only needed if you're doing multi-tenancy
        remove_keys             orgUUID, pipeUUID, runUUID, podName, containerName, createdAt, nodeId
        line_format             key_value # must be set to key_value, otherwise character escapes will appear
        drop_single_key         on

  parsers.conf: |
    [PARSER]
        Name        json
        Format      json