> For the complete documentation index, see [llms.txt](https://docs.pipekit.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pipekit.io/pipekit/organization/byo-logs.md).

# Managing Bring Your Own Logging Backend

Pipekit allows you to connect your self-hosted Loki instance to the Pipekit control plane. Once you add your Loki configuration to Pipekit, you will have to use a log collector on your Kubernetes clusters in order to send the logs from your cluster to your Loki instance.

## Connect a Loki Instance

Go to the [Org tab in Pipekit](https://pipekit.io/organization). Then navigate to the `BYO Logs` tab. Enter the address of your Loki instance along with a username and password that you use for authenticating at the gateway/ingress layer. Then submit the information. You'll have to make sure that the address is accessible to the Internet and that the following endpoints are reachable:

```yaml
/loki/api/v1/query
/loki/api/v1/query_range
/loki/api/v1/tail
/loki/api/v1/series
```

## Configuring a Log Collector

In order to get the logs from your Kubernetes clusters running Argo Workflows into your Loki instance, you'll have to configure a log collector. We use and test with [Fluent Bit](https://fluentbit.io/).

### Labels

Pipekit will append the following labels to pods ran by workflows on clusters where the Pipekit agent is present and configured to either submit workflows or read externally submitted workflows. The labels are the following:

```yaml
workflows.pipekit.io/org_uuid
workflows.pipekit.io/pipe_uuid
workflows.pipekit.io/run_uuid
```

These labels will need to be mapped to `orgUUID`, `pipeUUID`, and `runUUID` in Loki.

### Fluent Bit Config

Here's a sample fluent bit config that we'd recommend. Note that this configuration reads logs directly from the container file system:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-config
  namespace: <<REPLACE_WITH_NAMESPACE>>
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
    @INCLUDE input.conf
    @INCLUDE output.conf

  input.conf: |
    [INPUT]
        Name                tail
        Parser              json
        Path                /tmp/*.log # set to whatever path your logs will be stored in the filesystem
        Mem_Buf_Limit       100MB
        Skip_Long_Lines     On
        Refresh_Interval    5
        Rotate_Wait         15

  output.conf: |
    [OUTPUT]
        name                    loki
        match                   *
        host                    <<REPLACE_WITH_LOKI_HOST>>
        port                    <<REPLACE_WITH_LOKI_PORT>>
        labels                  orgUUID=$orgUUID, pipeUUID=$pipeUUID, runUUID=$runUUID, podName=$podName, containerName=$containerName, nodeId=$nodeId
        tenant_id_key           orgUUID # this line is only needed if you're doing multi-tenancy
        remove_keys             orgUUID, pipeUUID, runUUID, podName, containerName, createdAt, nodeId
        line_format             key_value # must be set to key_value, otherwise character escapes will appear
        drop_single_key         on

  parsers.conf: |
    [PARSER]
        Name        json
        Format      json
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.pipekit.io/pipekit/organization/byo-logs.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.