> For the complete documentation index, see [llms.txt](https://docs.pipekit.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pipekit.io/using-pipekit/operating-at-scale/vector-log-collection.md).

# Vector Log Collection

## Why Vector

The Pipekit Agent's built-in log transport is best-effort. It streams logs from each workflow pod via the Kubernetes API and forwards them to Pipekit. There is no write-ahead log and no retry. If shipment fails, those logs are gone.

Two failure modes follow.

*Log loss under API server pressure.* The agent reads logs through the Kubernetes API. Any sustained load on the API server can stall its log streams. Without a replay path, the agent does not recover those logs.

*Etcd cost at scale.* Streaming every container of every workflow pod through the Kubernetes API loads etcd. For workflows with many parallel pods, that pressure sets a practical ceiling on how big a Pipekit deployment can grow before the API server becomes the bottleneck.

Vector addresses both. It runs as a DaemonSet on every node and reads container logs directly from the node filesystem, never touching the Kubernetes API for log content. Vector buffers undelivered logs to disk (default 500 MB per node) and retries with backoff. Transient ingest failures no longer mean lost data.

The built-in agent path is fine at small scale. Switch to Vector once log loss or API-server load becomes a real concern.

## How it works

[Vector](https://vector.dev) is built by the Pipekit Agent operator, gated on `vector.enabled`. When enabled, the operator creates a DaemonSet named `pipekit-vector` with a ConfigMap that defines the full log pipeline.

Vector's `kubernetes_logs` source reads container log files directly from each node, filtered to pods carrying the `workflows.argoproj.io/workflow` label so the source picks up only Argo Workflow pods. A VRL transform extracts the Pipekit identifiers (`orgUUID`, `runUUID`, `pipeUUID`) from pod labels, the Argo workflow node ID from the `workflows.argoproj.io/node-id` pod annotation, and the pod name, container name, and log timestamp from Kubernetes metadata. A filter drops any record missing a required field. A final transform reshapes the remaining records to the field set the Pipekit messenger expects.

The HTTP sink batches up to 1024 events per request with a 1-second flush. It POSTs each batch to `{messengerBaseUri}/api/messenger/v1/logs/ingest/batch`, authenticated with `X-PIPEKIT-API-KEY`. A 500 MB on-disk buffer sits behind the sink: Vector retries delivery with backoff, and when the buffer fills, producers block rather than drop events.

## Enabling Vector

Set `vector.enabled=true` and disable the agent's built-in log path with `configMap.sendLogsToPipekit=false`:

```bash
helm upgrade -i -n argo \
  pipekit-agent pipekit/pipekit-agent \
  --set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
  --set secrets.pipekitClusterId="[provided Cluster ID]" \
  --set vector.enabled=true \
  --set configMap.sendLogsToPipekit=false
```

{% hint style="info" %}
When `vector.enabled=true`, the operator disables the agent's built-in log path regardless of `configMap.sendLogsToPipekit`. Set the flag to `false` anyway for hygiene, so your values match your intent.
{% endhint %}

## Self-hosted Pipekit

If you run a self-hosted Pipekit instance and have set `configMap.messengerBaseUri` to your own messenger URL, the chart automatically threads that value into Vector's HTTP sink. You do not need to configure the URL on the Vector side.

## Image pull secrets

If you mirror container images into a private registry, set `vector.image.pullSecrets` to a list of secret references the same way you would for the agent itself:

```yaml
vector:
  enabled: true
  image:
    pullSecrets:
      - name: my-registry-credentials
```

## Authenticating Vector

Vector inherits the agent's credentials automatically. Leave `vector.envFrom` empty and the operator mounts the agent's Secret for you, whether that Secret came from `secrets.existingSecret` or was created by the chart from `secrets.pipekitSecretAccessKey`. Vector's ConfigMap reads `${PIPEKIT_SECRET_ACCESS_KEY}` from that Secret at runtime.

Set `vector.envFrom` only when you want Vector to read a different Secret:

```yaml
vector:
  enabled: true
  envFrom:
    - secretRef:
        name: my-other-secret
```

{% hint style="info" %}
On charts before `7.0.0`, Vector ran as an upstream subchart and did not inherit the agent's Secret, so `vector.envFrom` had to repeat it. That entry is now redundant, though harmless if it names the agent's Secret. Refer to [Upgrading to the Operator-Managed Agent](/using-pipekit/clusters/agent-upgrade.md).
{% endhint %}

## Resources and tuning

Default Vector resources are conservative:

```yaml
vector:
  resources:
    requests:
      cpu: 30m
      memory: 130Mi
```

These defaults are enough for moderate log volume. For clusters running many large parallel workflows, increase CPU and memory. See [Vector's sizing guidance](https://vector.dev/docs/setup/going-to-prod/sizing/) for tuning under heavy load.

## Verifying it's working

1. Confirm the DaemonSet is Ready on every node:

   ```bash
   kubectl -n argo get ds
   ```
2. Port-forward to one Vector pod and check its local API on `127.0.0.1:8686`:

   ```bash
   kubectl -n argo port-forward ds/pipekit-vector 8686:8686
   curl localhost:8686/health
   ```
3. Run a workflow and confirm its logs appear in the Pipekit UI.

## Reference

For the full set of Vector-related Helm values, see the values table in [Pipekit Agent Helm Chart](/reference/helm-values/agent.md). The relevant rows are `vector.enabled`, `vector.affinity`, `vector.dataDir`, `vector.envFrom`, `vector.image.pullSecrets`, `vector.nodeSelector`, `vector.resources`, and `vector.tolerations`. When you leave `vector.tolerations` empty, Vector uses `scheduling.tolerations`, the chart-wide value. Vector never uses `scheduling.nodeSelector` or `scheduling.affinity`, because either one would stop the DaemonSet collecting logs on the nodes it skips. Leave `vector.nodeSelector` and `vector.affinity` empty for the same reason, unless you accept losing the logs of every workflow pod on the nodes you exclude. Keys that belonged to the old Vector subchart, including `vector.existingConfigMaps`, `vector.role`, and `vector.service`, were removed in chart `7.0.0` and are ignored.