Infrastructure as Code with OpenTofu

Use Pipekit and Argo Workflows to validate, plan, and apply infrastructure-as-code changes through your existing review pipeline.

The problem

OpenTofu and Terraform changes are high-blast-radius. A misconfigured security group or an unintended tofu destroy can take down production faster than almost any other class of change in a typical platform. The standard mitigations (security scanning, linting, tofu plan review, drift detection) are well understood, but stitching them together is not.

Most teams end up running tofu plan inside a generic CI runner (GitHub Actions, Jenkins, GitLab CI) and pasting the output into pull-request comments. That works for a single team and a single environment. It breaks down quickly: the plan output is hard to review at scale, the runner doesn't enforce mutual exclusion so two engineers can race each other into a locked state, nightly drift detection has nowhere natural to live, and the apply step turns into "whoever has the credentials on their laptop."

The teams that get this right tend to converge on the same shape: a real DAG, a real workflow engine, secrets the runner doesn't touch, mutual exclusion enforced at the engine level, and a separate path for plan vs apply. That shape is exactly what Argo Workflows gives you.

How Pipekit helps

Argo Workflows runs the DAG. Pipekit adds the surfaces an Argo deployment by itself doesn't have: a unified UI across Clusters, persisted Run history with searchable logs, per-environment secrets scoped to workspaces, Templates that let you share the apply step safely across teams, and alerting when drift detection fires.

The two worked examples below come from a real Pipekit customer's setup. The first is a pull-request validation Workflow that runs checkov, tflint, and tofu plan against three Terraform paths in parallel and posts the plan output as a PR comment. The second is a CronWorkflow that runs tofu plan -detailed-exitcode nightly to detect drift. Both are designed so the apply step is intentionally separate. See the Best practices section.

Worked example: Pull-request validation

When a pull request modifies infrastructure code, an automated Workflow validates the changes before merge.

The Workflow performs several validation steps:

1. Clone the infrastructure repository.

2. Run security scans to detect misconfigurations (checkov).

3. Run linting to enforce best practices (tflint).

4. Generate Terraform/OpenTofu plans showing what will change.

5. Post plan summaries as PR comments for review.

Workflow Structure

The PR validation Workflow consists of multiple tasks organized as a DAG (directed acyclic graph):

- name: main
  dag:
    tasks:
      - name: clone-repo
        template: clone-repo
      - name: checkov-scan
        template: checkov-scan
        depends: clone-repo
      - name: tflint
        template: tflint
        depends: clone-repo
      - name: tfplan
        template: tfplan
        depends: tflint
      - name: tfplan-to-comment
        template: tfplan-to-comment
        depends: tfplan

View complete PR validation workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: tf-pr-
  namespace: ci
spec:
  serviceAccountName: ci
  entrypoint: main
  synchronization:
    mutexes:
      - name: tf
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: nfs
      resources:
        requests:
          storage: 1Gi
  templates:
    - name: main
      dag:
        tasks:
          - name: clone-repo
            template: clone-repo
          - name: get-pr
            template: get-pr
          - name: checkov-scan
            template: checkov-scan
            arguments:
              parameters:
              - name: path
                value: "{{item}}"
            withItems: [
              terraform,
              terraform/region-1,
              terraform/region-2
            ]
            depends: clone-repo
          - name: tflint
            template: tflint
            arguments:
              parameters:
              - name: path
                value: "{{item}}"
            withItems: [
              terraform,
              terraform/region-1,
              terraform/region-2
            ]
            depends: clone-repo
          - name: tfplan
            template: tfplan
            arguments:
              parameters:
              - name: path
                value: "{{item}}"
            withItems: [
              terraform,
              terraform/region-1,
              terraform/region-2
            ]
            depends: tflint
          - name: tfplan-to-comment
            template: tfplan-to-comment
            arguments:
              parameters:
                - name: pr_num
                  value: "{{tasks.get-pr.outputs.parameters.pr_num}}"
                - name: index
                  value: "{{item}}"
            withItems: [
              terraform,
              terraform/region-1,
              terraform/region-2
            ]
            depends: (tfplan && get-pr)

    - name: clone-repo
      container:
        image: alpine
        command:
          - sh
          - -c
          - |
            apk --update add openssh-client git
            eval `ssh-agent -s`
            mkdir -p /workdir/src/github.com/<your-org>;
            cd /workdir/src/github.com/<your-org>;
            ssh-add /root/.ssh/ssh-deploy-key;
            ssh-keyscan github.com > /root/.ssh/known_hosts;
            git config --global --add safe.directory '*';
            git clone [e-mail]:<your-org>/<your-repo>.git;
            cd <your-repo>;
            git checkout $GIT_COMMIT;
        volumeMounts:
        - name: workdir
          mountPath: /workdir

    - name: checkov-scan
      inputs:
        parameters:
          - name: path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            mkdir -p /checkov-scan
            cp -R /workdir/src/github.com/<your-org>/<your-repo> /checkov-scan
            cd /checkov-scan/<your-repo>/{{inputs.parameters.path}}/
            tofu init
            checkov --quiet --compact --directory . --repo-root-for-plan-enrichment .
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key
        volumeMounts:
        - name: workdir
          mountPath: /workdir

    - name: tflint
      inputs:
        parameters:
          - name: path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            mkdir -p /tflint-dir
            cp -R /workdir/src/github.com/<your-org>/<your-repo> /tflint-dir
            cd /tflint-dir/<your-repo>/{{inputs.parameters.path}}/
            tofu init
            tflint --init --no-color
            tflint --no-color
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key
        volumeMounts:
        - name: workdir
          mountPath: /workdir

    - name: tfplan
      inputs:
        parameters:
          - name: path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            mkdir -p /tfplan-dir
            mkdir -p /workdir/terraform
            cp -R /workdir/src/github.com/<your-org>/<your-repo> /tfplan-dir
            cd /tfplan-dir/<your-repo>/{{inputs.parameters.path}}/
            tofu init
            tofu plan -lock-timeout=600s -out=tfplan
            tofu show -json tfplan | tf-summarize > /workdir/{{inputs.parameters.path}}-plan.txt
            tofu show tfplan
            tf-summarize tfplan
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key
        volumeMounts:
        - name: workdir
          mountPath: /workdir

    - name: tfplan-to-comment
      inputs:
        parameters:
          - name: pr_num
          - name: index
      container:
        image: cloudposse/github-commenter:0.16.2
        env:
          - name: GITHUB_TOKEN
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: github-token
          - name: GITHUB_COMMENT_FORMAT
            value: |
              Tofu plan ({{inputs.parameters.index}}):
              ```
              {{.}}
              ```
        command:
          - /bin/sh
          - -c
          - |
            if [ "${isPR}" == "true" ]
            then
              cat /workdir/{{inputs.parameters.index}}-plan.txt | github-commenter
            fi
        volumeMounts:
        - name: workdir
          mountPath: /workdir

Security scanning

The Workflow uses Checkov to scan infrastructure code for security and compliance issues:

- name: checkov-scan
  container:
    image: <your-registry>/terraform
    command:
      - bash
      - -c
      - |
        cd /workdir/terraform/
        tofu init
        checkov --quiet --compact --directory . --repo-root-for-plan-enrichment .

Terraform planning

The Workflow generates a plan showing what infrastructure changes would occur:

- name: tfplan
  container:
    image: <your-registry>/terraform
    command:
      - bash
      - -c
      - |
        cd /workdir/terraform/
        tofu init
        tofu plan -lock-timeout=600s -out=tfplan
        tofu show -json tfplan | tf-summarize > /workdir/plan.txt
        tofu show tfplan

The plan output is captured and posted as a comment on the pull request, giving reviewers clear visibility into the proposed changes.

Worked example: Nightly drift detection

Infrastructure drift occurs when the actual state of your infrastructure diverges from what is defined in your code. A scheduled CronWorkflow can detect this drift by running tofu plan regularly and alerting when changes are detected.

CronWorkflow structure

The nightly drift detection Workflow runs on a schedule and checks for infrastructure changes:

spec:
  schedules:
    - "0 13 * * 1-5"
  timezone: "UTC"
  workflowSpec:
    entrypoint: main
    templates:
      - name: main
        dag:
          tasks:
            - name: clone-repo
              template: clone-repo
            - name: check-and-notify
              template: check-and-notify
              depends: clone-repo

View complete nightly drift detection workflow

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: tf-nightly
  namespace: ci
  labels:
    cron: "true"
spec:
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  schedules:
    - "0 13 * * 1-5"
  timezone: "UTC"
  startingDeadlineSeconds: 0
  suspend: false
  workflowSpec:
    entrypoint: main
    synchronization:
      mutexes:
        - name: tf
    serviceAccountName: ci
    volumeClaimTemplates:
    - metadata:
        name: workdir
      spec:
        accessModes: [ "ReadWriteMany" ]
        storageClassName: nfs
        resources:
          requests:
            storage: 1Gi
    templates:
      - name: main
        dag:
          tasks:
            - name: clone-repo
              template: clone-repo
            - name: check-and-notify
              template: check-and-notify
              arguments:
                parameters:
                - name: path
                  value: "{{item}}"
              withItems: [
                terraform,
                terraform/region-1,
                terraform/region-2
              ]
              depends: clone-repo

      - name: check-and-notify
        inputs:
          parameters:
            - name: path
        dag:
          tasks:
            - name: tfplan
              template: tfplan
              arguments:
                parameters:
                - name: path
                  value: "{{inputs.parameters.path}}"
            - name: send-notification
              template: send-notification
              arguments:
                parameters:
                  - name: title
                    value: "Infrastructure drift detected"
                  - name: message
                    value: "Infrastructure has diverged from Terraform code in <your-repo>/{{inputs.parameters.path}}."
                  - name: exitcode
                    value: "{{tasks.tfplan.outputs.parameters.exitcode}}"
                  - name: path
                    value: "{{inputs.parameters.path}}"
              depends: tfplan

      - name: clone-repo
        container:
          image: alpine
          command:
            - sh
            - -c
            - |
              apk --update add openssh-client git
              eval `ssh-agent -s`
              mkdir -p /workdir/src/github.com/<your-org>;
              cd /workdir/src/github.com/<your-org>;
              ssh-add /root/.ssh/ssh-deploy-key;
              ssh-keyscan github.com > /root/.ssh/known_hosts;
              git config --global --add safe.directory '*';
              git clone [e-mail]:<your-org>/<your-repo>.git;
              cd <your-repo>;
              git checkout $GIT_COMMIT;
          volumeMounts:
          - name: workdir
            mountPath: /workdir

      - name: tfplan
        inputs:
          parameters:
            - name: path
        container:
          image: <your-registry>/terraform
          command:
            - bash
            - -c
            - |
              mkdir -p /tfplan-dir
              cp -R /workdir/src/github.com/<your-org>/<your-repo> /tfplan-dir
              cd /tfplan-dir/<your-repo>/{{inputs.parameters.path}}/
              tofu init
              tofu plan -detailed-exitcode -lock-timeout=600s
              if [ $? -eq 2 ]; then
                  echo "Tofu apply needed"
                  echo "0" > /tmp/exitcode
                  exit 0
              else
                  echo "Infrastructure matches code"
                  echo "1" > /tmp/exitcode
                  exit 0
              fi
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: <your-secret>
                  key: aws-access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: <your-secret>
                  key: aws-secret-access-key
          volumeMounts:
          - name: workdir
            mountPath: /workdir
        outputs:
          parameters:
          - name: exitcode
            valueFrom:
              path: /tmp/exitcode

      - name: send-notification
        inputs:
          parameters:
            - name: title
            - name: message
            - name: exitcode
            - name: path
        container:
          image: alpine
          command:
            - sh
            - -c
            - |
              if [ "{{inputs.parameters.exitcode}}" == "0" ]
              then
                A="{{inputs.parameters.title}}"
                B="{{inputs.parameters.message}}"
                AINPUT="${A}\n${B}"
                # Send notification via your preferred method (Slack, email, etc.)
                echo "$AINPUT"
                # Example: curl to Slack webhook, PagerDuty, etc.
                # curl -H "Content-type: application/json" -X POST -d "$data" ${WEBHOOK_URL}
              fi
          env:
            - name: WEBHOOK_URL
              valueFrom:
                secretKeyRef:
                  name: <your-secret>
                  key: webhook-url

Drift detection logic

The Workflow runs tofu plan with the -detailed-exitcode flag, which returns exit code 2 when changes are detected:

- name: tfplan
  container:
    image: <your-registry>/terraform
    command:
      - bash
      - -c
      - |
        cd /workdir/terraform/
        tofu init
        tofu plan -detailed-exitcode -lock-timeout=600s
        if [ $? -eq 2 ]; then
            echo "Infrastructure drift detected"
            echo "0" > /tmp/exitcode
        else
            echo "Infrastructure matches code"
            echo "1" > /tmp/exitcode
        fi

When drift is detected, the Workflow sends a notification to alert the team.

Best practices

Use mutual exclusion

Infrastructure operations should not run concurrently on the same resources. Use Argo's synchronization.mutexes to ensure only one Workflow modifies infrastructure at a time:

spec:
  synchronization:
    mutexes:
      - name: tf

Secure credentials management

Store sensitive credentials in Kubernetes Secrets and inject them into Workflow pods as environment variables:

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: <your-secret>
        key: aws-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: <your-secret>
        key: aws-secret-access-key

For tighter integration, use external secrets management solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault with the appropriate Kubernetes integrations.

Separate plan from apply

Never automatically run terraform apply or tofu apply in automated Workflows. Always generate plans for review, then apply changes manually or with explicit human approval.

A common pattern is to create a Workflow Template for the apply step. This lets authorized users submit the template to apply infrastructure changes without configuring their local environment with the correct credentials and tooling. The template carries the necessary credentials, container image, and configuration, giving you consistent and secure infrastructure deployments.

For example, create a template that takes the plan output as input and applies it:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: terraform-apply
spec:
  entrypoint: apply
  templates:
    - name: apply
      inputs:
        parameters:
          - name: terraform-path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            cd /workdir/{{inputs.parameters.terraform-path}}/
            tofu apply -auto-approve
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key

Users submit this template from Pipekit when they're ready to apply reviewed changes.

Monitor for drift

Run drift detection Workflows on a regular schedule (e.g. nightly) to catch unexpected infrastructure changes early. Alert your team when drift is detected so they can investigate and remediate.

Use node selectors

For resource-intensive operations like security scanning, specify node selectors so Workflows run on appropriate hardware:

nodeSelector:
  workload-type: compute-intensive

Related

• Pipes: how to create and run the Pipes that wrap these Workflows.

• Templates and the Templates how-to: share the apply step safely across teams.

• Secrets: per-environment secrets for cloud credentials.

• Alerting: fire alerts when drift detection triggers.

• Why Pipekit > Governance and Why Pipekit > Scale: the value-prop framing that maps onto this use case.

• Pipekit CLI: submit and manage these Workflows from your terminal.