> For the complete documentation index, see [llms.txt](https://docs.pipekit.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pipekit.io/use-cases/infrastructure-as-code.md).
# Infrastructure as Code with OpenTofu
## The problem
OpenTofu and Terraform changes are high-blast-radius. A misconfigured security group or an unintended `tofu destroy` can take down production faster than almost any other class of change in a typical platform. The standard mitigations (security scanning, linting, `tofu plan` review, drift detection) are well understood, but stitching them together is not.
Most teams end up running `tofu plan` inside a generic CI runner (GitHub Actions, Jenkins, GitLab CI) and pasting the output into pull-request comments. That works for a single team and a single environment. It breaks down quickly: the plan output is hard to review at scale, the runner doesn't enforce mutual exclusion so two engineers can race each other into a locked state, nightly drift detection has nowhere natural to live, and the `apply` step turns into "whoever has the credentials on their laptop."
The teams that get this right tend to converge on the same shape: a real DAG, a real workflow engine, secrets the runner doesn't touch, mutual exclusion enforced at the engine level, and a separate path for `plan` vs `apply`. That shape is exactly what Argo Workflows gives you.
## How Pipekit helps
Argo Workflows runs the DAG. Pipekit adds the surfaces an Argo deployment by itself doesn't have: a unified UI across [Clusters](/concepts/cluster.md), persisted [Run](/concepts/run.md) history with searchable [logs](/using-pipekit/runs/pod-logs.md), per-environment [secrets](/using-pipekit/pipes/edit/secrets.md) scoped to [workspaces](/concepts/access-control.md#workspaces), [Templates](/concepts/templates.md) that let you share the `apply` step safely across teams, and [alerting](/using-pipekit/pipes/edit/alerting.md) when drift detection fires.
The two worked examples below come from a real Pipekit customer's setup. The first is a pull-request validation Workflow that runs `checkov`, `tflint`, and `tofu plan` against three Terraform paths in parallel and posts the plan output as a PR comment. The second is a CronWorkflow that runs `tofu plan -detailed-exitcode` nightly to detect drift. Both are designed so the `apply` step is intentionally separate. See the [Best practices](#best-practices) section.
## Worked example: Pull-request validation
When a pull request modifies infrastructure code, an automated Workflow validates the changes before merge.
The Workflow performs several validation steps:
1. Clone the infrastructure repository.
2. Run security scans to detect misconfigurations (`checkov`).
3. Run linting to enforce best practices (`tflint`).
4. Generate Terraform/OpenTofu plans showing what will change.
5. Post plan summaries as PR comments for review.
### Workflow Structure
The PR validation Workflow consists of multiple tasks organized as a DAG (directed acyclic graph):
```yaml
- name: main
dag:
tasks:
- name: clone-repo
template: clone-repo
- name: checkov-scan
template: checkov-scan
depends: clone-repo
- name: tflint
template: tflint
depends: clone-repo
- name: tfplan
template: tfplan
depends: tflint
- name: tfplan-to-comment
template: tfplan-to-comment
depends: tfplan
```
View complete PR validation workflow
````yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: tf-pr-
namespace: ci
spec:
serviceAccountName: ci
entrypoint: main
synchronization:
mutexes:
- name: tf
volumeClaimTemplates:
- metadata:
name: workdir
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: nfs
resources:
requests:
storage: 1Gi
templates:
- name: main
dag:
tasks:
- name: clone-repo
template: clone-repo
- name: get-pr
template: get-pr
- name: checkov-scan
template: checkov-scan
arguments:
parameters:
- name: path
value: "{{item}}"
withItems: [
terraform,
terraform/region-1,
terraform/region-2
]
depends: clone-repo
- name: tflint
template: tflint
arguments:
parameters:
- name: path
value: "{{item}}"
withItems: [
terraform,
terraform/region-1,
terraform/region-2
]
depends: clone-repo
- name: tfplan
template: tfplan
arguments:
parameters:
- name: path
value: "{{item}}"
withItems: [
terraform,
terraform/region-1,
terraform/region-2
]
depends: tflint
- name: tfplan-to-comment
template: tfplan-to-comment
arguments:
parameters:
- name: pr_num
value: "{{tasks.get-pr.outputs.parameters.pr_num}}"
- name: index
value: "{{item}}"
withItems: [
terraform,
terraform/region-1,
terraform/region-2
]
depends: (tfplan && get-pr)
- name: clone-repo
container:
image: alpine
command:
- sh
- -c
- |
apk --update add openssh-client git
eval `ssh-agent -s`
mkdir -p /workdir/src/github.com/;
cd /workdir/src/github.com/;
ssh-add /root/.ssh/ssh-deploy-key;
ssh-keyscan github.com > /root/.ssh/known_hosts;
git config --global --add safe.directory '*';
git clone git@github.com:/.git;
cd ;
git checkout $GIT_COMMIT;
volumeMounts:
- name: workdir
mountPath: /workdir
- name: checkov-scan
inputs:
parameters:
- name: path
container:
image: /terraform
command:
- bash
- -c
- |
mkdir -p /checkov-scan
cp -R /workdir/src/github.com// /checkov-scan
cd /checkov-scan//{{inputs.parameters.path}}/
tofu init
checkov --quiet --compact --directory . --repo-root-for-plan-enrichment .
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
volumeMounts:
- name: workdir
mountPath: /workdir
- name: tflint
inputs:
parameters:
- name: path
container:
image: /terraform
command:
- bash
- -c
- |
mkdir -p /tflint-dir
cp -R /workdir/src/github.com// /tflint-dir
cd /tflint-dir//{{inputs.parameters.path}}/
tofu init
tflint --init --no-color
tflint --no-color
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
volumeMounts:
- name: workdir
mountPath: /workdir
- name: tfplan
inputs:
parameters:
- name: path
container:
image: /terraform
command:
- bash
- -c
- |
mkdir -p /tfplan-dir
mkdir -p /workdir/terraform
cp -R /workdir/src/github.com// /tfplan-dir
cd /tfplan-dir//{{inputs.parameters.path}}/
tofu init
tofu plan -lock-timeout=600s -out=tfplan
tofu show -json tfplan | tf-summarize > /workdir/{{inputs.parameters.path}}-plan.txt
tofu show tfplan
tf-summarize tfplan
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
volumeMounts:
- name: workdir
mountPath: /workdir
- name: tfplan-to-comment
inputs:
parameters:
- name: pr_num
- name: index
container:
image: cloudposse/github-commenter:0.16.2
env:
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
name:
key: github-token
- name: GITHUB_COMMENT_FORMAT
value: |
Tofu plan ({{inputs.parameters.index}}):
```
{{.}}
```
command:
- /bin/sh
- -c
- |
if [ "${isPR}" == "true" ]
then
cat /workdir/{{inputs.parameters.index}}-plan.txt | github-commenter
fi
volumeMounts:
- name: workdir
mountPath: /workdir
````
### Security scanning
The Workflow uses Checkov to scan infrastructure code for security and compliance issues:
```yaml
- name: checkov-scan
container:
image: /terraform
command:
- bash
- -c
- |
cd /workdir/terraform/
tofu init
checkov --quiet --compact --directory . --repo-root-for-plan-enrichment .
```
### Terraform planning
The Workflow generates a plan showing what infrastructure changes would occur:
```yaml
- name: tfplan
container:
image: /terraform
command:
- bash
- -c
- |
cd /workdir/terraform/
tofu init
tofu plan -lock-timeout=600s -out=tfplan
tofu show -json tfplan | tf-summarize > /workdir/plan.txt
tofu show tfplan
```
The plan output is captured and posted as a comment on the pull request, giving reviewers clear visibility into the proposed changes.
## Worked example: Nightly drift detection
Infrastructure drift occurs when the actual state of your infrastructure diverges from what is defined in your code. A scheduled [CronWorkflow](/concepts/cron-and-externally-triggered.md) can detect this drift by running `tofu plan` regularly and alerting when changes are detected.
### CronWorkflow structure
The nightly drift detection Workflow runs on a schedule and checks for infrastructure changes:
```yaml
spec:
schedules:
- "0 13 * * 1-5"
timezone: "UTC"
workflowSpec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: clone-repo
template: clone-repo
- name: check-and-notify
template: check-and-notify
depends: clone-repo
```
View complete nightly drift detection workflow
```yaml
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
name: tf-nightly
namespace: ci
labels:
cron: "true"
spec:
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
schedules:
- "0 13 * * 1-5"
timezone: "UTC"
startingDeadlineSeconds: 0
suspend: false
workflowSpec:
entrypoint: main
synchronization:
mutexes:
- name: tf
serviceAccountName: ci
volumeClaimTemplates:
- metadata:
name: workdir
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: nfs
resources:
requests:
storage: 1Gi
templates:
- name: main
dag:
tasks:
- name: clone-repo
template: clone-repo
- name: check-and-notify
template: check-and-notify
arguments:
parameters:
- name: path
value: "{{item}}"
withItems: [
terraform,
terraform/region-1,
terraform/region-2
]
depends: clone-repo
- name: check-and-notify
inputs:
parameters:
- name: path
dag:
tasks:
- name: tfplan
template: tfplan
arguments:
parameters:
- name: path
value: "{{inputs.parameters.path}}"
- name: send-notification
template: send-notification
arguments:
parameters:
- name: title
value: "Infrastructure drift detected"
- name: message
value: "Infrastructure has diverged from Terraform code in /{{inputs.parameters.path}}."
- name: exitcode
value: "{{tasks.tfplan.outputs.parameters.exitcode}}"
- name: path
value: "{{inputs.parameters.path}}"
depends: tfplan
- name: clone-repo
container:
image: alpine
command:
- sh
- -c
- |
apk --update add openssh-client git
eval `ssh-agent -s`
mkdir -p /workdir/src/github.com/;
cd /workdir/src/github.com/;
ssh-add /root/.ssh/ssh-deploy-key;
ssh-keyscan github.com > /root/.ssh/known_hosts;
git config --global --add safe.directory '*';
git clone git@github.com:/.git;
cd ;
git checkout $GIT_COMMIT;
volumeMounts:
- name: workdir
mountPath: /workdir
- name: tfplan
inputs:
parameters:
- name: path
container:
image: /terraform
command:
- bash
- -c
- |
mkdir -p /tfplan-dir
cp -R /workdir/src/github.com// /tfplan-dir
cd /tfplan-dir//{{inputs.parameters.path}}/
tofu init
tofu plan -detailed-exitcode -lock-timeout=600s
if [ $? -eq 2 ]; then
echo "Tofu apply needed"
echo "0" > /tmp/exitcode
exit 0
else
echo "Infrastructure matches code"
echo "1" > /tmp/exitcode
exit 0
fi
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
volumeMounts:
- name: workdir
mountPath: /workdir
outputs:
parameters:
- name: exitcode
valueFrom:
path: /tmp/exitcode
- name: send-notification
inputs:
parameters:
- name: title
- name: message
- name: exitcode
- name: path
container:
image: alpine
command:
- sh
- -c
- |
if [ "{{inputs.parameters.exitcode}}" == "0" ]
then
A="{{inputs.parameters.title}}"
B="{{inputs.parameters.message}}"
AINPUT="${A}\n${B}"
# Send notification via your preferred method (Slack, email, etc.)
echo "$AINPUT"
# Example: curl to Slack webhook, PagerDuty, etc.
# curl -H "Content-type: application/json" -X POST -d "$data" ${WEBHOOK_URL}
fi
env:
- name: WEBHOOK_URL
valueFrom:
secretKeyRef:
name:
key: webhook-url
```
### Drift detection logic
The Workflow runs `tofu plan` with the `-detailed-exitcode` flag, which returns exit code 2 when changes are detected:
```yaml
- name: tfplan
container:
image: /terraform
command:
- bash
- -c
- |
cd /workdir/terraform/
tofu init
tofu plan -detailed-exitcode -lock-timeout=600s
if [ $? -eq 2 ]; then
echo "Infrastructure drift detected"
echo "0" > /tmp/exitcode
else
echo "Infrastructure matches code"
echo "1" > /tmp/exitcode
fi
```
When drift is detected, the Workflow sends a notification to alert the team.
## Best practices
### Use mutual exclusion
Infrastructure operations should not run concurrently on the same resources. Use Argo's `synchronization.mutexes` to ensure only one Workflow modifies infrastructure at a time:
```yaml
spec:
synchronization:
mutexes:
- name: tf
```
### Secure credentials management
Store sensitive credentials in Kubernetes Secrets and inject them into Workflow pods as environment variables:
```yaml
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
```
For tighter integration, use external secrets management solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault with the appropriate Kubernetes integrations.
### Separate plan from apply
Never automatically run `terraform apply` or `tofu apply` in automated Workflows. Always generate plans for review, then apply changes manually or with explicit human approval.
A common pattern is to create a [Workflow Template](/using-pipekit/templates.md) for the apply step. This lets authorized users submit the template to apply infrastructure changes without configuring their local environment with the correct credentials and tooling. The template carries the necessary credentials, container image, and configuration, giving you consistent and secure infrastructure deployments.
For example, create a template that takes the plan output as input and applies it:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: terraform-apply
spec:
entrypoint: apply
templates:
- name: apply
inputs:
parameters:
- name: terraform-path
container:
image: /terraform
command:
- bash
- -c
- |
cd /workdir/{{inputs.parameters.terraform-path}}/
tofu apply -auto-approve
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name:
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name:
key: aws-secret-access-key
```
Users submit this template from Pipekit when they're ready to apply reviewed changes.
### Monitor for drift
Run drift detection Workflows on a regular schedule (e.g. nightly) to catch unexpected infrastructure changes early. Alert your team when drift is detected so they can investigate and remediate.
### Use node selectors
For resource-intensive operations like security scanning, specify node selectors so Workflows run on appropriate hardware:
```yaml
nodeSelector:
workload-type: compute-intensive
```
## Related
* [Pipes](/using-pipekit/pipes.md): how to create and run the Pipes that wrap these Workflows.
* [Templates](/concepts/templates.md) and the [Templates how-to](/using-pipekit/templates.md): share the `apply` step safely across teams.
* [Secrets](/using-pipekit/pipes/edit/secrets.md): per-environment secrets for cloud credentials.
* [Alerting](/using-pipekit/pipes/edit/alerting.md): fire alerts when drift detection triggers.
* [Why Pipekit > Governance](/why-pipekit/governance.md) and [Why Pipekit > Scale](/why-pipekit/scale.md): the value-prop framing that maps onto this use case.
* [Pipekit CLI](/reference/cli.md): submit and manage these Workflows from your terminal.