Skip to content

CI/CD Guide: Terraform and Docker Buildx/Bake

This document explains the modern CI/CD approach implemented in this repository using reusable GitHub Actions workflows for Terraform and Docker Buildx/Bake.

Terraform Workflows

Reusable Terraform Workflow

We've implemented a reusable GitHub Actions workflow for Terraform operations that follows best practices:

  • File: .github/workflows/terraform-reusable.yml
  • Purpose: Standardized workflow for Terraform plan and apply operations
  • Features:
  • Secure credential handling
  • Terraform validation and formatting checks
  • Plan visualization in PR comments
  • Support for multiple environments
  • Dynamic variable handling
  • Intelligent auto-approve logic

Infrastructure Deployment Workflow

This workflow uses the reusable Terraform workflow for infrastructure deployment:

  • File: .github/workflows/terraform-infra.yml
  • Purpose: Deploy different components of infrastructure
  • Features:
  • Intelligent path filtering to determine what components need deployment
  • Sequential dependency handling (base infra → services)
  • Environment-specific deployment
  • Manual trigger support via workflow dispatch

Usage

To trigger a Terraform deployment manually:

  1. Go to Actions → Infrastructure Deployment → Run workflow
  2. Select the environment (dev, staging, prod)
  3. Choose whether to auto-approve the apply step
  4. Run the workflow

Docker Buildx/Bake Workflows

Reusable Buildx/Bake Workflow

We've implemented a reusable GitHub Actions workflow for Docker builds using Buildx and Bake:

  • File: .github/workflows/docker-buildx-bake.yml
  • Purpose: Build and publish Docker images with modern tooling
  • Features:
  • Multi-platform builds (amd64, arm64)
  • Docker Buildx for efficient layer caching
  • Docker Bake for complex build scenarios
  • Registry-agnostic authentication
  • GitHub Packages, Docker Hub, and Google Artifact Registry support
  • Build caching for faster builds

App Build Workflow

This workflow uses the reusable Buildx/Bake workflow for building specific apps:

  • File: .github/workflows/build-datastream.yml
  • Purpose: Build and publish the Datastream app
  • Features:
  • Path-based triggering
  • Automatic dependency detection
  • Integration with deployment workflows
  • Manual build triggers

Docker Bake Configuration

The docker-bake.hcl file provides a declarative way to define Docker builds:

  • File: docker-bake.hcl
  • Purpose: Define Docker build targets and their configurations
  • Features:
  • Inheritance and composition for DRY configuration
  • Matrix builds for multi-environment configuration
  • Variable interpolation
  • Multi-platform support
  • Build caching

Getting Started

Using Terraform Workflows

  1. Define your Terraform code in the appropriate directories:

  2. terraform/infra/ for base infrastructure

  3. terraform/ for service-specific resources

  4. terraform/support_infrastructure/ for auxiliary resources

  5. Create required secrets in your GitHub repository:

  6. GCP_CREDENTIALS: JSON credentials for Google Cloud

  7. TF_API_TOKEN: Terraform Cloud API token (if using Terraform Cloud)

  8. Push your changes or open a PR to trigger the workflow

Using Docker Buildx/Bake

  1. Define your Dockerfile in the app directory (e.g., apps/datastream/Dockerfile)

  2. Add a build target in docker-bake.hcl:

target "your-app" {
  inherits = ["common-config"]
  dockerfile = "apps/your-app/Dockerfile"
  # Additional app-specific configuration...
}
  1. Create or update a workflow file for your app:
uses: ./.github/workflows/docker-buildx-bake.yml
with:
  context: '.'
  bake-file: 'docker-bake.hcl'
  target-app: 'your-app'
  # Additional configuration...
  1. Push your changes or open a PR to trigger the workflow

Best Practices

Terraform

  • Always run terraform fmt locally before committing
  • Use modules for reusable infrastructure components
  • Use variables for environment-specific configuration
  • Use remote state to ensure consistent state management

Docker

  • Use multi-stage builds to minimize image size
  • Set appropriate cache settings for efficient builds
  • Use BuildKit features like secrets for secure builds
  • Consider multi-platform builds for maximum compatibility

Troubleshooting

Terraform Workflow Issues

  • Check that your GCP credentials have the necessary permissions
  • Verify that the Terraform state is properly configured
  • Check for formatting issues with terraform fmt -check

Docker Build Issues

  • Verify your Dockerfile works locally with docker build
  • Test your bake file locally with docker buildx bake -f docker-bake.hcl
  • Check registry permissions for pushing images