Skip to content

CI Optimization Strategy

Date: 2025-04-02

Status

Accepted

Context

Our CI/CD pipeline is critical for ensuring code quality and security in our monorepo structure. However, as the codebase grows, so does the runtime of our CI/CD pipelines. With limited CI/CD machine instances available, running full validation on every PR is becoming resource-intensive and time-consuming.

We need a strategy to optimize our CI/CD pipeline to:

  1. Reduce unnecessary builds
  2. Skip tests that aren't relevant to the changes
  3. Focus resources on the most critical checks
  4. Minimize wait times for developers
  5. Make efficient use of our limited CI runners

Decision

We will implement a comprehensive CI optimization strategy using skip-duplicate-actions and custom logic to intelligently determine which checks to run based on the changes in a PR or commit.

Key Components:

  1. Change Analysis: We'll analyze which files have changed to determine:

  2. Which languages are affected (TypeScript, Python, etc.)

  3. Which applications and libraries are affected
  4. Whether configuration or Docker files have changed

  5. If only tests or documentation have changed

  6. Dynamic Test Matrix: Based on the change analysis, we'll:

  7. Create a dynamic build matrix that only includes affected applications

  8. Skip language-specific checks if no relevant files changed
  9. Skip Docker builds if no Docker files changed

  10. Prioritize security checks for sensitive changes

  11. Duplicate Detection: We'll use fkirc/skip-duplicate-actions to:

  12. Skip checks if they're identical to a previous successful run

  13. Identify when checks can be safely skipped

  14. Provide visibility into skipped checks

  15. Transparency: We'll provide clear visibility by:

  16. Adding comments to PRs with optimization decisions
  17. Showing estimated time/resource savings
  18. Ensuring developers understand why checks were skipped

Implementation Details

  1. Change Detection:

  2. Use Git to determine which files changed

  3. Group changes by file type, language, and component

  4. Skip Criteria:

  5. Skip JavaScript/TypeScript checks if only Python files changed

  6. Skip Python checks if only JavaScript/TypeScript files changed
  7. Skip security checks if only tests or documentation changed

  8. Skip Docker builds if no Docker-related files changed

  9. Affected Component Detection:

  10. Extract application names from file paths

  11. Extract library names from file paths

  12. Build dependency graph to determine indirect impacts

  13. Build Matrix Generation:

  14. Create a matrix that only includes affected applications

  15. Include language-specific information for each app

  16. Optimization Reporting:

  17. Add PR comments showing which checks were skipped
  18. Show estimated time and resource savings
  19. Maintain a single, updated comment per PR

Consequences

Positive

  • Reduced CI runtime for most PRs
  • More efficient use of CI/CD resources
  • Faster feedback cycles for developers
  • Ability to scale the monorepo without proportional increase in CI resources
  • Clear visibility into optimization decisions

Negative

  • Increased complexity in CI/CD configuration
  • Risk of skipping relevant checks if dependencies aren't properly mapped
  • Potential for different behavior between local and CI environments

Mitigations

  • Comprehensive documentation of the optimization strategy
  • Regular validation of the dependency mapping
  • Clear PR comments explaining optimization decisions
  • Option to force full CI checks when needed

Usage Notes

For Developers

  1. Understanding Skipped Checks:

  2. Look for the "CI Optimization Summary" comment on your PR

  3. Review which checks were skipped and why

  4. Note the estimated time and resource savings

  5. When to Force Full Checks:

  6. For releases or significant changes

  7. When dependencies have changed

  8. When you're unsure about the impact of your changes

  9. Optimizing Your PRs:

  10. Group related changes together
  11. Separate documentation/test changes from code changes
  12. Consider breaking large PRs into smaller, focused ones

For Maintainers

  1. Monitoring Effectiveness:

  2. Track actual time savings vs. estimated

  3. Monitor for cases where relevant checks were incorrectly skipped

  4. Adjust skip criteria based on real-world effectiveness

  5. Maintaining the System:

  6. Update dependency mappings as the codebase evolves
  7. Adjust skip criteria as needed
  8. Ensure documentation stays current

References