Skip to content

Robust Python Tasks for NX

Status

Implemented

Date

2025-04-01

Context

Our monorepo was experiencing significant flakiness with Python-related NX tasks such as lock, install, and deps-check in projects like gleam, chainpyon, and others. This flakiness was causing:

  1. CI failures requiring manual intervention or retries
  2. Developer frustration and reduced productivity
  3. Inconsistent local development environments

The main causes of flakiness were identified as:

  • Poetry virtual environment inconsistencies
  • Broken or incorrect virtual environment references
  • Lack of robust error handling and retry mechanisms
  • Non-standardized tags in project.json files

Decision

We have implemented a comprehensive solution to make Python tasks more reliable:

  1. Create a robust Poetry wrapper script (scripts/python/robust-poetry.sh) that:

  2. Implements retry logic with exponential backoff

  3. Automatically cleans up broken environments
  4. Provides clear logging for debugging

  5. Falls back to a clean install when needed

  6. Update all Python project.json files to:

  7. Use the robust script for lock and install operations

  8. Add an explicit install:retry target for clean installations

  9. Make deps-check non-blocking but still informative

  10. Standardize project tags across all Python projects:

  11. Use language:python (not language-python)
  12. Use type:library (not type-library)

Consequences

Positive

  • Python tasks are now significantly more reliable
  • CI builds succeed more consistently
  • Developers can work more efficiently without manual interventions
  • New projects can adopt the pattern for consistent behavior

Negative

  • Added complexity with additional scripts
  • Potential increased build times due to retries and clean installations

Neutral

  • Some warnings may still appear but won't block execution

Implementation

The implementation consists of:

  1. A core helper script:

  2. scripts/python/robust-poetry.sh: Robust wrapper for Poetry operations

  3. A configuration script:

  4. scripts/python/update-python-projects.sh: Updates project.json files to use the robust approach

  5. Updates to all Python project.json files:

  6. Modified lock, install, and deps-check targets
  7. Standardized tags across projects

To apply the solution to a new Python project, run:

bash scripts/python/update-python-projects.sh

References