Skip to content

Secret Detection Solution

Date: 2025-04-02

Status

Accepted

Context

In a financial services application handling sensitive data such as trading APIs and cryptographic keys, there's a significant risk of accidentally committing secrets to the codebase. This can lead to security breaches if these secrets are exposed in public repositories or to unauthorized personnel.

We need a standardized approach to detect and prevent the accidental commitment of secrets, API keys, credentials, and other sensitive information across our codebase.

Decision

We will implement a comprehensive secret detection solution using secretlint, with tailored rules specific to our financial trading platform. This solution will be integrated at multiple levels:

  1. Developer Environment: Local pre-commit hooks to catch secrets before they're committed
  2. CI/CD Pipeline: Automated scanning in GitHub Actions workflows
  3. Scheduled Scans: Regular scans of the codebase to catch secrets that might have been missed

Specifics

Tools

  • Secretlint: A pluggable linting tool for detecting secrets in code
  • Custom Patterns: Additional patterns specific to financial services (trading API keys, blockchain private keys)
  • Shell Script Wrapper: A utility script to provide consistent developer experience

Implementation

  1. Enhanced .secretlintrc.json configuration with:

  2. Standard secret patterns (AWS keys, Google API keys, etc.)

  3. Financial-specific patterns (Deribit API keys, Block Scholes API keys)
  4. Blockchain-specific patterns (private keys, mnemonic phrases)

  5. Database connection strings with credentials

  6. Pre-commit hook integration to:

  7. Scan staged files before commit

  8. Block commits with detected secrets

  9. Provide clear, actionable feedback

  10. CI/CD workflow integration to:

  11. Scan pull requests for secrets

  12. Report findings in PR comments

  13. Apply appropriate labels for security review

  14. Developer utilities:

  15. Command-line tools for scanning specific parts of the codebase
  16. Options for masked output to prevent further exposure
  17. Documentation and guidance

Consequences

Positive

  • Reduces the risk of secret exposure in the codebase
  • Creates awareness among developers about secure coding practices
  • Provides multiple layers of protection (local, CI/CD, scheduled)
  • Tailored to our specific financial trading application needs

Negative

  • May produce false positives requiring developer judgment
  • Additional step in the development workflow
  • Small performance impact in pre-commit hooks and CI/CD pipelines

Mitigations

  • Carefully tuned rules to minimize false positives
  • Documentation and training on handling potential findings
  • Efficient implementation to minimize performance impact

Usage Guidelines

For Developers

  1. Local Scanning:
# Scan all files
yarn secrets:check

# Scan only staged files
yarn secrets:check:staged

# Scan specific application
yarn secrets:check:app orderbook

# Scan specific library
yarn secrets:check:lib commons-ts

# Scan with masked output (safer)
yarn secrets:check:masked
  1. Pre-commit Hook:
  2. Automatically runs on git commit
  3. Reviews staged changes for secrets
  4. Blocks commit if secrets are found

For CI/CD

  • Automatically runs on pull requests
  • Results appear as PR comments
  • Adds "security-review-needed" label when secrets are detected

False Positive Handling

If you encounter a false positive:

  1. Review the finding to confirm it's not actually sensitive
  2. Update the allowed patterns in .secretlintrc.json if appropriate
  3. Use contextual comments for specific exceptions: 
    // secretlint-disable
const exampleApiKey = 'THIS_IS_A_DUMMY_KEY_FOR_TESTING';
// secretlint-enable

References