Skip to content

Visual Documentation Guide

This document serves as a central resource for all visual documentation in the v4-monorepo. Visual documentation helps developers understand complex systems and processes more efficiently, especially useful for visual learners and when explaining intricate workflows.

Table of Contents

System Architecture

High-Level System Architecture

High-Level Architecture

This diagram illustrates the complete system architecture, showing:

  • Core services (Orderbook, Sequencer, Risk Engine)
  • Auxiliary services (Market Maker Protection, Web3 Relayer, Liquidation Engine, WS Service)
  • Market Data services (Funding Rate Service, Data Stream)
  • Persistence layers (Databases, Cloud SQL, Cloud Redis)
  • External services and integrations

Component Architecture Diagrams

flowchart TB
    subgraph "Service Layer"
        Orderbook["Orderbook Service<br>(Trading Engine)"]
        Sequencer["Sequencer<br>(Risk & Accounts)"]
        Datastream["Datastream<br>(Market Data)"]
        Relayer["Relayer<br>(Web3 Transaction)"]
        Websockets["Websockets<br>(Real-time Events)"]
        Funding["Funding<br>(Rate Calculation)"]
        Instruments["Instruments<br>(Asset Management)"]
    end

    subgraph "Libraries Layer"
        CommonsTS["commons-ts<br>(TS Utilities)"]
        Commons["commons<br>(Python Utilities)"]
        ConnectionsTS["connections-ts<br>(DB Connectors)"]
        Chainpyon["chainpyon<br>(Web3 Python)"]
        Tools["tools<br>(Build Tools)"]
    end

    subgraph "External Integrations"
        Blockchain["Blockchain<br>(Smart Contracts)"]
        PriceFeeds["Price Feeds<br>(Index & IV)"]
    end

    subgraph "Storage Layer"
        Redis["Redis<br>(Cache)"]
        CloudSQL["Cloud SQL<br>(State)"]
        BigQuery["BigQuery<br>(Analytics)"]
    end

    %% Service dependencies
    Orderbook -->|uses| CommonsTS
    Orderbook -->|uses| ConnectionsTS
    Sequencer -->|uses| CommonsTS
    Sequencer -->|uses| ConnectionsTS
    Datastream -->|uses| CommonsTS
    Datastream -->|uses| ConnectionsTS
    Relayer -->|uses| CommonsTS
    Relayer -->|uses| ConnectionsTS
    Websockets -->|uses| CommonsTS
    Websockets -->|uses| ConnectionsTS
    Funding -->|uses| Commons
    Instruments -->|uses| CommonsTS
    Instruments -->|uses| ConnectionsTS

    %% Service interactions
    Orderbook <-->|events| Sequencer
    Orderbook -->|publishes to| Websockets
    Sequencer -->|executes via| Relayer
    Datastream -->|provides data to| Orderbook
    Datastream -->|provides data to| Sequencer

    %% External integrations
    Relayer <-->|transactions| Blockchain
    Datastream <-->|market data| PriceFeeds

    %% Storage connections
    Orderbook <-->|cache| Redis
    Orderbook <-->|storage| CloudSQL
    Orderbook -->|analytics| BigQuery
    Sequencer <-->|cache| Redis
    Sequencer <-->|storage| CloudSQL
    Sequencer -->|analytics| BigQuery
    Datastream <-->|cache| Redis

    classDef service fill:#c4e3ff,stroke:#666,stroke-width:1px
    classDef library fill:#f9f7c0,stroke:#666,stroke-width:1px
    classDef external fill:#d7f7c9,stroke:#666,stroke-width:1px
    classDef storage fill:#f5d2d3,stroke:#666,stroke-width:1px

    class Orderbook,Sequencer,Datastream,Relayer,Websockets,Funding,Instruments service
    class CommonsTS,Commons,ConnectionsTS,Chainpyon,Tools library
    class Blockchain,PriceFeeds external
    class Redis,CloudSQL,BigQuery storage

Dependency Graphs

Core Services Dependencies

flowchart LR
    subgraph "App Layer"
        direction TB

        Orderbook["apps/orderbook"]
        Datastream["apps/datastream"]
        Sequencer["apps/sequencer"]
        Relayer["apps/relayer"]
        Websockets["apps/websockets"]
        Funding["apps/funding"]
        Instruments["apps/instruments"]
        Poseidon["apps/poseidon"]
    end

    subgraph "Library Layer"
        direction TB

        CommonsTS["libs/commons-ts"]
        Commons["libs/commons"]
        ConnectionsTS["libs/connections-ts"]
        Chainpyon["libs/chainpyon"]
        Tools["libs/tools"]
    end

    %% TypeScript Services
    Orderbook -->|depends on| CommonsTS
    Orderbook -->|depends on| ConnectionsTS
    Orderbook -->|depends on| Tools

    Datastream -->|depends on| CommonsTS
    Datastream -->|depends on| ConnectionsTS
    Datastream -->|depends on| Tools

    Sequencer -->|depends on| CommonsTS
    Sequencer -->|depends on| ConnectionsTS
    Sequencer -->|depends on| Tools

    Relayer -->|depends on| CommonsTS
    Relayer -->|depends on| ConnectionsTS
    Relayer -->|depends on| Tools

    Websockets -->|depends on| CommonsTS
    Websockets -->|depends on| ConnectionsTS
    Websockets -->|depends on| Tools

    Instruments -->|depends on| CommonsTS
    Instruments -->|depends on| ConnectionsTS
    Instruments -->|depends on| Tools

    %% Python Services
    Funding -->|depends on| Commons

    Poseidon -->|depends on| Commons

    %% Library dependencies
    CommonsTS -.->|testing utils| Tools
    ConnectionsTS -->|depends on| CommonsTS
    Commons -.->|Python utils| Chainpyon

    classDef tsApp fill:#c4e3ff,stroke:#666,stroke-width:1px
    classDef pyApp fill:#d5c4ff,stroke:#666,stroke-width:1px
    classDef tsLib fill:#f9f7c0,stroke:#666,stroke-width:1px
    classDef pyLib fill:#f7e9c0,stroke:#666,stroke-width:1px

    class Orderbook,Datastream,Sequencer,Relayer,Websockets,Instruments tsApp
    class Funding,Poseidon pyApp
    class CommonsTS,ConnectionsTS,Tools tsLib
    class Commons,Chainpyon pyLib

Library Dependency Details

flowchart TB
    subgraph "TypeScript Libraries"
        CommonsTS["libs/commons-ts<br>(Core TypeScript Utilities)"]
        ConnectionsTS["libs/connections-ts<br>(DB & External Connections)"]
        Tools["libs/tools<br>(Build & Development Tools)"]
    end

    subgraph "Python Libraries"
        Commons["libs/commons<br>(Core Python Utilities)"]
        Chainpyon["libs/chainpyon<br>(Web3 Interactions)"]
    end

    %% Primary dependencies
    ConnectionsTS -->|imports| CommonsTS
    Tools -.->|dev dependency| CommonsTS

    Chainpyon -->|imports| Commons

    %% Library contents - CommonsTS
    CommonsTS --> CommonsTS_Constants["Constants<br>- ABI definitions<br>- Chain configurations<br>- Contract addresses"]
    CommonsTS --> CommonsTS_Types["Types<br>- Shared type definitions<br>- Zod schemas"]
    CommonsTS --> CommonsTS_Utils["Utilities<br>- Logging<br>- Authentication<br>- Formatting helpers"]
    CommonsTS --> CommonsTS_Middleware["Middleware<br>- Auth middleware<br>- Request processing"]

    %% Library contents - ConnectionsTS
    ConnectionsTS --> ConnectionsTS_Redis["Redis<br>- Connection setup<br>- Caching patterns"]
    ConnectionsTS --> ConnectionsTS_SQL["SQL<br>- Connection management<br>- Query helpers"]
    ConnectionsTS --> ConnectionsTS_BQ["BigQuery<br>- Schema management<br>- Query builders"]
    ConnectionsTS --> ConnectionsTS_PubSub["Pub/Sub<br>- Event publishing<br>- Subscription management"]

    %% Library contents - Commons
    Commons --> Commons_Models["Models<br>- Data models<br>- Validation schemas"]
    Commons --> Commons_Utils["Utilities<br>- Time operations<br>- Formatting"]
    Commons --> Commons_Storage["Storage<br>- Cache abstractions<br>- State management"]
    Commons --> Commons_HTTP["HTTP<br>- API clients<br>- Request handling"]

    classDef tsLib fill:#f9f7c0,stroke:#666,stroke-width:1px
    classDef pyLib fill:#f7e9c0,stroke:#666,stroke-width:1px
    classDef subComponent fill:#f9f9f9,stroke:#999,stroke-width:1px,stroke-dasharray: 5 5

    class CommonsTS,ConnectionsTS,Tools tsLib
    class Commons,Chainpyon pyLib
    class CommonsTS_Constants,CommonsTS_Types,CommonsTS_Utils,CommonsTS_Middleware,ConnectionsTS_Redis,ConnectionsTS_SQL,ConnectionsTS_BQ,ConnectionsTS_PubSub,Commons_Models,Commons_Utils,Commons_Storage,Commons_HTTP subComponent

Development Workflows

Monorepo Development Workflow

flowchart TD
    Start([Start Development]) --> Clone[Clone Repository]
    Clone --> Install[Install Dependencies<br>`yarn install`]

    subgraph "Setup Phase"
        Install --> SetupLocal[Setup Local Environment<br>`cp .env.example .env.local`]
        SetupLocal --> SetupDatabase[Setup Databases<br>`docker-compose up -d`]
    end

    subgraph "Development Phase"
        SetupDatabase --> ImplementFeature[Implement Feature/Fix]

        ImplementFeature --> BuildLib[Build Libraries<br>`nx build lib-name`]
        ImplementFeature --> Lint[Lint Code<br>`nx lint app-name`]
        ImplementFeature --> UnitTest[Run Unit Tests<br>`nx test app-name`]

        BuildLib --> DevelopmentFlow
        Lint --> DevelopmentFlow
        UnitTest --> DevelopmentFlow

        DevelopmentFlow{Development Complete?}
        DevelopmentFlow -->|No| ImplementFeature
    end

    subgraph "Validation Phase"
        DevelopmentFlow -->|Yes| ValidateChanges[Validate All Changes<br>`nx affected -t build lint test`]
        ValidateChanges --> IntegrationTest[Run Integration Tests<br>`nx test app-name --config=jest.integration.config.ts`]
    end

    subgraph "Submission Phase"
        IntegrationTest --> CreateBranch[Create Branch<br>`git checkout -b feature/name`]
        CreateBranch --> CommitChanges[Commit Changes<br>`git commit -m "feat: description"`]
        CommitChanges --> PushChanges[Push Changes<br>`git push origin feature/name`]
        PushChanges --> CreatePR[Create Pull Request]
        CreatePR --> CodeReview[Code Review Process]

        CodeReview -->|Approved| MergePR[Merge PR]
        CodeReview -->|Changes Requested| ImplementFeature
    end

    MergePR --> End([End Development])

    classDef phase fill:#f5f5f5,stroke:#333,stroke-width:1px
    classDef flowStep fill:#c4e3ff,stroke:#666,stroke-width:1px
    classDef decisionStep fill:#f9f7c0,stroke:#666,stroke-width:1px

    class Start,Clone,Install,SetupLocal,SetupDatabase,ImplementFeature,BuildLib,Lint,UnitTest,ValidateChanges,IntegrationTest,CreateBranch,CommitChanges,PushChanges,CreatePR,CodeReview,MergePR,End flowStep
    class DevelopmentFlow decisionStep

Feature Development Lifecycle

stateDiagram-v2
    [*] --> Planning

    state Planning {
        [*] --> RequirementAnalysis
        RequirementAnalysis --> DesignDiscussion
        DesignDiscussion --> TaskBreakdown
        TaskBreakdown --> [*]
    }

    Planning --> Development

    state Development {
        [*] --> SetupBranch
        SetupBranch --> ImplementCode
        ImplementCode --> LocalTesting
        LocalTesting --> CodeRefactoring
        CodeRefactoring --> WriteTests
        WriteTests --> DocumentChanges
        DocumentChanges --> [*]
    }

    Development --> Review

    state Review {
        [*] --> CodeSubmission
        CodeSubmission --> AutomatedChecks
        AutomatedChecks --> PeerReview
        PeerReview --> RevisionRequests
        RevisionRequests --> AddressComments
        AddressComments --> FinalApproval
        FinalApproval --> [*]
    }

    Review --> Deployment

    state Deployment {
        [*] --> MergeToMain
        MergeToMain --> CICDPipeline
        CICDPipeline --> StagingDeployment
        StagingDeployment --> QATesting
        QATesting --> ProductionDeployment
        ProductionDeployment --> [*]
    }

    Deployment --> Maintenance

    state Maintenance {
        [*] --> MonitorPerformance
        MonitorPerformance --> GatherFeedback
        GatherFeedback --> FixIssues
        FixIssues --> PlanImprovements
        PlanImprovements --> [*]
    }

    Maintenance --> [*]

Decision Trees

Command Selection Decision Tree

flowchart TD
    Start([Command Needed]) --> TaskType{What task are you doing?}

    TaskType -->|Building| Build{What to build?}
    TaskType -->|Testing| Test{What to test?}
    TaskType -->|Developing| Dev{Development task?}

    Build -->|One project| BuildOne[nx build project-name]
    Build -->|Affected projects| BuildAffected[nx affected -t build]
    Build -->|All projects| BuildAll[nx run-many -t build]

    Test -->|Unit tests| TestUnit[nx test project-name]
    Test -->|Integration tests| TestIntegration[nx test project-name --config=jest.integration.config.ts]
    Test -->|Affected tests| TestAffected[nx affected -t test]

    Dev -->|Lint| Lint[nx lint project-name]
    Dev -->|Type check| TypeCheck[nx run project-name:typecheck]
    Dev -->|Start service| Serve[nx serve project-name]
    Dev -->|Full validation| Validate[nx run project-name:validate]

    BuildOne --> Done([Command Selected])
    BuildAffected --> Done
    BuildAll --> Done
    TestUnit --> Done
    TestIntegration --> Done
    TestAffected --> Done
    Lint --> Done
    TypeCheck --> Done
    Serve --> Done
    Validate --> Done

    classDef start fill:#c4e3ff,stroke:#666,stroke-width:1px
    classDef decision fill:#f9f7c0,stroke:#666,stroke-width:1px
    classDef command fill:#d7f7c9,stroke:#666,stroke-width:1px
    classDef end fill:#f5d2d3,stroke:#666,stroke-width:1px

    class Start,Done start
    class TaskType,Build,Test,Dev decision
    class BuildOne,BuildAffected,BuildAll,TestUnit,TestIntegration,TestAffected,Lint,TypeCheck,Serve,Validate command

Troubleshooting Decision Tree

flowchart TD
    Start([Error Encountered]) --> ErrorType{What type of error?}

    ErrorType -->|Build Failure| BuildError{Build error type?}
    ErrorType -->|Test Failure| TestError{Test error type?}
    ErrorType -->|Runtime Error| RuntimeError{Runtime error type?}

    BuildError -->|Dependency| DependencyIssue[Check package versions<br>yarn install]
    BuildError -->|TypeScript| TSError[Check types<br>nx run project:typecheck]
    BuildError -->|Memory| MemoryError[Use retry build<br>nx run project:build:retry]

    TestError -->|Test not found| TestNotFound[Check test path<br>Check test configuration]
    TestError -->|Timeout| TestTimeout[Increase timeout<br>Fix slow tests]
    TestError -->|Environment| TestEnvError[Setup mock environment<br>Check .env.test]

    RuntimeError -->|Connection| ConnectionError[Check service connectivity<br>Verify database connection]
    RuntimeError -->|Configuration| ConfigError[Verify environment variables<br>Check config files]
    RuntimeError -->|Resource| ResourceError[Check resource limits<br>Increase allocation]

    DependencyIssue --> Solution([Solution Found])
    TSError --> Solution
    MemoryError --> Solution
    TestNotFound --> Solution
    TestTimeout --> Solution
    TestEnvError --> Solution
    ConnectionError --> Solution
    ConfigError --> Solution
    ResourceError --> Solution

    classDef start fill:#c4e3ff,stroke:#666,stroke-width:1px
    classDef decision fill:#f9f7c0,stroke:#666,stroke-width:1px
    classDef action fill:#d7f7c9,stroke:#666,stroke-width:1px
    classDef end fill:#f5d2d3,stroke:#666,stroke-width:1px

    class Start start
    class ErrorType,BuildError,TestError,RuntimeError decision
    class DependencyIssue,TSError,MemoryError,TestNotFound,TestTimeout,TestEnvError,ConnectionError,ConfigError,ResourceError action
    class Solution end

CI/CD Processes

GitHub Actions CI/CD Pipeline

sequenceDiagram
    participant Dev as Developer
    participant GitHub as GitHub
    participant Actions as GitHub Actions
    participant Cache as Cache
    participant Runner as Runner
    participant Registry as Artifact Registry
    participant Cloud as Cloud Environment

    Dev->>GitHub: Push code
    GitHub->>Actions: Trigger workflow

    Note over Actions: CI/CD Pipeline Start

    Actions->>Runner: Start build job
    Runner->>Cache: Check for cached dependencies

    alt Cache Hit
        Cache-->>Runner: Return cached dependencies
    else Cache Miss
        Runner->>Runner: Install dependencies
        Runner->>Cache: Store dependencies in cache
    end

    Runner->>Runner: Lint code
    Runner->>Runner: Type check
    Runner->>Runner: Build affected projects
    Runner->>Runner: Run unit tests

    alt Tests Pass
        Runner->>Runner: Run integration tests

        alt Integration Tests Pass
            Runner->>Registry: Build and push Docker images
            Runner->>Cloud: Deploy to development environment
            Runner-->>Actions: Report success
            Actions-->>GitHub: Update PR check status

            alt Merge to Main Branch
                Dev->>GitHub: Merge PR to main
                GitHub->>Actions: Trigger deployment workflow
                Actions->>Runner: Start deployment job
                Runner->>Registry: Pull Docker images
                Runner->>Cloud: Deploy to staging environment

                alt Staging Tests Pass
                    Runner->>Cloud: Promote to production
                    Runner-->>Actions: Report deployment success
                    Actions-->>GitHub: Update deployment status
                else Staging Tests Fail
                    Runner-->>Actions: Report deployment failure
                    Actions-->>GitHub: Update deployment status
                end
            end

        else Integration Tests Fail
            Runner-->>Actions: Report failure
            Actions-->>GitHub: Update PR check status
        end

    else Tests Fail
        Runner-->>Actions: Report failure
        Actions-->>GitHub: Update PR check status
    end

    Note over Actions: CI/CD Pipeline End

Pull Request Review Process

sequenceDiagram
    participant Dev as Developer
    participant GitHub as GitHub
    participant CI as CI System
    participant Reviewer as Reviewer

    Dev->>GitHub: Create Pull Request

    Note over GitHub: PR Created

    GitHub->>CI: Trigger CI workflow
    CI->>CI: Run automated checks

    par Automated Checks
        CI->>CI: Lint check
        CI->>CI: Type check
        CI->>CI: Run unit tests
        CI->>CI: Run integration tests
        CI->>CI: Build check
    end

    CI-->>GitHub: Report CI results

    alt CI Passes
        GitHub->>Reviewer: Notify ready for review
        Reviewer->>GitHub: Start code review

        loop Until Approved
            Reviewer->>GitHub: Request changes
            GitHub->>Dev: Notify change requests
            Dev->>GitHub: Push updates
            GitHub->>CI: Trigger CI workflow again
            CI-->>GitHub: Report updated CI results
            Reviewer->>GitHub: Continue review
        end

        Reviewer->>GitHub: Approve PR
        GitHub->>Dev: Notify PR approved

        alt Auto-merge Enabled
            GitHub->>GitHub: Merge PR
            GitHub->>Dev: Notify PR merged
        else Manual Merge
            Dev->>GitHub: Merge PR
        end

    else CI Fails
        GitHub->>Dev: Notify CI failure
        Dev->>GitHub: Push fixes
        GitHub->>CI: Trigger CI workflow again
    end

How to Contribute to Visual Documentation

Creating New Diagrams

To add new diagrams to this documentation:

  1. Use Mermaid for creating diagrams in markdown
  2. Add your diagram to the appropriate section in this document or create a new markdown file in the /documentation/diagrams/ directory
  3. Use consistent styles and colors:
  4. Blue (#c4e3ff) for primary services
  5. Yellow (#f9f7c0) for libraries
  6. Green (#d7f7c9) for external systems
  7. Red (#f5d2d3) for storage systems
  8. Grey (#f5f5f5) for grouping

Diagram Types and When to Use Them

  • Flowcharts: For processes with decision points
  • Sequence Diagrams: For interactions between components over time
  • State Diagrams: For showing state transitions in a system
  • Entity Relationship Diagrams: For data models
  • Class Diagrams: For code architecture

Best Practices

  1. Keep it simple: Focus on clarity over complexity
  2. Consistent terminology: Use the same terms across diagrams
  3. Hierarchy: Move from high-level to detailed views
  4. Colors with purpose: Use colors to distinguish different component types
  5. Regular updates: Keep diagrams in sync with code changes