Scheduling Heterogeneous GPU Clusters Without Central Masters

The Monoculture Problem in AI Compute

Enterprise compute infrastructure has historically been designed around homogeneous server racks. Traditional cluster schedulers assume that every node possesses identical hardware configurations, predictable driver APIs, and dedicated orchestration daemons.

In modern practice, engineering organizations find themselves managing an extraordinarily fragmented hardware footprint:

• Cloud-hosted NVIDIA A100 / H100 instances for heavy model pre-training
• On-premise workstations equipped with AMD Radeon RX / Instinct cards running ROCm
• Developer laptops powered by Apple M2/M3/M4 Ultra with unified memory
• Edge and inference boxes hosting Intel Arc accelerators

Leaving non-NVIDIA silicon idle simply because traditional schedulers cannot coordinate them is a massive operational inefficiency.

Decentralized Gossip Architecture

Rather than deploying a centralized master node that requires Raft quorum consensus and complex multi-region failover, our scheduling engine adopts an encrypted peer-to-peer gossip protocol.

┌────────────────────────────────┐        ┌────────────────────────────────┐
│   Node A (Linux amd64)         │◄──────►│   Node B (Linux arm64)         │
│   2x NVIDIA A100 (CUDA)        │ Gossip │   4x AMD RX7900 (ROCm)         │
│   Native NVML Prober           │ + gRPC │   Native sysfs / ROCm Prober   │
└────────────────────────────────┘        └────────────────────────────────┘
                 ▲                                         ▲
                 │                                         │
                 ▼                                         ▼
┌────────────────────────────────┐        ┌────────────────────────────────┐
│   Node C (macOS arm64)         │◄──────►│   Node D (Windows x64)         │
│   Apple M2 Ultra (Metal)       │ Gossip │   Intel Arc A770 (XPU)         │
│   Metal Performance Prober     │ + gRPC │   Level Zero / sysfs Prober    │
└────────────────────────────────┘        └────────────────────────────────┘

Key Architectural Tenets:

1. Single-Binary Zero-CGO Distribution: Built in pure Go and Rust, the scheduler runs as a single binary with zero runtime shared-library dependencies, cross-compiling cleanly for Linux, macOS, and Windows.
2. Native Hardware Probing: Rather than requiring heavy vendor daemons, the scheduler queries hardware metrics directly:
   • NVIDIA: Direct NVML driver calls
   • AMD: Direct Linux /sys/class/drm kernel filesystem probing
   • Apple: Metal device memory allocation queries
   • Intel: Level-Zero / xpu-smi state inspection
3. Dynamic Topology Gossip: Every node continuously broadcasts its local GPU memory utilization, temperature, and pending queue length across the cluster. When a task is submitted to any node, it is automatically routed to the peer with the lowest latency and optimal memory capacity.

Production Results

By abstracting GPU dispatch across vendor boundaries, organizations can route batch data preprocessing to AMD fleets, high-throughput inference to Apple Silicon clusters, and FP8 fine-tuning to NVIDIA nodes, achieving near 100% compute utilization across all capital assets.