// readme

NVSentinel

NVSentinel detects and remediates GPU faults on Kubernetes nodes

A single bad GPU can silently corrupt a training run or leave a node sitting idle for hours before anyone notices. NVSentinel detects faults as they happen, protects jobs by cordoning and draining the affected node, and remediates it with a GPU reset or a reboot, returning it to service with no paging required.

  • 🔍 Detect: real-time GPU, NIC, and system-level fault detection via DCGM, syslog, and cloud provider maintenance events
  • 🛡️ Protect: cordon and drain the affected node before a fault spreads to other jobs
  • 🔧 Remediate: auto-repair with a targeted GPU reset or a full reboot, then bring the node back into service
  • 🧩 Extensible: pluggable health monitors, drain strategies, and remediation actions

Prerequisites

The commands below get you ready for NVSentinel: the first makes sure the GPU Operator exposes DCGM as its own service, since NVSentinel queries it directly…