Fault-Tolerant Llama Training with 2000 Synthetic Failures Every 15 Seconds and No Checkpoints on Crusoe L40S

We recently pushed the boundaries of fault-tolerant training by running a large-scale Llama model under extreme conditions. Using PyTorch’s torchft and torchtitan, we demonstrated that it's possible to train models in highly unreliable environments without relying on checkpoints. This is particularly relevant for real-world deployments where node failures are common.

What Changed

The key technical advancements here are:

• Fault-Tolerant HSDP (Hybrid Sharded Data Parallelism): An extension of FSDPv2 that incorporates fault tolerance into the gradient synchronization process.
• LocalSGD/DiLoCo: Semi-sync training algorithms designed to minimize communication overhead, making them ideal for environments with limited network bandwidth.

Why It Matters

For practitioners, this means:

• Reliability in Unreliable Environments: Training can continue seamlessly even when nodes fail frequently.
• No Checkpoints Needed: Eliminates the overhead and storage costs associated with checkpointing.
• Scalability: Proven to work at scale with large models and distributed systems.

Technical Details

Architecture Overview

The training job is structured as follows:

• Replica Groups: Each group contains multiple workers (nodes) that run a sharded model using HSDP2.
• Fault-Tolerant DDP: Synchronizes gradients across replica groups, ensuring fault tolerance at each step.
• Lighthouse Server: A global server that tracks the health of all workers via heartbeats.
• Managers: Local to each replica group, these manage real-time coordination and recovery.

Key Components

• Fault Tolerant HSDP:

  • Gradient Synchronization: Uses a fault-tolerant all-reduce mechanism to ensure that gradients are synchronized correctly even if some workers fail.
  • Best for Large-Scale Training: Ideal for setups with fast backend networks like InfiniBand.

• LocalSGD/DiLoCo:

  • Semi-Sync Training: Reduces communication overhead by synchronizing at specified intervals rather than every step.
  • Communication-Limited Scenarios: Suitable for environments where network bandwidth is limited, such as over Ethernet/TCP or geographically distributed locations.

Implementation and Results

We ran the training job on Crusoe L40S GPUs with the following setup:

• Model: A large-scale Llama model
• Failures: Simulated 2000 synthetic failures every 15 seconds
• No Checkpoints: The training process did not rely on checkpoints for recovery

Training Loss Over Failures

The training loss graph shows that the model maintained its performance despite frequent worker recoveries. Each small spike represents a non-participating worker recovering, which affects metrics but not the overall model accuracy.

Benchmarks and Performance

• Training Efficiency: The fault-tolerant system was able to maintain high efficiency even under extreme failure conditions.
• Recovery Time: Workers were quickly brought back online, minimizing downtime and ensuring continuous training.

Conclusion

This experiment demonstrates the robustness of torchft and torchtitan in handling highly unreliable environments. For practitioners looking to deploy models in real-world scenarios where node failures are common, this setup provides a reliable and efficient solution without the need for frequent checkpoints.