A new way of TRAINING LLMs with 3x less memory...!?

A new AI research initiative could drastically reduce GPU memory usage by two to three times, potentially even up to six times, for tasks like text generation, image generation, and image classification. This breakthrough also addresses a significant communication bottleneck in distributed training. However, its full potential hinges on successfully scaling up to models with billions of parameters, as current confirmations are limited to a few hundred million. The core of the problem lies in the VRAM memory bottlenecks of Large Language Models (LLMs), which fall into three main categories: context window memory, inferencing memory, and training memory. The context window strain is due to the growing KV cache with increased user context, an issue many, including Google's Turbo Con and Deep CV4, are trying to resolve as long-context reasoning becomes prevalent. For inferencing, the entire model weights must reside in GPU memory. A 10 billion parameter model, for instance, requires about 20 GB at FP16. This can be reduced through quantization, storing parts of the weights at lower precision (e.g., int8 or int4), which can cut VRAM needs to 10 GB or 5 GB, respectively, albeit with some performance loss. This method is already used in production by Deep CV4 to save inference costs. However, VRAM savings for training have been minimal because the shortcuts used in inference are largely unsuitable. Training is more sensitive, requiring precise tuning, and aggressive quantization can lead to model instability. Weights, gradients, activations, and optimizer updates all interact at every step. For our 10 billion parameter example, FP32 training requires 40 GB for weights alone, double the FP16 inference footprint. This number escalates further as activations for every layer must be stored for backpropagation, and optimizers like Adam keep additional copies of parameters. Cumulatively, this can lead to roughly 160 GB, four times the memory used for full-precision inferencing. The new paper by Sakana AI, titled "blocks," aims to solve this by reframing LLM training as a diffusion process. It interprets the training path through transformer layers as a denoising trajectory, leveraging the residual connections in each transformer layer. Mathematically, each layer applies a small correction to a current state to reach a next state, mirroring a diffusion model's process of iteratively denoising from noise to a clean state. This interpretation suggests that transformer depth can be seen as a denoising trajectory. Consequently, the network doesn't need to be trained as one giant connected blob. Instead, it can be sliced into independent "blocks," each assigned its own slice of noise to clean up and trained autonomously, much like a typical denoising process. This breaks the end-to-end training limitation that LLMs have faced. For example, a 12-layer transformer, typically requiring forward and backward passes through all 12 layers, can be split into three blocks (e.g., layers 1-4, 5-8, 9-12). Each block is responsible for denoising targets within a specific noise range. Since each block has its own denoising objective, backpropagation isn't needed through the entire model to train a single block. This allows for a training process where one block is sampled, a noise level within its range is chosen, the target is corrupted, only that block is run, and only that block is updated. This method can lead to memory reductions proportional to the number of blocks. Three blocks could mean three times less memory, and six blocks, six times less. The model itself doesn't shrink, but the part requiring gradients at any moment is smaller. These blocks can be trained independently, either in parallel with minimal synchronization or sequentially, offering a trade-off between memory and total training time. This could make models previously requiring expensive H100 GPUs (around $30,000) trainable on more affordable RTX 4090s (around $2,000), theoretically. Initial results on toy models (50 million to a few hundred million parameters) are promising. For image classification, a 12-layer ViT split into three blocks achieved 59.3% accuracy on CIFAR-100, close to the normal ViT's 60.5%, with a three-times memory saving. For image generation using DiT, diffusion blocks not only preserved but slightly improved performance, achieving lower (better) FID scores on CIFAR-10 and ImageNet. Masked diffusion language modeling also showed improvement. The most anticipated application is autoregressive language models. A 12-layer Llama 2-style transformer split into four blocks, where each block denoises target token embeddings conditioned on previous tokens, showed comparable or improved performance in toy experiments. For instance, on LLM 1B, diffusion blocks achieved a MAW score of 0.71 compared to 0.5 for the normal model, and 0.82 versus 0.85 on OpenWebText. While these results are promising, the scale is too small to draw definitive conclusions, and evaluation differs from full LLM pre-training benchmarks. An interesting ablation study revealed that more blocks don't always equate to better performance. While memory and speed increase with more blocks, each block becomes weaker. Blocks need sufficient depth for effective denoising. For ImageNet image generation, splitting into two blocks improved FID, but beyond three blocks, performance degraded. This suggests a balance or a minimum layer count per block is necessary. If a 128-layer model only needs six layers per block, it could yield 20 times memory savings, scaling linearly with depth, particularly if trained sequentially. Parallel training would also significantly reduce communication requirements and pipeline bubbles. Furthermore, diffusion blocks can be applied to looped transformers, which reuse the same block to simulate depth, saving parameter memory but not automatically activation memory. By training the loop itself as a denoiser, diffusion blocks allow a single forward pass per training step, a significant improvement. This research could revolutionize LLM training by breaking the end-to-end training constraint and eliminating communication