LLM Diffusion Models - New paradigm in language model architecture

Large Language Diffusion Models, abbreviated LLaDA (for Large Language Diffusion with Masking), represent a new paradigm in language model architecture. Traditional large language models (LLMs) are autoregressive (AR) Transformers that generate text left-to-right, predicting the next token sequentially. In contrast, diffusion-based LLMs like LLaDA use a denoising diffusion process to generate text in a parallel, coarse-to-fine manner rather than one token at a time.

Essentially, LLaDA starts with a highly masked or noisy text and iteratively refines it into a coherent sequence, analogous to how image diffusion models start from noise and gradually form a clear image. This approach challenges the long-held notion that the strengths of LLMs (scalability, in-context learning, etc.) are inherently tied to autoregressive generation. Recent research and industry discussions suggest diffusion LLMs could redefine language generation by addressing some limitations of AR models while introducing new capabilities.