Diffusion Model Architecture: The Mathematical Journey from Thermodynamics to AI Image Generation

In today's age of artificial intelligence (AI), models like Stable Diffusion, DALL-E, and Midjourney generate highly visual and realistic images from just a small text prompt. The technology working behind this seemingly miraculous creation is called 'Diffusion Models' or 'Score-based generative models'. These models have brought a revolutionary transformation in the field of generative AI. However, understanding how these models fundamentally work, what their mathematical framework is, and how a beautiful image is produced from chaotic digital noise is extremely fascinating.

Mathematically, a diffusion model is a type of 'latent variable generative models'. There are two main parts to this model: the first is the 'Forward Diffusion Process' and the second is the 'Reverse Sampling Process'. The primary objective of the model is to learn the patterns of a given dataset of information or images and generate new data of the same type. It views data as a process where each data point takes a random walk with drift in a vast space of probabilities.

The scientific foundation of diffusion models lies in a branch of physics called 'Non-equilibrium Thermodynamics'. First in 2015, researchers proposed this method of generating data from extremely complex probability distributions by using the laws of physics of diffusion. Considering a collection of natural images, each image is like a point in the imaginary space of images. When noise (fuzzy clutter) is repeatedly added to an image, it transforms into a 'Gaussian distribution'. By reversing this process, original data can be recovered from completely chaotic noise.

This process can be compared to the physics 'Maxwell-Boltzmann distribution', where particle movement occurs in a potential well V(x) = 1/2 ||x||². If the system has only gradient descent (a bias towards a specific direction), all particles gather at the center and are destroyed. Therefore, to maintain equilibrium, it becomes necessary to add randomness to it. A 2020 research paper introduced the 'Denoising Diffusion Probabilistic Model' (DDPM), which wonderfully improved this technique using variational inference.

In the language of mathematics, the forward diffusion process follows a 'Markov Chain'. Starting from the original image x₀, a little bit of Gaussian noise is added in stages over a time period T. For this, certain fixed constants β₁, ..., β_T are determined, which lie between 0 and 1. At each stage, noise is added to x_{t-1} to calculate the new data point x_t. The pleasant thing is that without passing through each stage, the noisy photo of any stage x_t can be calculated directly from x₀ through just a mathematical formula. As the stage t tends to infinity, the original image is completely destroyed and becomes pure Gaussian distribution N(0, I).

The true magical power of diffusion models lies in their 'Reverse Sampling Process'. Because the noise added in the forward process is known to us, but reversing it directly is intractable. Therefore, an artificial neural network is trained that estimates the mathematics of this reverse process. This neural network is called the 'Backbone' of the model. Usually, a 'U-Net' or 'Transformer' architecture is used as this backbone. During training, knowing x_t and time t, the network accurately estimates the noise z_θ added to it.

To reduce computational cost, researchers developed 'Latent Diffusion Models' (LDMs). Running diffusion in actual pixel space is very heavy, so first an autoencoder compresses the image into a 'Latent Space' with small mathematical dimensions. Thereafter, all diffusion and denoising processes happen in this latent space. Stable Diffusion works on this very principle. Cross-attention modules and text-encoders are used to turn the text prompt given by the user into an image.

To increase user control over the generated image, 'Classifier-free guidance' (CFG) is used. In the CFG technique, the model is trained simultaneously in both conditional (i.e., with text) and unconditional (without text) ways. During sampling, the difference between these two is multiplied by a guidance scale 'w'. This makes it possible to maintain a balance between image diversity and accuracy towards the text prompt.

In 2020, Song and his colleagues presented diffusion models in the universal framework of 'Continuous-Time Stochastic Differential Equations' (SDEs). With this calculation, the diffusion process can be divided into infinitely small time periods. This approach gave birth to the 'Probability Flow ODE', which enabled fast sampling of data and accurate computation. In recent years, techniques like 'Flow Matching' and 'Rectified Flow' have come, which make the paths between noise and data straight trajectories, so that the model can generate an image in just a few steps.

By 2024, diffusion models are no longer limited to just image creation. They are being widely used in computer vision for image denoising, inpainting, super-resolution, and high-quality video generation. Besides this, diffusion architecture is also playing an important role in natural language processing for text summarization, audio-sound generation, and reinforcement learning.

For a country like India, where the use of AI technology is growing rapidly in education, agriculture, and design fields, the mathematical and architectural understanding of diffusion models is extremely important. However, training these models requires massive computing power (GPUs). In the future, more efficient, fewer-step working, and lower computational cost rectified diffusion models will make the AI revolution more accessible to the public.