Generative Adversarial Networks (GANs)

GANs (Goodfellow et al., 2014) frame generative modelling as a minimax game between two networks:

Generator G: maps random noise z ~ p_z to fake data G(z)
Discriminator D: maps data x to D(x) — probability that x is real

Objective

min_G max_D V(D, G) = E[log D(x)] + E[log(1 - D(G(z)))]

D tries to maximise: correctly classifying real as real, fake as fake
G tries to minimise: making D(G(z)) close to 1 (fool the discriminator)

At the Nash equilibrium: G(z) matches the real data distribution; D(x) = 0.5 everywhere.

Training Loop

for real_batch in data_loader:
    # 1. Train discriminator
    fake  = G(noise)
    d_loss = -torch.mean(torch.log(D(real_batch)) + torch.log(1 - D(fake.detach())))
    d_optimizer.zero_grad(); d_loss.backward(); d_optimizer.step()

    # 2. Train generator
    fake   = G(noise)
    g_loss = -torch.mean(torch.log(D(fake)))
    g_optimizer.zero_grad(); g_loss.backward(); g_optimizer.step()

Common Failure Modes

Problem	Symptom	Fix
Mode collapse	Generator produces few distinct samples	Use minibatch discrimination, Wasserstein loss
Oscillation	D and G loss oscillate, no convergence	Reduce learning rate, use EMA of G
Discriminator too strong	G gradient vanishes	Balance capacity, label smoothing

Key Variants

Variant	Innovation
DCGAN	Convolutional architecture guidelines for stable training
WGAN	Wasserstein distance as loss — more stable, meaningful metric
WGAN-GP	Gradient penalty instead of weight clipping
Conditional GAN (cGAN)	Condition G and D on class label for controlled generation
CycleGAN	Unpaired image-to-image translation without paired data
StyleGAN	High-quality face generation with style control

DCGAN PyTorch Sketch

class Generator(nn.Module):
    def __init__(self, z_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.ConvTranspose2d(z_dim, 512, 4, 1, 0),
            nn.BatchNorm2d(512), nn.ReLU(),
            nn.ConvTranspose2d(512, 256, 4, 2, 1),
            nn.BatchNorm2d(256), nn.ReLU(),
            nn.ConvTranspose2d(256, 3, 4, 2, 1),
            nn.Tanh(),
        )

    def forward(self, z):
        return self.net(z.view(-1, z.shape[1], 1, 1))

Evaluation: FID Score

Fréchet Inception Distance (FID) measures the distance between the distribution of generated images and real images in the feature space of an Inception network. Lower is better.

[[Autoencoder]] — learns a compressed representation; VAE is a probabilistic alternative to GAN
[[Semi-Supervised GANs]] — adapts GAN for semi-supervised classification
[[Feedforward]] — D and G are feedforward networks at their core