Deep Learning for Computer Vision

Personal study notes written in my own words from publicly available lectures and course pages. Not affiliated with or endorsed by Stanford University. No assignment solutions are published here.

What the course is about

CS231n asks one question and keeps widening it: how do you turn a grid of pixel intensities into a statement about what is in the picture? It starts with image classification because almost every other vision task — detection, segmentation, captioning, generation — is built from the same parts. The answer it develops is the convolutional network trained end to end with gradient descent, and later the vision Transformer, which swaps convolution for attention over image patches.

You should be comfortable with Python and NumPy, with the chain rule applied to matrices, and with the vocabulary of supervised learning (train/validation/test splits, overfitting, regularization).

The lectures fall into three blocks. The first builds the training machinery from scratch: linear classifiers, losses, stochastic gradient descent, backpropagation, and the tricks that make deep networks train at all. The second specializes to images: convolution, the canonical architectures, hardware, and the shift to attention. The third applies the machinery to harder tasks — sequences, detection, segmentation, interpretation, generation, self-supervised pretraining, video and 3D — and closes with the ethics of pointing a camera at real people.

Lecture notes

01 · Image classification as a problem

Before deep learning, vision pipelines were hand-designed: detect keypoints, describe them with local gradient histograms, pool the descriptors into a fixed-length vector, train a linear classifier on top. The turning point was a large labeled dataset (ImageNet, with on the order of a million training images across a thousand categories) meeting a deep convolutional network trained on GPUs, which cut the ImageNet challenge error rate dramatically in 2012.

The lecture frames why the problem is hard. To the computer an image is an array of integers in $[0, 255]$, and the semantic gap between those numbers and the label “cat” is enormous. The same cat produces wildly different arrays under changes of viewpoint, illumination, scale, deformation, occlusion and clutter, and two cats can differ more in pixel space than a cat and a dog.

The data-driven approach replaces rules with examples: collect a labeled dataset, fit a model, measure accuracy on images the model never saw.

What goes wrong at this stage is evaluation discipline: tuning on the test set, or reporting accuracy on data that overlaps the training set. The next lecture starts with the simplest data-driven classifier there is.

02 · Nearest neighbours and linear classifiers

A k-nearest-neighbour classifier memorizes the training set and labels a new image by majority vote among the $k$ closest training images under an $L_1$ or $L_2$ pixel distance. It also fails on images because pixel distance does not track meaning: shifting or darkening a picture moves it far in $L_2$ without changing the class. The lasting lesson is methodological: $k$ and the distance are hyperparameters, chosen on a validation split, never on the test set.

A linear classifier computes one score per class with a single matrix multiply, $s = Wx + b$, where $x$ is the flattened image. Each row of $W$ is a template for one class; the decision boundaries are hyperplanes in pixel space. To learn $W$ you need a loss:

The full objective adds a regularizer, usually $\lambda \sum_{k,l} W_{k,l}^2$, to prefer small, spread-out weights among the many that fit the data. A common bug is overflow in the exponentials — subtract the maximum score first. Both are differentiable almost everywhere, which is what the next lecture needs.

03 · Optimization and SGD

Training means finding the $W$ that minimizes average loss over the training set. Beyond a linear model the surface is non-convex, so the strategy is local: compute the gradient and step against it. Numerical gradients (finite differences) cost one forward pass per parameter; analytic gradients are exact and fast but easy to get wrong, so the practice is to derive the analytic gradient and check it numerically on a few coordinates.

Computing the gradient over the full dataset per step is wasteful. Minibatch SGD samples a small batch, treats its gradient as a noisy estimate of the true one, and updates $w \leftarrow w - \alpha \nabla_w L$. Plain SGD zigzags in badly conditioned valleys, stalls at saddle points, and is thrown by noise. Momentum accumulates a velocity:

with $\rho$ typically around $0.9$. AdaGrad divides each coordinate’s step by the root of its accumulated squared gradient; RMSProp uses a moving average so steps do not decay to zero; Adam combines a momentum-style first moment with an RMSProp-style second moment and corrects both for zero initialization.

The learning rate is the most important hyperparameter: too high and the loss explodes or plateaus early, too low and training crawls. All of this assumes you can compute $\nabla_w L$ for an arbitrary model, which is backpropagation.

04 · Backpropagation and computational graphs

Backpropagation is the chain rule applied to a computational graph. Each operation stores what it needs in the forward pass; in the backward pass it receives the gradient of the loss with respect to its output, multiplies by its local derivative, and passes the result to its inputs.

A few gates are worth memorizing: addition copies the upstream gradient to both inputs; multiplication swaps them (the gradient for one factor is the upstream gradient times the other); max routes the whole gradient to the winning input. A variable feeding several nodes sums their contributions.

The vectorized case is what you implement. For an affine layer $y = xW$ with $x$ of shape $(N, D)$, $W$ of shape $(D, M)$, and upstream gradient $\partial L / \partial y$ of shape $(N, M)$:

You never form the full Jacobian — for elementwise operations it is a huge diagonal matrix — you write the product directly.

def affine_forward(x, W, b):
    return x @ W + b, (x, W)

def affine_backward(dout, cache):
    x, W = cache
    return dout @ W.T, x.T @ dout, dout.sum(axis=0)

Frameworks automate this with autograd, but the mental model matters for debugging: a vanishing gradient is a long chain of local derivatives each below one. That is the story of the next lecture.

05 · Neural networks, activations, initialization

Two stacked affine maps are still affine, so the nonlinearity between layers is what makes depth useful. A two-layer network is $s = W_2 \, \sigma(W_1 x)$: hidden units learn intermediate templates that the second layer combines.

The choice of $\sigma$ shapes training. The sigmoid saturates at both ends, so gradients die passing through it, and its outputs are all positive, so weight gradients share a sign. Tanh is zero-centred but still saturates. ReLU, $\max(0, x)$, does not saturate for positive inputs, is cheap, and trains much faster; its cost is that a unit whose pre-activation is always negative gets no gradient and stays dead.

Initialization decides whether signals survive the forward pass. All-zero weights make every hidden unit identical, and identical gradients keep them so. Weights too small shrink activations toward zero layer by layer; too large push tanh units into saturation. The fix is to scale the initial variance to the fan-in. Xavier initialization uses

which keeps activation variance roughly constant for tanh-like units; Kaiming (He) initialization doubles it to $2 / n_{\mathrm{in}}$ because ReLU zeroes half its inputs. Xavier with ReLU in a deep network is a classic reason a model trains briefly and then stalls. A good initialization is not enough on its own, which motivates the next lecture.

06 · Training tricks

Preprocess the input: subtract the mean (per channel for images) and, for non-image data, divide by the standard deviation, so the first layer sees zero-centred inputs.

Batch normalization normalizes inside the network. For each feature it standardizes across the minibatch, then lets the network undo it with two learned parameters:

It allows larger learning rates, softens the dependence on initialization, and mildly regularizes. At test time it uses running averages of $\mu$ and $\sigma^2$ — forgetting to switch modes is a frequent bug.

Beyond weight decay, dropout zeroes each activation with probability $p$ in training and, in the inverted form, scales the survivors by $1/(1-p)$ so inference needs no change. Data augmentation (random crops, flips, colour jitter, cutout, mixup) is the cheapest accuracy available.

For hyperparameters, sample randomly on a log scale, run a coarse search for a few epochs, then narrow. Learning-rate schedules (step decay, cosine decay, a short warmup) buy a few more points; ensembles of independently trained models add a couple more, and averaging a model’s own weights over training approximates that for free. The next lecture makes the model image-shaped.

07 · Convolution layers, receptive fields, pooling

A fully connected layer on a $224 \times 224 \times 3$ image needs over 150,000 weights per hidden unit and has no notion of locality. A convolution layer slides a small filter (say $3 \times 3 \times C_{\mathrm{in}}$) across the image, computing a dot product at each position. The same weights are reused everywhere and each output depends only on a small patch. A layer has $C_{\mathrm{out}}$ filters and produces an activation volume with that many channels.

For input width $W$, filter size $K$, padding $P$ and stride $S$, the output width is

The parameter count is $K \cdot K \cdot C_{\mathrm{in}} \cdot C_{\mathrm{out}}$ weights plus $C_{\mathrm{out}}$ biases, independent of image size — this is why CNNs scale to large inputs.

def conv_params(K, c_in, c_out):
    return K * K * c_in * c_out + c_out

def conv_out_size(W, K, P, S):
    return (W - K + 2 * P) // S + 1

print(conv_params(3, 64, 128))      # 73856
print(conv_out_size(32, 3, 1, 1))   # 32

The receptive field of a unit is the input region that can influence it. Each $3 \times 3$ layer adds 2, so $L$ such layers see $1 + 2L$ pixels; stride and pooling grow it faster. Max pooling (usually $2 \times 2$, stride 2) halves the spatial size with no parameters.

Typical mistakes are off-by-one sizes and forgetting that memory, not parameter count, dominates in early layers with large spatial maps. How to assemble these layers well is next.

08 · CNN architectures

The ImageNet winners form a compact history of design ideas.

VGG’s argument is that two stacked $3 \times 3$ layers match the receptive field of one $5 \times 5$ with fewer parameters and an extra nonlinearity.

ResNet starts from a puzzle: a deeper plain network can have higher training error than a shallower one, which should not happen if the extra layers could learn the identity. The residual block makes the identity the default:

The layers learn a correction rather than the whole map, and the skip gives gradients a direct path backwards. Deeper variants use a bottleneck block ($1 \times 1$ down, $3 \times 3$, $1 \times 1$ up).

The efficient family optimizes accuracy per FLOP or parameter. MobileNet replaces a standard conv with a depthwise conv followed by a $1 \times 1$ pointwise conv; EfficientNet scales depth, width and resolution together. What training one costs in hardware is next.

09 · Hardware and software

Deep learning is matrix multiplication at scale, and a GPU is a chip with thousands of simple cores built for exactly that. A CPU has a few fast cores good at branching code; GPU throughput on dense linear algebra is one to two orders of magnitude higher for the money. TPUs and similar accelerators push the same idea further with dedicated matrix units.

Frameworks do three jobs: build a computational graph, differentiate it automatically, and run it on accelerators. The early generation (Caffe, Theano, the first TensorFlow) used static graphs defined once then executed; PyTorch popularized dynamic graphs built as Python runs, which makes debugging and control flow natural, and TensorFlow later moved that way too.

Mixed-precision training keeps activations and weights in 16-bit floats (fp16 or bfloat16) with an fp32 master copy of the weights, roughly halving memory and often doubling speed on hardware with tensor cores; fp16 needs loss scaling so small gradients do not underflow. Data loading is the hidden bottleneck — if the GPU sits idle waiting for JPEG decoding, no optimizer change will help. The next lecture leaves fixed-size inputs behind.

10 · Recurrent networks and image captioning

Images have a fixed size, but many vision outputs are sequences. A recurrent network carries a hidden state through time with the same weights at every step:

Training unrolls the recurrence into a deep graph and backpropagates through time; for long sequences you truncate to a window of steps to bound memory.

Repeated multiplication by $W_{hh}$ makes gradients through many steps explode or vanish. Gradient clipping handles the explosion; vanishing needs a different cell. The LSTM keeps a separate cell state updated by additive, gated changes (forget, input, output gates), so gradients can pass through unchanged.

Image captioning stitches the two halves of the course together: a CNN summarizes the image into a feature vector that initializes or conditions the RNN, which emits words one at a time until an end token. Soft attention lets the decoder, at each word, take a weighted average over spatial CNN features instead of one global vector, so the model looks at the relevant region as it writes. Failure modes are those of any language model: repetition, generic captions that ignore the image, and exposure bias from training only on ground-truth prefixes. The attention mechanism introduced here became the whole model in the next lecture.

11 · Attention and Transformers for vision

Attention is a differentiable lookup. Given queries $Q$, keys $K$ and values $V$, each query takes a weighted sum of values, weighted by its similarity to the keys:

The $\sqrt{d_k}$ keeps the softmax from saturating. In self-attention all three come from the same inputs, so every element can gather information from every other in one step. A Transformer block is self-attention followed by a per-token MLP, each wrapped in a residual connection and layer normalization. Attention is permutation-invariant, so position must be injected through positional embeddings.

The Vision Transformer applies this with a deliberately minimal recipe: cut the image into non-overlapping patches (commonly $16 \times 16$ pixels), flatten each, project it linearly to the model width, add a learned position embedding, prepend a learnable class token, and run a standard Transformer encoder.

The catch is inductive bias. Convolution assumes locality and translation equivariance; ViT assumes almost nothing and must learn both from data. On ImageNet alone a ViT trails a comparable ResNet, but with much larger pretraining sets it overtakes it, and strong augmentation plus distillation (DeiT) narrow the gap. Swin restores locality by attending within shifted windows and building a hierarchy, which also makes it a usable backbone for the dense tasks of the next lecture.

12 · Object detection and segmentation

Classification says what; the harder tasks say where.

Detection cannot be fixed-output regression because the number of objects varies. R-CNN runs a CNN on each of a couple of thousand class-agnostic region proposals — accurate but very slow. Fast R-CNN runs the CNN once and crops proposal features from the shared map (RoI pooling). Faster R-CNN removes that with a region proposal network: at each feature-map location, anchor boxes of preset scales and aspect ratios are scored for objectness and regressed to refined boxes, then a second stage classifies and refines again. Single-stage detectors (YOLO, SSD) predict class and box directly from anchors or grid cells; RetinaNet showed that the resulting foreground/background imbalance is what hurts them and fixed it with the focal loss.

Two tools appear everywhere. Intersection over union scores a predicted box against a true one:

Non-maximum suppression removes duplicates: sort by score, keep the best, drop remaining boxes that overlap it above a threshold, repeat.

Segmentation replaces the classifier head with dense prediction. A fully convolutional network keeps the convolutional trunk, downsamples to a coarse map, upsamples (transposed convolution or interpolation) to per-pixel scores, and adds skips from earlier high-resolution layers to sharpen boundaries. U-Net makes the encoder–decoder symmetric with a skip at every resolution. Mask R-CNN adds a mask branch to Faster R-CNN and replaces RoI pooling with RoIAlign, which avoids quantizing coordinates so masks line up with pixels. What these networks actually respond to is next.

13 · Visualizing and understanding CNNs

A trained network is a black box only if you do not look. First-layer filters of any image CNN are interpretable directly: oriented edges and colour blobs. Deeper layers are examined indirectly. Occluding patches of an image and watching the class score drop maps which regions matter.

Gradient methods are sharper. A saliency map is the gradient of the class score with respect to the input pixels, highlighting the pixels the score is most sensitive to. Grad-CAM weights the last convolutional feature maps by their gradient to produce a coarse heatmap of evidence.

Gradient ascent runs the process in reverse: start from noise and update the image to maximize a neuron or class score under a regularizer that keeps it natural. Feature inversion reconstructs an image from a layer’s activations, showing that early layers keep nearly everything and late layers keep only layout and semantics.

The same sensitivity has a dark side. An adversarial example adds a small chosen perturbation — one step of the fast gradient sign method is $x' = x + \epsilon \, \mathrm{sign}(\nabla_x L)$ — and flips the prediction while the image looks unchanged. Robustness remains open and is a warning against reading confidence as understanding. The next lecture optimizes images for a constructive purpose.

14 · Generative models

Everything so far learned $p(y \mid x)$ from labels. Generative models learn $p(x)$ from unlabeled data. Explicit models such as autoregressive models and VAEs work with the density itself; implicit models such as GANs only learn to sample.

A plain autoencoder compresses $x$ to a code $z$ and reconstructs it, but nothing makes the code space one you can sample from. The VAE makes $z$ a random variable with prior $p(z) = \mathcal{N}(0, I)$, an encoder $q_\phi(z \mid x)$ and a decoder $p_\theta(x \mid z)$, and maximizes the evidence lower bound:

The first term is reconstruction; the second keeps the encoder near the prior so decoding a prior sample gives something sensible. The reparameterization trick, $z = \mu + \sigma \odot \varepsilon$ with $\varepsilon \sim \mathcal{N}(0, I)$, moves the randomness out of the gradient’s way. VAE samples are coherent but blurry.

A GAN pits a generator $G$ mapping noise to images against a discriminator $D$ scoring real versus fake:

Samples are sharp, but training is unstable and the generator can collapse onto a few modes. Diffusion models, now the dominant family, learn to reverse a gradual noising process one small step at a time, trading slow sampling for stable training and the best sample quality. The next lecture keeps the unlabeled-data goal but aims at features rather than samples.

15 · Self-supervised and contrastive learning

Labels are expensive; pixels are free. Self-supervised learning invents a task whose labels come from the data, trains on it, and evaluates the features with a linear classifier on frozen features (linear probe) or by fine-tuning.

Contrastive learning generalizes them. Make two random augmentations of one image (crop, flip, colour distortion, blur) — a positive pair — and treat other images in the batch as negatives. Train the encoder so positives are close and negatives far. The InfoNCE loss for anchor $z_i$ and its positive $z_j$ is

with cosine similarity and temperature $\tau$: a softmax classifier over “which candidate is my partner”.

SimCLR is the plain version: large batches supply negatives, a small projection head sits between encoder and loss, and the composition of augmentations matters more than any one of them. MoCo avoids giant batches with a queue of past embeddings as negatives, produced by a slowly updated momentum copy of the encoder so the queue stays consistent. Later methods (BYOL and relatives) drop negatives and avoid collapse through asymmetry between the branches.

Masked autoencoders suit ViTs: hide a large fraction of patches, encode only the visible ones, and reconstruct the missing pixels with a light decoder. It is simple, scales well and fine-tunes strongly. Pitfalls are weak augmentation (the task becomes trivial), an untuned temperature, and judging only by linear probe when the downstream use is fine-tuning. Next, the same ideas leave the still image.

16 · Video and 3D

Video adds a time axis, and the simplest baselines are strong: classify a single frame, or average predictions across frames. Fusing frames earlier lets the model see motion. Early fusion stacks frames along the channel axis; late fusion runs a 2D CNN per frame and merges at the end; a 3D CNN slides filters over space and time together at a large compute cost. Two-stream networks add a branch on precomputed optical flow. SlowFast pairs a low-frame-rate branch for semantics with a fast, lightweight branch for motion. The constraint is memory: train on short clips and average clip predictions at test time.

3D vision begins with a choice of representation.

Neural radiance fields learn a scene as a function from position and view direction to colour and density, rendered by integrating along camera rays. Both areas are pointers rather than full treatments; what closes the sequence is not a technique but a caution.

17 · Fairness, ethics and dataset bias

Every model in the course inherits the world view of its training data. Datasets are built by particular people from particular sources with particular labeling instructions, and the model learns those choices as truth. Documented consequences include face-analysis systems whose error rates differ sharply across skin tone and gender, image datasets whose person categories carried offensive or arbitrary labels, and geographic skew where most images come from a few wealthy countries so everyday objects elsewhere are misclassified.

The concerns go past accuracy gaps. Vision systems are used for surveillance and identification, where the cost of an error falls on someone who never consented to be in the data.

The practical advice is discipline. Document datasets (what, from where, by whom, for what) and models (intended use, evaluation across subgroups, known failures). Report performance by demographic and geographic slice, not only in aggregate. Treat a benchmark score as evidence about the benchmark, not the world. Ask whether the task should exist at all, and keep a person accountable for decisions the model informs. None of this has an equation, which is exactly why it is easy to skip.

Key equations at a glance

Common mistakes

• Tuning hyperparameters on the test set. Split off a validation set and touch the test set once, at the end.
• Computing softmax on raw scores and getting NaN. Subtract the maximum score before exponentiating; the result is unchanged and the overflow is gone.
• Using Xavier or tiny random weights with ReLU in a deep network, so activations shrink to zero. Use Kaiming initialization and check activation statistics per layer.
• Leaving batch norm and dropout in training mode at evaluation. Switch to eval mode so batch norm uses running statistics and dropout is off.
• Searching the learning rate on a linear grid. Sample on a log scale, run short coarse trials, then refine around the best region.
• Miscounting convolution shapes. Apply $(W - K + 2P)/S + 1$ layer by layer, and remember the parameter count depends on filter size and channels, not image size.
• Reading high confidence as understanding. Saliency maps, per-subgroup evaluation and adversarial tests each expose a way the model can be right for the wrong reason.

Related courses

• CS229 — supplies the supervised-learning foundations (losses, regularization, gradient descent) that the first lectures here assume.
• CS230 — covers the same training machinery in a general setting, with less on vision-specific architectures.
• CS224n — develops RNNs, attention and Transformers for language; the vision Transformer lecture borrows that toolkit almost unchanged.
• CS336 — goes deep on hardware, mixed precision and scaling, expanding the brief hardware lecture here.
• CS234 — reinforcement learning, where vision models become the perception front end for agents acting in video or 3D environments.