Deep Learning

Personal study notes written in my own words from publicly available lectures and course pages. Not affiliated with or endorsed by Stanford University. No assignment solutions are published here.

What the course is about

CS230 is the course that turns deep learning from a set of equations into a working project. The lectures assume you already know what a gradient is and can write a loss function; the emphasis is on the decisions that decide whether a model ships: how to split data, which metric to chase, how to diagnose why a model is failing, and how to organize a search over architectures and hyperparameters without wasting a week of GPU time. Where a theory course asks why a method works, this course asks what to do on Tuesday when dev accuracy is stuck.

You should be comfortable with linear algebra, basic probability, Python, and NumPy. Familiarity with logistic regression and gradient descent (the CS229 level) makes the first weeks much faster, because the neural network material is presented as a direct generalization of a single logistic unit.

The lecture arc has three layers. The first builds the machinery: forward and backward passes, activations, initialization, regularization, optimizers, and normalization. The second is the project lifecycle: data collection, train/dev/test strategy, bias-variance diagnosis, hyperparameter search, and error analysis. The third is a tour of the architectures that dominate applied work (CNNs, detection and segmentation heads, siamese networks, recurrent and attention models, GANs) with just enough depth to read a paper and reproduce its core idea. The notes below follow that order.

Lecture notes

01 · What deep learning is and when it wins

Deep learning is function approximation with many stacked, differentiable layers trained end to end on a loss. The practical claim behind the field is simple: for tasks with large labeled datasets and rich raw inputs (pixels, audio, text), models that learn their own features beat models that rely on hand-designed features, and the gap grows with data. On a small tabular dataset, gradient-boosted trees or a linear model often win, and a deep model mostly adds variance.

The mental picture to keep is a curve of performance versus data size. Traditional methods plateau early. Larger networks keep improving as data grows, which is why scale (data, compute, model size) is the lever that drove the last decade of progress. Two supporting factors made this possible: GPUs, which make the dense matrix multiplies cheap, and algorithmic changes such as ReLU that let gradients flow through deep stacks.

Deep learning is not free. It needs labeled data at a scale that is often the real project cost, it is hard to interpret, and it fails silently on distribution shift. A useful habit is to ask three questions before starting: is there enough data, is the input rich enough that learned features matter, and is there a clear metric to optimize.

The next step is to open the box and see what a single layer computes, then stack layers and differentiate through them.

02 · Neural network basics: forward pass, backpropagation, vectorization

A layer computes an affine map followed by a nonlinearity. For layer $l$ with weights $W^{[l]}$ and bias $b^{[l]}$:

Stacking these from the input $a^{[0]} = x$ to the output gives the forward pass. The loss compares the final activation with the label; for binary classification the standard choice is cross-entropy. Backpropagation is the chain rule applied layer by layer from the loss backward, caching each $z^{[l]}$ and $a^{[l-1]}$ on the way forward so that the gradients $dW^{[l]}$ and $db^{[l]}$ can be computed without recomputation.

Vectorization is the difference between a working prototype and a toy. Instead of looping over examples, stack the $m$ examples as columns of a matrix $X$ and compute $Z = W X + b$ once, broadcasting the bias. The gradient of the weights is then a single matrix product with a $1/m$ factor.

import numpy as np
def dense_forward(A_prev, W, b):
    Z = W @ A_prev + b            # (n_l, m)
    A = np.maximum(0, Z)          # ReLU
    return A, (A_prev, W, Z)
def dense_backward(dA, cache):
    A_prev, W, Z = cache
    dZ = dA * (Z > 0)
    m = A_prev.shape[1]
    return (dZ @ A_prev.T) / m, dZ.sum(axis=1, keepdims=True) / m, W.T @ dZ

What goes wrong: shape bugs. Every tensor should have its shape written next to it, and a gradient check on a tiny network (compare backprop with finite differences) catches most sign and transpose errors. The next question is which nonlinearity $g$ to use and how to initialize $W$ so that signals neither vanish nor explode as depth grows.

03 · Activations and initialization

The nonlinearity is what makes depth useful; without it a stack of affine maps collapses into one affine map. The choices that matter in practice:

Initialization decides whether training starts at all. Setting all weights to zero makes every unit in a layer identical forever, because they receive identical gradients. Random weights that are too large push sigmoid or tanh units into saturation; too small and the activations shrink to zero through the layers. The standard fix scales the variance of the weights by the fan-in $n_{\text{in}}$. For ReLU the He initialization draws $W \sim \mathcal{N}(0, 2/n_{\text{in}})$; for tanh the Xavier initialization uses $1/n_{\text{in}}$. Biases start at zero.

A practical check: after a random initialization, look at the histogram of activations per layer. If they collapse toward zero or pile up at the saturation values, the scale is wrong.

With a network that trains, the question becomes what to train it on and how to tell if it is working, which is the project lifecycle.

04 · The full cycle of a deep learning project

A model is a small part of a deployed system. The cycle that the course keeps returning to looks like this: define the task and the metric, collect and label data, split it, train a first model quickly, analyze errors, then loop back to data or model changes, and finally deploy and monitor. Most of the calendar time goes to the data steps, not the model.

Data collection questions come first. Where will the examples come from, what does a label mean exactly, and does the data you can get match the data the system will see in production? A face-unlock system trained on studio photos and deployed on phone cameras at night is the canonical mismatch. Labeling guidelines should be written down, and a sample should be double-labeled to estimate human agreement, because that number is a ceiling on what the metric can mean.

The metric should be a single number you can optimize, plus satisficing constraints (latency under some bound, model size under some limit). Two competing metrics make every comparison a judgment call; combine them (an F1 score, a weighted average) or pick one to optimize and treat the rest as thresholds.

The first model should be built in days, not weeks. Its purpose is not to be good but to reveal which errors dominate. Deployment adds a monitoring loop: the input distribution drifts, so the dev set and the metric need periodic refreshing from live traffic.

This lifecycle depends on a careful data split, which is the next lecture.

05 · Train, dev, and test strategy and bias-variance in the deep learning era

The training set fits parameters, the dev (validation) set chooses among models and hyperparameters, and the test set gives an unbiased final estimate. The dev and test sets must come from the same distribution, and that distribution should be the one you care about in production. With millions of examples, the old 60/20/20 split is wasteful; a dev set of a few thousand examples is enough to distinguish candidate models, and the rest goes to training.

Bias-variance analysis in this setting uses four numbers. Let human-level error stand in for the Bayes error estimate.

The train-dev set is a held-out slice of the training distribution. Without it you cannot tell variance from mismatch: a model that has 1% training error and 10% dev error might be overfitting, or it might be doing fine on its own distribution and failing on a different one. The two problems have opposite fixes.

The main thing that goes wrong is chasing the wrong gap. Adding data when the problem is avoidable bias does nothing; adding capacity when the problem is variance makes things worse. Diagnose first. When the diagnosis says variance, the tools are the regularization methods that come next.

06 · Regularization: L2, dropout, augmentation, early stopping

Regularization trades a little bias for less variance. L2 (weight decay) adds $\frac{\lambda}{2m} \sum_l \lVert W^{[l]} \rVert_F^2$ to the cost, which shrinks weights toward zero at every step and keeps activations in the near-linear region where the network behaves like a simpler model. The gradient step becomes $W \leftarrow (1 - \alpha \lambda / m) W - \alpha\, dW$.

Dropout randomly zeros each hidden unit with probability $1 - p$ during training and rescales the survivors by $1/p$ (inverted dropout) so that expected activations match at test time, when no units are dropped. The effect is that no unit can rely on any single input, which spreads the weights out. Dropout should be turned off during evaluation; forgetting this is a common source of noisy dev numbers.

Data augmentation adds variance-reducing examples for free: flips, crops, small rotations, color jitter for images; noise or speed perturbation for audio. The rule is that the transform must preserve the label. Horizontal flips are fine for cats, wrong for digits.

Early stopping halts training when dev error stops improving. It works, but it couples two jobs (fitting the training set and controlling variance) into one knob, which makes the bias-variance analysis of the previous lecture harder to read. The course’s preference is to fit training data as well as possible and control variance separately with L2 or dropout.

What goes wrong: applying dropout to the input layer or the output layer, or regularizing when the diagnosis said bias. The next lecture covers the other side, making training itself faster and more stable.

07 · Optimization: mini-batch, momentum, RMSProp, Adam, schedules, batch norm

Mini-batch gradient descent processes a batch of 32 to 512 examples per step instead of the full set. The gradient is noisy, but you take many more steps per epoch and the noise helps escape flat regions. Batch sizes that are powers of two fit hardware well.

Momentum smooths the noisy gradient with an exponentially weighted average: $v \leftarrow \beta v + (1-\beta)\, dW$, then $W \leftarrow W - \alpha v$, with $\beta \approx 0.9$. RMSProp divides by a running average of squared gradients so that directions with consistently large gradients take smaller steps. Adam combines both, with bias correction for the early steps:

Default values $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$ rarely need tuning; the learning rate $\alpha$ always does. Learning-rate decay (step, exponential, or cosine) lets the model settle into a narrower minimum late in training.

Batch normalization normalizes each unit’s pre-activation across the mini-batch, then lets the network undo the normalization with learned scale $\gamma$ and shift $\beta$:

It makes deeper networks train with larger learning rates and reduces sensitivity to initialization. At test time the batch statistics are replaced by running averages collected during training. The common bug is forgetting that switch, or using batch sizes so small that the batch statistics are noise.

Once training is stable, the remaining question is which of the dozens of knobs to search, and how.

08 · Hyperparameter search and orchestration

Hyperparameters are not equally important. The learning rate matters most; then momentum terms, number of hidden units, mini-batch size, and number of layers; then the Adam constants, which almost never move. Because the important ones are unknown in advance, random search beats grid search: a grid with five values per axis tests only five distinct learning rates, while twenty-five random points test twenty-five.

Sample on the right scale. A learning rate between $10^{-4}$ and $1$ should be drawn uniformly in $\log_{10}$ space, not linearly, or almost every draw lands near the top of the range. The same applies to $1 - \beta$ for momentum, where the difference between $0.9$ and $0.99$ is a tenfold change in effective averaging window.

Search coarse to fine: sample broadly, find the region where good runs cluster, then resample within it. Two operating modes exist. If compute is scarce, babysit one model, watching the curves and adjusting the learning rate by hand. If compute is plentiful, launch many runs in parallel and compare at the end.

Orchestration is the unglamorous part: every run should log its configuration, seed, git commit, and metric history to a place where they can be compared later. Runs that cannot be reproduced are wasted. Save checkpoints so a promising run can be resumed rather than restarted.

What goes wrong: tuning on the test set, comparing runs with different seeds and reading noise as signal, and re-using hyperparameters from one dataset on another without re-searching. With the training toolkit complete, the next lectures cover the architectures it is applied to, starting with convolutions.

09 · CNN foundations and classic architectures

A convolutional layer slides a small filter over the input and computes a dot product at each position. Two properties make it the right tool for images: parameter sharing (the same edge detector is useful everywhere) and sparse connectivity (each output depends on a local patch). The output size for input width $n$, filter size $f$, padding $p$, and stride $s$ is $\lfloor (n + 2p - f)/s \rfloor + 1$. Pooling shrinks the spatial size and adds a little translation invariance; max pooling is the default.

The classic architectures tell a story of increasing depth:

A 1x1 convolution mixes channels without touching spatial structure, which makes it a cheap way to reduce depth before an expensive 3x3 or 5x5 filter. The skip connection in ResNet computes $a^{[l+2]} = g(z^{[l+2]} + a^{[l]})$. Because the identity is easy to learn (set the residual weights to zero), adding blocks cannot hurt training error, which is exactly the failure that plain deep networks showed.

What goes wrong: mismatched channel counts across a skip connection (fix with a 1x1 projection), and forgetting that a convolution’s receptive field grows slowly, so early layers see only tiny patches. Transfer learning from an ImageNet-pretrained model is the default starting point for any image task with limited data.

Classification produces one label per image. The next lecture asks where the objects are.

10 · Detection and segmentation practice

Object detection outputs a bounding box and a class for each object. The YOLO-style approach divides the image into an $S \times S$ grid and has each cell predict, for a few anchor box shapes, a confidence $p_c$, box offsets $(b_x, b_y, b_w, b_h)$, and class probabilities. The whole image is processed in one forward pass of a convolutional network, which is why it is fast enough for video.

Two pieces of geometry run the training and the post-processing. Intersection over union measures how well two boxes agree:

During training, a predicted box is counted as correct when its IoU with the ground truth exceeds a threshold, commonly $0.5$. At inference the network produces many overlapping boxes for the same object; non-max suppression keeps the highest-confidence box, discards every other box of the same class with IoU above a threshold, and repeats. Anchor boxes let one cell predict two objects of different shapes, such as a tall pedestrian and a wide car sharing a center.

Semantic segmentation labels every pixel. The U-Net architecture is an encoder that shrinks spatial size while growing channels, then a decoder that upsamples with transposed convolutions back to full resolution. Skip connections copy encoder feature maps to the matching decoder stage, which restores the fine edges the bottleneck lost. The loss is per-pixel cross-entropy, sometimes combined with a Dice term when the foreground is rare.

What goes wrong: class imbalance (most grid cells contain nothing, so the loss must weight objects up), label boxes drawn inconsistently across annotators, and NMS thresholds tuned for one object density and applied to another. The next lecture handles the case where the number of classes is unknown in advance.

11 · Face verification and one-shot learning

Face verification asks whether two images show the same person; face recognition asks which of $K$ people an image shows. A softmax over $K$ people fails for the obvious reason that new employees join and each person has one enrollment photo. The fix is one-shot learning: learn an embedding function $f(x)$ such that distance in embedding space reflects identity, then compare a new image to a stored embedding.

A siamese network runs the same convolutional encoder on two images and trains on the distance $d(x_1, x_2) = \lVert f(x_1) - f(x_2) \rVert^2$. The triplet loss trains the encoder on an anchor $A$, a positive $P$ of the same identity, and a negative $N$ of a different identity:

The margin $\alpha$ prevents the trivial solution of mapping every image to the same point. Triplet selection matters: random negatives are too easy and produce zero loss almost immediately, so training mines hard triplets where $d(A, N)$ is close to $d(A, P)$. An alternative is to treat verification as binary classification on the pair, feeding the elementwise difference of the two embeddings to a logistic unit.

At deployment, embeddings for enrolled faces are precomputed and stored, so the per-query cost is one forward pass and a nearest-neighbor lookup. Normalizing embeddings to unit length makes the threshold on distance stable across identities.

What goes wrong: the embedding collapses when the margin is too small or the negatives too easy, and dev-set pairs that share photographic conditions with training pairs give an optimistic verification rate. The same embedding-and-compare idea reappears in the next lecture, applied to words.

12 · Sequence models, word embeddings, and attention

A recurrent network processes a sequence one step at a time, carrying a hidden state $a^{\langle t \rangle} = g(W_a [a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_a)$. The same weights are used at every step, so the model handles variable-length input, and backpropagation through time is ordinary backprop on the unrolled graph. Plain RNNs struggle with long dependencies because gradients through many steps vanish or explode; gradient clipping fixes exploding, but not vanishing.

Gated units fix vanishing. The GRU keeps a memory cell and learns an update gate $\Gamma_u$ that decides how much of the old state to keep; when $\Gamma_u \approx 0$ the cell passes through unchanged and the gradient with it. The LSTM separates this into forget, update, and output gates and is the more common default. A bidirectional RNN runs one pass forward and one backward and concatenates the states, which helps whenever the whole sequence is available at prediction time, as in tagging, but not in real-time generation.

Word embeddings replace one-hot vectors with dense learned vectors so that similar words are close. They can be learned as a byproduct of a language modeling task (skip-gram, negative sampling) and transferred to small downstream datasets; averaging or sequencing them gives a sentence representation.

Attention removes the bottleneck of encoding a whole input sequence into one vector. For each output step $t$ the decoder computes weights $\alpha^{\langle t, t' \rangle}$ over the encoder states, normalized with a softmax, and forms the context as the weighted sum. The weights are learned by a small network on the previous decoder state and each encoder state. This is the mechanism that transformers later generalize with self-attention.

What goes wrong: padding tokens counted in the loss, and evaluating a machine translation model with a metric that ignores fluency. The next lecture turns from prediction to generation.

13 · GANs and adversarial examples

A generative adversarial network trains two networks against each other. The generator $G$ maps noise $z$ to a sample; the discriminator $D$ outputs the probability that an input is real. The minimax objective is

In practice the generator maximizes $\log D(G(z))$ instead of minimizing $\log(1 - D(G(z)))$, because the latter has vanishing gradients early in training when the discriminator wins easily. Training alternates a few discriminator steps with one generator step. Failure modes are well known: mode collapse (the generator produces a few samples that fool $D$ and ignores the rest of the distribution), oscillation instead of convergence, and a discriminator that becomes too good and stops giving useful gradient. Conditional GANs add a label or an image as input to both networks, which is the basis of image-to-image translation.

Adversarial examples are the other side of the same idea. A tiny perturbation $\delta$, invisible to a person, can flip a classifier’s prediction. The fast gradient sign method computes $\delta = \epsilon \cdot \text{sign}(\nabla_x \mathcal{L}(x, y))$: step in the direction that increases the loss, clipped per pixel. The reason it works is that networks are close to linear in high dimensions, so many small aligned changes add up. Defenses include adversarial training (adding perturbed examples to the training set) and preprocessing, and none is complete.

What goes wrong: judging a GAN by loss curves, which say little about sample quality, and assuming a model is robust because clean test accuracy is high. Both problems ask what the network is actually looking at, which is the subject of the next lecture.

14 · Interpretability: saliency and class activation maps

A trained classifier gives a number; interpretability methods try to say why. The simplest is the saliency map: take the gradient of the class score with respect to the input pixels, $\partial s_c / \partial x$, and display its magnitude. Pixels with large gradient are the ones where a small change moves the score most. Saliency maps are cheap and often noisy, but they catch a certain kind of bug, such as a model that classifies snow rather than the animal standing in it.

Class activation maps work at the feature-map level. For a network that ends in global average pooling followed by a dense layer, the class score is a weighted sum of the pooled feature maps, so the same weights can be applied to the unpooled maps to produce a coarse heatmap of where the class evidence sits. Grad-CAM generalizes this to any architecture by using the gradient of the score with respect to the last convolutional feature maps as the weights, then upsampling the weighted sum to the input size.

Other tools: occlusion sensitivity (slide a gray patch across the image and watch the score drop), visualizing what individual filters respond to by gradient ascent on the input, and inspecting nearest neighbors in embedding space.

None of these is a proof. They are diagnostic aids that tell you where to look, and they should be used the same way as error analysis: sample failures, look at the map, and ask whether the evidence the model used is evidence a person would accept. What goes wrong is treating a clean heatmap as validation; a model can attend to the right region for the wrong reason.

The final lecture pulls the project material together into a strategy for the whole course of a project.

15 · Reading papers and AI project strategy

Two skills separate practitioners who keep improving from those who plateau: reading the literature efficiently and deciding what to do next on a project. For papers, the method is multiple passes. First the title, abstract, and figures, which tell you whether to continue. Then the introduction, conclusion, and the figures again, skipping the math. Then the full paper, skipping only the parts you do not need. For each paper, be able to state what the authors tried to do, what the key elements were, what you can use, and what to read next. Compile a list of ten to twenty papers on a topic and read a few deeply rather than all shallowly.

Project strategy starts with error analysis. Take a hundred misclassified dev examples and tally the causes: blurry images, mislabeled examples, a confusable class. The tally gives a ceiling on how much each fix could help, which decides where to spend the next week. Mislabeled dev examples matter only if they are a large share of the error; mislabeled training examples matter less, because a network is robust to random label noise.

Transfer learning applies when the target task has little data and a related task has much: pretrain on the large task, replace the output layer, and fine-tune. Whether to freeze early layers depends on how much target data there is. Multi-task learning shares a network across tasks with a summed loss when the tasks are related and data per task is similar in size.

Collect more data when the diagnosis says variance or mismatch, and only when the data you can collect resembles the dev distribution. End-to-end learning replaces a pipeline with one model; it wins when there is enough data for the whole mapping and loses when hand-designed intermediate stages encode knowledge the data cannot supply.

Key equations at a glance

Common mistakes

• Drawing the dev and test sets from a convenient distribution rather than the production distribution. Fix: build dev and test from the data the system will actually see, even if the training set has to come from elsewhere.
• Reading a large training-to-dev gap as variance without a train-dev set, then adding regularization when the real problem is data mismatch. Fix: hold out a slice of the training distribution and compare the four error numbers.
• Sampling the learning rate uniformly on a linear scale. Fix: sample the exponent uniformly and use $10^r$; do the same for $1 - \beta$.
• Leaving dropout on or using batch statistics in batch norm at evaluation time. Fix: switch the model to evaluation mode and use running averages before computing any dev number.
• Initializing all weights to the same value or to a scale that ignores fan-in. Fix: He initialization for ReLU, Xavier for tanh, and check activation histograms after the first forward pass.
• Training a siamese network on random triplets and seeing the loss hit zero in an epoch. Fix: mine hard or semi-hard negatives so the margin term stays active.
• Applying non-max suppression across classes, so a person standing next to a bicycle deletes the bicycle. Fix: run NMS per class.
• Tuning hyperparameters against the test set. Fix: touch the test set once at the end; every model selection decision uses dev.

Related courses

• CS229 — the logistic regression, gradient descent, and bias-variance foundations that CS230 generalizes to deep networks.
• CS231n — goes much deeper on convolutional architectures, detection, and segmentation than the survey here.
• CS224n — the full treatment of embeddings, recurrent models, attention, and transformers that lecture 12 sketches.
• CS234 — reinforcement learning, which reuses the same networks as function approximators for policies and values.
• CS336 — building language models from scratch, where the optimization and orchestration habits from lectures 07 and 08 matter at scale.