Machine Learning

Personal study notes written in my own words from publicly available lectures and course pages. Not affiliated with or endorsed by Stanford University. No assignment solutions are published here.

What the course is about

CS229 turns “machine learning” into a small number of repeatable moves: write down a model with parameters, write down a loss or a likelihood, and optimize it. Linear regression, logistic regression, GLMs, support vector machines, and neural networks are all that story with a different hypothesis class and objective. The second half asks the harder questions: why should a model fit on one sample work on the next one (learning theory), how to learn when there are no labels (clustering, EM, PCA, ICA), and how to learn when feedback arrives as delayed rewards (reinforcement learning).

You should be comfortable with multivariable calculus (gradients, Hessians), linear algebra (inverses, eigenvectors, positive semidefinite matrices), and basic probability (conditional probability, expectations, the Gaussian). Python and NumPy help for the snippets but are not needed to follow the ideas.

The lectures build in order. Regression introduces gradient descent and the maximum-likelihood view. Classification introduces logistic regression, generalizes it to GLMs, and contrasts it with generative models. SVMs introduce margins and kernels. A block on bias, variance, validation, and learning theory explains why models overfit. Trees, ensembles, and neural networks round out supervised learning; k-means, EM, PCA, ICA, and MDPs close the arc, followed by advice on debugging learning systems.

Lecture notes

01 · Linear regression, LMS, and the normal equations

A linear model predicts $h_\theta(x) = \theta^T x$ with a constant feature $x_0 = 1$ folded in. The training objective is the sum of squared errors over $n$ examples:

Differentiating one term gives $(h_\theta(x^{(i)}) - y^{(i)})\, x^{(i)}$, so gradient descent on one example is the LMS (Widrow-Hoff) rule $\theta := \theta + \alpha\,(y^{(i)} - h_\theta(x^{(i)}))\,x^{(i)}$. Batch gradient descent sums that over all examples per step; stochastic gradient descent applies it one example at a time and is what you use when $n$ is large.

Because $J$ is a convex quadratic, you can also set the gradient to zero in matrix form. With the design matrix $X$ (one row per example) and target vector $y$, $\nabla_\theta J = X^T X \theta - X^T y$, so

These are the normal equations. They are exact, but the inverse costs roughly cubic time in the number of features, and $X^T X$ is singular when features are linearly dependent; then use the pseudoinverse or add regularization.

What goes wrong in practice: forgetting to scale features, which makes gradient descent zigzag and forces a tiny learning rate; and reading a good training fit as a good model. The next lecture asks what a “good fit” even means and shows that the squared loss is not arbitrary.

02 · Locally weighted regression and the probabilistic interpretation

Locally weighted linear regression (LWR) keeps the linear model but refits it for every query point $x$, weighting nearby training points more heavily:

The bandwidth $\tau$ controls how local the fit is. LWR is non-parametric: there is no fixed parameter vector to store, and prediction requires the whole training set, so it is a poor choice when you need cheap inference or have millions of points. The weighted problem has its own normal equations, $\theta = (X^T W X)^{-1} X^T W y$ with $W$ diagonal.

The second theme answers why squared error is the right loss. Assume $y^{(i)} = \theta^T x^{(i)} + \epsilon^{(i)}$ with $\epsilon^{(i)} \sim \mathcal{N}(0, \sigma^2)$ independently. Then $p(y^{(i)} \mid x^{(i)}; \theta)$ is Gaussian with mean $\theta^T x^{(i)}$, and the log-likelihood of the whole training set is

Maximizing $\ell$ is the same as minimizing $J$. Least squares is maximum likelihood under Gaussian noise, and $\sigma$ does not affect the answer. This is the pattern you reuse constantly: choose a distribution for $y \mid x$, and the loss falls out. Choosing a Bernoulli instead of a Gaussian is the next lecture.

03 · Logistic regression and Newton’s method

For binary labels $y \in \lbrace 0, 1 \rbrace$, force the output into $(0, 1)$ with the sigmoid $g(z) = 1 / (1 + e^{-z})$ and set $h_\theta(x) = g(\theta^T x)$, interpreted as $p(y = 1 \mid x; \theta)$. The likelihood of one example is $h^{y}(1-h)^{1-y}$, so the log-likelihood is

Using $g'(z) = g(z)(1 - g(z))$, the gradient of one term is $(y^{(i)} - h_\theta(x^{(i)}))\, x^{(i)}$, exactly the LMS form again. Gradient ascent on $\ell$ looks identical to linear regression; only the hypothesis changed. Negating $\ell$ gives the logistic (cross-entropy) loss that most libraries minimize.

Newton’s method converges much faster when the number of features is moderate. It replaces the gradient step with

Each step solves a weighted least-squares problem, which is why this is also called iteratively reweighted least squares. Newton converges quadratically near the optimum but each iteration costs a $d \times d$ solve, so it loses to gradient methods when $d$ is in the tens of thousands.

Typical failures: perfectly separable data makes $\ell$ unbounded and the weights diverge (regularization fixes this); and reading $h_\theta$ as a calibrated probability when the model is badly misspecified. The next lecture shows that the sigmoid was not a guess.

04 · Perceptron, the exponential family, and GLMs

The perceptron replaces the sigmoid with a hard threshold, $g(z) = \mathbf{1}[z \geq 0]$, and uses the same update rule. It has no probabilistic interpretation and only converges on linearly separable data.

The exponential family is a class of distributions of the form

with natural parameter $\eta$, sufficient statistic $T(y)$ (usually $y$), and log-partition function $a(\eta)$ for normalization. The Bernoulli, Gaussian (known variance), Poisson, and multinomial are all members. For the Bernoulli, $\eta = \log(\phi / (1 - \phi))$, which inverts to $\phi = 1 / (1 + e^{-\eta})$: the sigmoid is the Bernoulli’s canonical response function.

A generalized linear model makes three assumptions: $y \mid x; \theta$ is exponential-family with natural parameter $\eta$; the prediction is $\mathbb{E}[T(y) \mid x]$; and $\eta = \theta^T x$ (the linear part). These give you a family of models by picking a distribution:

For any GLM with canonical link, the log-likelihood gradient has the familiar form $(y - h_\theta(x))\,x$, which is why the same update kept appearing. The common mistake is fitting linear regression to counts or bounded outputs when a Poisson or Bernoulli GLM matches the data’s support. Next, a different approach: model $x$ given $y$ instead of $y$ given $x$.

05 · Generative learning: GDA and naive Bayes

Discriminative models learn $p(y \mid x)$ directly. Generative models learn $p(x \mid y)$ and the prior $p(y)$, then classify with Bayes’ rule, $\arg\max_y p(x \mid y)\,p(y)$.

Gaussian discriminant analysis (GDA) assumes $y \sim \text{Bernoulli}(\phi)$ and $x \mid y = k \sim \mathcal{N}(\mu_k, \Sigma)$ with one covariance shared across classes. The maximum-likelihood estimates are what you would guess: $\phi$ is the fraction of positives, $\mu_k$ is the mean of the examples in class $k$, and $\Sigma$ is the pooled covariance of the residuals $x^{(i)} - \mu_{y^{(i)}}$. Because the two Gaussians share $\Sigma$, the quadratic terms cancel in the log-odds and the posterior has the form

for some $\theta$ that is a function of $\phi, \mu_0, \mu_1, \Sigma$. GDA therefore implies logistic regression, but not the converse: many other class-conditional choices (Poisson, for example) also give a logistic posterior. GDA needs less data when its assumptions hold and does worse when they do not; logistic regression is the more robust default.

Naive Bayes handles discrete, high-dimensional $x$ such as a bag-of-words vector by assuming the features are conditionally independent given the class:

The MLE for each $\phi_{j \mid y=1} = p(x_j = 1 \mid y = 1)$ is a simple count ratio. The assumption is false for language, yet the classifier is a strong baseline because only the argmax needs to be right. Its most visible failure, a zero probability for an unseen word, is next.

06 · Laplace smoothing and event models

If a word never appears in the spam training set, naive Bayes estimates $p(\text{word} \mid \text{spam}) = 0$, and one occurrence at test time drives the posterior to zero regardless of other evidence. Laplace smoothing pretends you saw every outcome once more than you did. For a variable with $k$ values,

The estimates still sum to one, and the adjustment vanishes as $n$ grows.

There are two ways to turn text into features. The multivariate Bernoulli event model has one binary feature per vocabulary word. The multinomial event model treats a document as a sequence of word draws, so a document of length $m$ contributes $m$ observations and every position shares the same $p(\text{word} \mid y)$ table.

The multinomial version usually classifies text better because repeated words carry evidence. Both are trained by counting, both should be smoothed, and both should multiply probabilities in log space to avoid underflow. The independence assumption also makes the posterior overconfident, so do not read it as a calibrated probability. Next, a classifier that sidesteps probability entirely and reasons about margins.

07 · SVMs and the optimal margin classifier

Write the linear classifier as $h(x) = \text{sign}(w^T x + b)$ with labels $y \in \lbrace -1, +1 \rbrace$. The functional margin of an example is $y^{(i)}(w^T x^{(i)} + b)$; it is positive when the example is classified correctly, but it can be inflated by scaling $w$ and $b$. The geometric margin divides by $\lVert w \rVert$ and is the actual signed distance from the point to the hyperplane. The margin of the whole training set is the minimum over examples.

The optimal margin classifier chooses the hyperplane with the largest geometric margin. Fixing the functional margin of the closest points to 1 turns this into a convex quadratic program:

Lagrangian duality converts this into a problem over multipliers $\alpha_i \geq 0$:

with $w = \sum_i \alpha_i y^{(i)} x^{(i)}$. The KKT conditions force $\alpha_i = 0$ for every example strictly outside the margin, so only the support vectors carry weight. Two consequences matter. First, the prediction $w^T x + b = \sum_i \alpha_i y^{(i)} \langle x^{(i)}, x \rangle + b$ depends on the data only through inner products. Second, when the data is not separable, adding slack variables with a penalty $C \sum_i \xi_i$ produces the soft-margin SVM, whose dual is identical except for the box constraint $0 \leq \alpha_i \leq C$. The primal loss is the hinge loss, $\max(0, 1 - y(w^T x + b))$, plus an $L_2$ penalty.

Typical mistakes are leaving $C$ at a default without validation and skipping feature scaling, which distorts the geometry the margin is measuring. The inner-product observation is the door to the next lecture.

08 · Kernels and SMO

Since the SVM dual and its prediction rule only touch the data through $\langle x^{(i)}, x^{(j)} \rangle$, you can replace that inner product with $K(x, z) = \langle \phi(x), \phi(z) \rangle$ for some feature map $\phi$, and train a linear classifier in the feature space without ever computing $\phi$. Two workhorse kernels:

The polynomial kernel corresponds to all monomials up to degree $d$; the Gaussian (RBF) kernel corresponds to an infinite-dimensional $\phi$, and its width $\sigma$ becomes a second hyperparameter alongside $C$. Mercer’s theorem says a symmetric $K$ is a valid kernel exactly when the kernel matrix $K_{ij} = K(x^{(i)}, x^{(j)})$ is positive semidefinite for every finite set of points. Sums, positive scalings, and products of valid kernels are valid, and the trick applies to any algorithm written in terms of inner products.

Sequential minimal optimization (SMO) solves the dual. Plain coordinate ascent fails because the constraint $\sum_i \alpha_i y^{(i)} = 0$ pins any single $\alpha_i$ once the others are fixed. SMO therefore updates a pair $(\alpha_i, \alpha_j)$ at a time: with everything else held, $\alpha_i$ is a linear function of $\alpha_j$, the objective becomes a one-dimensional quadratic, and the maximizer is found in closed form and clipped to the box $[0, C]$.

Common problems: a very small $\sigma$ in the RBF kernel memorizes the training set, and kernel matrices are $n \times n$, so kernel SVMs stop being practical at hundreds of thousands of examples. Both point at the next topic, overfitting.

09 · Bias, variance, and regularization

Fit a straight line to curved data and it underfits: error is high on training and test sets alike because the hypothesis class cannot represent the truth. That is bias. Fit a degree-ten polynomial to twelve points and it overfits: training error is near zero but test error is large because the fit tracks noise. That is variance. Expected test squared error decomposes as noise plus squared bias plus variance, and model choice trades one against the other.

Regularization is the main lever for variance. For linear regression, a weight penalty gives

A side benefit: $X^T X + \lambda I$ is always invertible. The Bayesian reading is that $L_2$ regularization is maximum a posteriori estimation under a Gaussian prior on $\theta$ with variance inversely proportional to $\lambda$; an $L_1$ penalty (lasso) corresponds to a Laplace prior and drives many weights to exactly zero. The same penalties work in logistic regression, where they also cure the divergence on separable data.

The usual mistake is tuning $\lambda$ on the training set, which always picks $\lambda = 0$. Picking it honestly is the next lecture.

10 · Cross-validation and feature selection

Model selection means choosing among hypothesis classes or hyperparameters, and training error is useless for it because a more flexible model always fits training data better. Hold-out validation splits the data into a training set and a validation (dev) set, fits each candidate on the first, scores it on the second, and retrains the winner on everything; 70/30 splits are common when data is plentiful. When it is scarce, $k$-fold cross-validation trains $k$ times, each time leaving out one fold for validation, and averages the scores; $k = 10$ is standard, and leave-one-out sets $k = n$. The test set is touched once, at the end; if you use it for decisions it is no longer a test set.

Feature selection is model selection over subsets of features. Forward search starts with no features, adds the one that most improves validation error, and repeats; it is accurate but costs many training runs. Filter methods score each feature independently, for example by mutual information with the label, $\text{MI}(x_j, y) = \sum_{x_j, y} p(x_j, y) \log \frac{p(x_j, y)}{p(x_j)\,p(y)}$, and keep the top scorers; they are cheap and a good first pass on text.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge
for lam in [0.01, 0.1, 1, 10, 100]:
    scores = cross_val_score(Ridge(alpha=lam), X, y, cv=10,
                             scoring="neg_mean_squared_error")
    print(lam, -scores.mean())

The classic error is leakage: computing feature statistics, scaling, or a feature selection step on the full data before splitting. Everything learned from data must be learned inside the training fold. The next lecture asks why any of this should generalize at all.

11 · Learning theory: empirical risk, uniform convergence, VC dimension

Define the training (empirical) error of a classifier as the fraction of mistakes on the sample, $\hat\varepsilon(h) = \frac{1}{n} \sum_i \mathbf{1}[h(x^{(i)}) \neq y^{(i)}]$, and the generalization error $\varepsilon(h)$ as the probability of a mistake on a fresh draw. Empirical risk minimization picks $\hat h = \arg\min_{h \in \mathcal{H}} \hat\varepsilon(h)$; the question is how far $\varepsilon(\hat h)$ can be from the best achievable $\varepsilon(h^{\star})$.

For a single fixed $h$, Hoeffding’s inequality bounds the gap between a mean of $n$ Bernoulli draws and its expectation. For a finite class of size $k$, the union bound extends this to every hypothesis at once: with probability at least $1 - \delta$,

This is uniform convergence, and it gives $\varepsilon(\hat h) \leq \varepsilon(h^{\star}) + 2\sqrt{\frac{1}{2n}\log\frac{2k}{\delta}}$. The first term falls as the class grows (bias); the second rises with $\log k$ (variance): the trade-off as a theorem. Rearranging gives sample complexity: the number of examples needed to be within $\gamma$ of optimal scales like $\frac{1}{\gamma^2} \log \frac{k}{\delta}$.

Infinite classes need a different size measure. A class shatters a set of points if it can realize every labeling of them, and the VC dimension is the size of the largest shatterable set. Linear classifiers in $\mathbb{R}^d$ have VC dimension $d + 1$. The bound becomes $\varepsilon(\hat h) \leq \varepsilon(h^{\star}) + O\big(\sqrt{\tfrac{\text{VC}}{n} \log \tfrac{n}{\text{VC}} + \tfrac{1}{n} \log \tfrac{1}{\delta}}\big)$, so the number of examples you need grows roughly linearly with the number of parameters. The bounds are loose; their value is in the shape of the dependence. Next, nonlinear hypothesis classes.

12 · Decision trees and ensembles

A decision tree partitions the input space with axis-aligned splits, one threshold on one feature per node, and predicts a constant in each leaf. Trees are grown greedily, picking at each node the split that most reduces a loss. Misclassification rate is a poor criterion because many useful splits leave it unchanged; entropy and the Gini index are strictly concave and reward purity gains,

A fully grown tree has near-zero training error and high variance; limit depth, require a minimum leaf size, or prune.

Ensembles average that variance away. Bagging fits $B$ trees on bootstrap resamples and averages them. If each tree has variance $\sigma^2$ and pairwise correlation $\rho$, the average has variance $\rho \sigma^2 + \frac{1 - \rho}{B}\sigma^2$: more trees kill the second term, but the first remains unless the trees are decorrelated, which random forests do by considering only a random subset of features at each split. Boosting fits a sequence of shallow, high-bias learners, each concentrating on what the previous ones got wrong: AdaBoost reweights examples, and gradient boosting fits each new tree to the negative gradient of the loss.

Boosting overfits when run for too many rounds on noisy labels; forests cannot extrapolate outside the training range. Neural networks, next, learn features instead of splitting on them.

13 · Neural networks and backpropagation

A neural network composes affine maps with elementwise nonlinearities. For a network with layers $l = 1, \dots, L$,

With $g$ the identity everywhere the whole thing collapses to a single linear map, so the nonlinearity is what buys capacity. ReLU, $g(z) = \max(0, z)$, is the default hidden activation; the output layer uses a sigmoid or softmax for classification with the same cross-entropy loss as logistic regression, or no activation with squared error for regression. Logistic regression is the special case $L = 1$.

Backpropagation is the chain rule organized to reuse work. Define $\delta^{[l]} = \partial J / \partial z^{[l]}$. For softmax with cross-entropy, the output layer gives $\delta^{[L]} = a^{[L]} - y$, the same “prediction minus target” that every GLM produced. Then going backward,

A forward pass caches every $z^{[l]}$ and $a^{[l]}$; a backward pass produces all gradients at roughly the cost of the forward pass. Mini-batch gradient descent with a tuned learning rate does the optimization; the loss is non-convex, so different initializations reach different local optima, though in practice good ones.

def relu(z): return np.maximum(0, z)
a1 = relu(W1 @ x + b1)                  # forward
p = softmax(W2 @ a1 + b2)
d2 = p - y                              # backward
dW2 = np.outer(d2, a1)
d1 = (W2.T @ d2) * (a1 > 0)
dW1 = np.outer(d1, x)

What goes wrong: initializing all weights to zero, so every hidden unit stays identical; sigmoid activations in deep stacks, whose derivatives shrink gradients toward zero; and skipping a gradient check against finite differences. The next lectures drop the labels.

14 · k-means, mixtures of Gaussians, and EM

Clustering asks for structure in unlabeled points. k-means alternates two steps until nothing changes: assign each point to the nearest centroid, $c^{(i)} = \arg\min_j \lVert x^{(i)} - \mu_j \rVert^2$, then move each centroid to the mean of its points. Each step lowers the distortion $J = \sum_i \lVert x^{(i)} - \mu_{c^{(i)}} \rVert^2$, so the algorithm converges, but only to a local minimum; run it from several random starts and keep the lowest $J$.

A mixture of Gaussians is the probabilistic version. A latent $z^{(i)} \sim \text{Multinomial}(\phi)$ picks a component, then $x^{(i)} \mid z^{(i)} = j \sim \mathcal{N}(\mu_j, \Sigma_j)$. If the $z$’s were observed this would be GDA with $k$ classes. They are not, so the expectation-maximization algorithm alternates guessing them and refitting:

Soft responsibilities $w_j^{(i)}$ replace k-means’ hard assignments, and per-cluster covariances replace spherical distance. In general, EM maximizes a lower bound on the log-likelihood obtained from Jensen’s inequality,

The E-step makes the bound tight by setting $Q_i(z) = p(z \mid x^{(i)}; \theta)$; the M-step maximizes it over $\theta$. Because the bound touches $\ell$ at the current parameters, $\ell$ never decreases. The same recipe covers factor analysis and hidden Markov models. Common failures are a component collapsing onto a single point with vanishing covariance (add a small ridge to $\Sigma_j$) and choosing $k$ by likelihood alone, which always favors more clusters. Next, reducing dimension rather than grouping points.

15 · PCA and ICA

Principal component analysis finds the directions of largest variance. After centering and scaling each feature, the direction $u$ that maximizes the variance of the projections is

The maximizer is the top eigenvector of $\Sigma$; the first $k$ principal components are the top $k$ eigenvectors, and the reduced representation is $y^{(i)} = (u_1^T x^{(i)}, \dots, u_k^T x^{(i)})$. Equivalently, PCA minimizes the squared reconstruction error of projecting onto a $k$-dimensional subspace. The eigenvalues say how much variance each component keeps, which is how you choose $k$; in practice the SVD of the centered design matrix gives the same components.

Xc = (X - X.mean(0)) / X.std(0)
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
Z = Xc @ Vt[:k].T          # k principal component scores
explained = S[:k]**2 / (S**2).sum()

Independent component analysis solves a different problem: $n$ microphones record mixtures $x = As$ of $n$ independent sources, and you want the unmixing matrix $W = A^{-1}$. The order and scale of the recovered sources are unavoidably ambiguous, and the sources must be non-Gaussian: a rotation of independent Gaussians is still independent Gaussians, so $W$ could not be identified. Choosing a heavy-tailed prior with the sigmoid as its CDF and maximizing the likelihood gives a stochastic gradient update on $W$ of the form $W := W + \alpha\big((1 - 2g(Wx))x^T + (W^T)^{-1}\big)$, where $g$ is the sigmoid applied elementwise.

PCA is a variance criterion with orthogonal directions; ICA is an independence criterion and its directions need not be orthogonal. The frequent mistake is running PCA without scaling, so one feature with large units dominates every component. The final block turns to sequential decisions with delayed reward.

16 · Reinforcement learning: MDPs, value iteration, policy iteration

A Markov decision process has states $S$, actions $A$, transition probabilities $P_{sa}(s')$, a discount $\gamma \in [0, 1)$, and a reward $R(s)$. A policy $\pi$ maps states to actions, and its value is the expected discounted return

The Markov property lets you write this recursively as the Bellman equation,

a linear system in the unknowns $V^\pi(s)$ for a finite state space. The optimal value function maximizes over actions, $V^{\star}(s) = R(s) + \gamma \max_a \sum_{s'} P_{sa}(s') V^{\star}(s')$, and the optimal policy is greedy with respect to it, $\pi^{\star}(s) = \arg\max_a \sum_{s'} P_{sa}(s') V^{\star}(s')$.

Value iteration starts from $V = 0$ and repeatedly applies the Bellman optimality update as an assignment; it converges because the update is a contraction with factor $\gamma$. Policy iteration alternates solving the linear system for $V^\pi$ and making $\pi$ greedy with respect to it.

When $P_{sa}$ is unknown, estimate it by counting observed transitions, plan in the estimated model, act, and repeat. Continuous states need discretization or a function approximator for $V$. The usual mistake is a $\gamma$ so small, or a reward so sparse, that the agent never values the goal. The last section returns to practice.

17 · Evaluating and debugging learning algorithms

When a model underperforms, the tempting fixes (more data, more features, a bigger or smaller model, more regularization) pull in opposite directions, so diagnose first. Learning curves plot training and validation error against the number of training examples. If both curves converge to a high plateau, the model has high bias: more data will not help, but more features or less regularization will. If a wide gap persists between low training error and high validation error, the model has high variance: more data, fewer features, or stronger regularization will help.

A second diagnostic separates the optimizer from the objective. If a reference solution $\theta_{\text{ref}}$ does better on the metric you care about, compare the training objective at both: when $J(\theta_{\text{ref}})$ is lower, the optimizer has not converged; when $J(\theta_{\text{learned}})$ is lower yet the metric is worse, you are minimizing the wrong objective.

For multi-stage pipelines, error analysis replaces each component in turn with ground truth and measures how much final accuracy rises; work on the stage with the largest jump. Ablative analysis runs the other way, removing one addition at a time from the full system to see which pieces earned their place. Report precision, recall, and $F_1$ rather than accuracy alone when classes are imbalanced. The general advice is to get a simple end-to-end system running first, then let diagnostics decide where the time goes.

Key equations at a glance

Common mistakes

• Running gradient descent on unscaled features, then blaming the learning rate. Standardize features first; the same fix helps SVMs, k-means, and PCA.
• Treating training error as a quality measure. Hold out a validation set or use k-fold cross-validation, and reserve the test set for a single final number.
• Computing preprocessing statistics or selecting features on the full dataset before splitting. Fit every data-dependent step inside the training fold only.
• Fitting logistic regression to separable data and getting weights that grow without bound. Add an $L_2$ penalty, which also makes the Hessian better conditioned for Newton’s method.
• Forgetting Laplace smoothing in naive Bayes, so one unseen word zeroes out the posterior. Add one to every count and the number of categories to the denominator, and work in log probabilities.
• Running k-means or EM from a single random initialization. Use several restarts, keep the best objective, and for EM add a small ridge to each covariance.
• Choosing GDA or naive Bayes for speed without checking their assumptions. When the class-conditional model is wrong, logistic regression usually generalizes better.
• Debugging a learning system by trying fixes at random. Plot learning curves first to decide between bias and variance, then spend effort where the diagnostic points.

Related courses

• CS230 — expands the neural network and backpropagation lecture into a full course on deep learning practice.
• CS231n — builds convolutional networks on top of the same loss functions, gradient descent, and regularization ideas.
• CS224n — naive Bayes and softmax regression on text are the starting point; that course replaces bag-of-words with learned representations.
• CS234 — the MDP, Bellman equation, and value/policy iteration lecture is the first week of that course, which then covers model-free and deep RL.
• CS246 — k-means, PCA/SVD, and the scaling limits of kernel methods reappear there at data-mining scale.
• CS224w — extends the feature-map and kernel view of this course to graph-structured data.