Machine Learning | Portfolio

1. Introduction

Machine learning automatically learns patterns from data to make predictions or decisions.
Main paradigms: supervised (classification/regression), unsupervised (clustering/dimensionality reduction), reinforcement learning.
Key challenges: generalisation, data quality, computational efficiency, interpretability.

Assume input space $\mathcal{X}$, output space $\mathcal{Y}$, and an unknown data distribution $\mathcal{D}$.
Training set $S={(x_i, y_i)}_{i=1}^{n}$ is drawn i.i.d. from $\mathcal{D}$.
Goal: pick a hypothesis $h \in \mathcal{H}$ that minimises the risk $$R(h) = \mathbb{E}_{(x,y)\sim\mathcal{D}}[\ell(h(x), y)]$$ Since $\mathcal{D}$ is unknown, we minimise the empirical risk $$\hat{R}S(h) = \frac{1}{n}\sum{i=1}^{n} \ell(h(x_i), y_i)$$ possibly with regularisation (RERM).

VC dimension: Measures expressive power of $\mathcal{H}$ and bounds generalisation error.
Bias–variance trade-off: Complex models have high variance; simple models incur high bias.
Overfitting: Low training error but poor unseen performance. Mitigate with regularisation, early stopping, cross-validation.
PAC learning: Describes when a concept class is learnable—Probably Approximately Correct.

Represent data points as vectors in high-dimensional space and use a metric to measure similarity.
$k$-NN classification: majority vote among the $k$ closest labelled neighbours.
$k$-NN regression: predict the (possibly weighted) average of the neighbours’ targets.

Virtually zero training cost—just store the data—while inference is heavy.
Distance metric choice matters: Euclidean, Manhattan, cosine, etc.
Curse of dimensionality: distances become less informative; use dimensionality reduction or distance weighting.

Define nonlinear similarities with kernel functions to capture local structure.
Manifold assumption: high-dimensional data may live on lower-dimensional surfaces—neighbourhood information remains meaningful.

Pre-pruning: depth limits, minimum samples per node, etc.
Post-pruning: build the full tree, then remove branches that raise validation error (e.g., reduced error pruning).

Interpretable, no need for feature scaling, handles nonlinearity.
Sensitive to data splits and prone to overfitting → use ensembles (Random Forest, Gradient Boosting) for robustness.

Learns weights $w$ and bias $b$ to classify via $\text{sign}(w^\top x + b)$.
Update rule for misclassified sample $(x,y)$:
$w \leftarrow w + \eta y x$, $b \leftarrow b + \eta y$.
Perceptron convergence theorem: If data is linearly separable, convergence occurs in finite steps.

Cannot solve nonlinearly separable problems (e.g., XOR) without kernels or multilayer networks.
Sensitive to learning rate $\eta$ and sample order.

Preprocessing: Handle missing values and outliers, apply scaling, encode categorical features.
Evaluation: Maintain train/validation/test splits; use k-fold CV or time-series CV as appropriate.
Regularisation: L1 encourages sparsity; L2 stabilises weights. Dropout, batch normalisation, etc., regulate neural nets.
Hyperparameter search: Grid, random, or Bayesian optimisation.
Interpretability: SHAP, LIME, and feature importance expose model behaviour.

Multiclass: One-vs-rest, one-vs-one, softmax regression.
Multi-label: A sample may belong to multiple classes—use binary relevance, classifier chains, sigmoid outputs.
Regression: Predict continuous targets using linear regression, regression trees, Gaussian processes.
Imbalanced data: Apply class weighting, SMOTE, or cost-sensitive losses.
Structured outputs: Sequence labelling with HMM/CRF, image segmentation with U-Net, etc.

Models binary probability with the sigmoid: $$P(y=1|x) = \sigma(w^\top x + b) = \frac{1}{1 + e^{-(w^\top x + b)}}$$
Train by maximising log-likelihood or minimising cross-entropy.
Decision boundary is linear—extend with kernels or polynomial features for nonlinearity.

Pros: explainable, fast to train, strong with high-dimensional sparse data.
Cons: limited expressiveness for complex nonlinear relationships—upgrade with kernels or deep networks.