MachineLearning | Portfolio

Machine Learning

Mon, 20 Oct 2025 00:00:00 +0000

1. Introduction

Machine learning automatically learns patterns from data to make predictions or decisions.
Main paradigms: supervised (classification/regression), unsupervised (clustering/dimensionality reduction), reinforcement learning.
Key challenges: generalisation, data quality, computational efficiency, interpretability.

2. Formalising Inductive Machine Learning

2.1 Problem Setup

Assume input space $\mathcal{X}$, output space $\mathcal{Y}$, and an unknown data distribution $\mathcal{D}$.
Training set $S={(x_i, y_i)}_{i=1}^{n}$ is drawn i.i.d. from $\mathcal{D}$.
Goal: pick a hypothesis $h \in \mathcal{H}$ that minimises the risk $$R(h) = \mathbb{E}_{(x,y)\sim\mathcal{D}}[\ell(h(x), y)]$$ Since $\mathcal{D}$ is unknown, we minimise the empirical risk $$\hat{R}S(h) = \frac{1}{n}\sum{i=1}^{n} \ell(h(x_i), y_i)$$ possibly with regularisation (RERM).

2.2 Inductive Bias

The No Free Lunch theorem states no learner dominates across all distributions.
We encode prior knowledge as bias and learn the weights from data.

3. Limits of Learning and Generalisation

VC dimension: Measures expressive power of $\mathcal{H}$ and bounds generalisation error.
Bias–variance trade-off: Complex models have high variance; simple models incur high bias.
Overfitting: Low training error but poor unseen performance. Mitigate with regularisation, early stopping, cross-validation.
PAC learning: Describes when a concept class is learnable—Probably Approximately Correct.

4. Geometry and Nearest Neighbours

4.1 Distance-Based Learning

Represent data points as vectors in high-dimensional space and use a metric to measure similarity.
$k$-NN classification: majority vote among the $k$ closest labelled neighbours.
$k$-NN regression: predict the (possibly weighted) average of the neighbours’ targets.

4.2 Properties

Virtually zero training cost—just store the data—while inference is heavy.
Distance metric choice matters: Euclidean, Manhattan, cosine, etc.
Curse of dimensionality: distances become less informative; use dimensionality reduction or distance weighting.

4.3 Kernels and Manifolds

Define nonlinear similarities with kernel functions to capture local structure.
Manifold assumption: high-dimensional data may live on lower-dimensional surfaces—neighbourhood information remains meaningful.

5. Decision Tree Learning

5.1 Structure

Recursively split the feature space with questions. Leaves output predictions.
Splitting criteria: information gain, Gini index, chi-squared tests.

5.2 Training (ID3/C4.5/CART)

Choose the best attribute to split on.
Recurse on child nodes.
Stop when leaves are pure or the sample count hits thresholds.

Continuous features use threshold-based binary splits.

5.3 Pruning

Pre-pruning: depth limits, minimum samples per node, etc.
Post-pruning: build the full tree, then remove branches that raise validation error (e.g., reduced error pruning).

5.4 Pros and Cons

Interpretable, no need for feature scaling, handles nonlinearity.
Sensitive to data splits and prone to overfitting → use ensembles (Random Forest, Gradient Boosting) for robustness.

6. Perceptron

6.1 Linear Separator

Learns weights $w$ and bias $b$ to classify via $\text{sign}(w^\top x + b)$.
Update rule for misclassified sample $(x,y)$:
$w \leftarrow w + \eta y x$, $b \leftarrow b + \eta y$.
Perceptron convergence theorem: If data is linearly separable, convergence occurs in finite steps.

6.2 Limitations

Cannot solve nonlinearly separable problems (e.g., XOR) without kernels or multilayer networks.
Sensitive to learning rate $\eta$ and sample order.

7. Practical Considerations

Preprocessing: Handle missing values and outliers, apply scaling, encode categorical features.
Evaluation: Maintain train/validation/test splits; use k-fold CV or time-series CV as appropriate.
Regularisation: L1 encourages sparsity; L2 stabilises weights. Dropout, batch normalisation, etc., regulate neural nets.
Hyperparameter search: Grid, random, or Bayesian optimisation.
Interpretability: SHAP, LIME, and feature importance expose model behaviour.

8. Beyond Binary Classification

Multiclass: One-vs-rest, one-vs-one, softmax regression.
Multi-label: A sample may belong to multiple classes—use binary relevance, classifier chains, sigmoid outputs.
Regression: Predict continuous targets using linear regression, regression trees, Gaussian processes.
Imbalanced data: Apply class weighting, SMOTE, or cost-sensitive losses.
Structured outputs: Sequence labelling with HMM/CRF, image segmentation with U-Net, etc.

9. Linear Models

9.1 Linear Regression

Fits $y \approx w^\top x + b$ via ordinary least squares.
Regularisation: Ridge (L2), Lasso (L1), Elastic Net.

9.2 Logistic Regression

Models binary probability with the sigmoid: $$P(y=1|x) = \sigma(w^\top x + b) = \frac{1}{1 + e^{-(w^\top x + b)}}$$
Train by maximising log-likelihood or minimising cross-entropy.
Decision boundary is linear—extend with kernels or polynomial features for nonlinearity.

9.3 Softmax Regression

For $K$ classes,
$\mathrm{softmax}(z)_k = \frac{e^{z_k}}{\sum_j e^{z_j}}$.
Optimise with cross-entropy; works well with stochastic gradient descent.

9.4 Pros and Cons

Pros: explainable, fast to train, strong with high-dimensional sparse data.
Cons: limited expressiveness for complex nonlinear relationships—upgrade with kernels or deep networks.

10. Suggested Learning Workflow

Define the problem and capture domain knowledge.
Clean and explore the data; engineer informative features.
Establish baselines with simple models (linear, tree, k-NN).
Decide on validation strategy and tune hyperparameters.
Compare models on performance, interpretability, and operational cost.
Deploy, monitor for drift, and retrain periodically.

Data Mining

Tue, 16 Apr 2024 00:00:00 +0000

1. Overview

Data mining extracts meaningful patterns and knowledge from massive datasets.
Pipeline: data cleaning → feature construction → modelling/pattern discovery → evaluation & visualisation.
Algorithms must account for the “4 V’s” of big data: volume, variety, velocity, veracity.

2. Frequent Itemset Mining

2.1 Concepts

Identify sets of items that co-occur in transaction data.
Support: fraction of transactions containing itemset $X$
$$\mathrm{support}(X) = \frac{#\text{transactions containing }X}{#\text{all transactions}}$$
Confidence: probability that $Y$ appears when $X$ does in rule $X \rightarrow Y$.
Lift: $ \frac{P(X,Y)}{P(X)P(Y)}$ measures how much more often $X$ and $Y$ appear together than by independence.

2.2 Apriori Algorithm

Top-down search that grows larger frequent itemsets from smaller ones.
Apriori principle: if an itemset is not frequent, none of its supersets can be frequent.
Steps:
1. Generate candidate 1-itemsets and compute support.
2. Keep itemsets above the minimum support threshold.
3. Combine survivors to form $(k+1)$-item candidates.
4. Repeat until no more frequent sets remain.
Limitation: repeated data scans and huge candidate sets.

2.3 FP-Growth

Compress transactions into an FP-tree without candidate generation.
Process: order items by support → insert transactions into the tree sharing prefixes → build conditional FP-trees recursively.
Requires less I/O than Apriori and scales well to large datasets.

2.4 Association Rules

Split each frequent itemset into antecedent/consequent pairs to form rule candidates.
Filter by support, confidence, lift, etc.
Applications: recommendation, market-basket analysis, intrusion detection.

3. Similar Item Search: Locality-Sensitive Hashing (LSH)

3.1 Problem

Quickly find similar objects in very large datasets (duplicate documents, image search, record linkage).

3.2 Core Idea

Design hash families so that similar items collide with high probability.
Choose hash functions according to similarity metrics:
- Jaccard → MinHash
- Cosine → Random hyperplane
- $L_p$ distance → $p$-stable hashes

3.3 MinHash with Banding

Represent documents as sets and build MinHash signatures (minimum values of multiple hash functions).
Split the signature matrix into $b$ bands of $r$ rows; hash each band to buckets.
If two signatures match in any band, treat them as a candidate pair and compute exact similarity.
Adjust $(r, b)$ to balance false negatives (more bands) vs. false positives (more rows).

3.4 LSH Pipeline

Preprocess data into sets or vectors.
Generate LSH signatures.
Extract candidate pairs.
Compute exact distances and return top-$k$ or threshold matches.

4. Clustering

4.1 Essentials

Discover natural groupings in unlabeled data.
Objectives: maximise intra-cluster similarity and minimise inter-cluster similarity.
Distance metrics: Euclidean, cosine, Hamming, DTW, etc.

4.2 K-means

Classic algorithm for roughly spherical clusters.
Procedure: choose $K$ centres (random or k-means++), assign points to nearest centre, recompute centres, repeat until convergence.
Watch out for $K$ selection (elbow, silhouette), sensitivity to outliers, and initial seeds.

4.3 Hierarchical Clustering

Builds a dendrogram based on pairwise distances.
Agglomerative (bottom-up): start with singletons and merge clusters using linkage criteria (single, complete, average, Ward).
Divisive (top-down): start with all points and split recursively.
Cut the tree at a chosen level to obtain the desired number of clusters.

4.4 Density-Based Clustering

DBSCAN/HDBSCAN treat dense regions as clusters and sparse areas as noise.
Parameters: radius $\varepsilon$ and minimum points MinPts.
Captures arbitrarily shaped clusters and automatically flags outliers.

5. Community Detection in Graphs

5.1 Definition

Partition a graph into subgraphs with dense internal connections and sparse external ones.
Use cases: social networks, recommendation engines, biological networks.

5.2 Modularity

Measures the difference between intra-cluster and inter-cluster edge density.
Louvain/Leiden algorithms greedily maximise modularity for large graphs.

5.3 Spectral Clustering

Uses eigenvectors of the graph Laplacian to embed nodes into a low-dimensional space.
Workflow: build adjacency matrix $W$, degree matrix $D$, compute normalised Laplacian $L = D^{-1/2}(D-W)D^{-1/2}$, take the smallest $k$ eigenvectors, and run k-means.
Handles nonlinear structures and non-spherical clusters.

5.4 Comparison

Modularity optimisation is fast but subject to resolution limits.
Spectral methods are more costly (eigen decomposition) yet yield high-quality partitions.
Choose based on graph size, sparsity, and runtime constraints.

6. Dimensionality Reduction

6.1 Motivation

High-dimensional data worsens computation, visualisation, and the curse of dimensionality.
Dimensionality reduction keeps as much information as possible while compressing representation.

6.2 Principal Component Analysis (PCA)

Eigen-decompose the covariance matrix to find directions of maximal variance.
Steps: standardise data → compute covariance → eigenvalues/vectors → project onto top-$k$ components.
Linear, orthogonal transform that preserves maximum variance; sensitive to nonlinearity/outliers.

6.3 Linear Discriminant Analysis (LDA)

Maximises between-class variance while minimising within-class variance.
Supervised dimensionality reduction; boosts classification performance.

6.4 Nonlinear Techniques

t-SNE: excellent for local structure visualisation, but distorts global layout.
UMAP: faster than t-SNE with better global preservation.
Autoencoders: neural networks that learn nonlinear embeddings.

6.5 Working with LSH/Clustering

Reduce dimensionality first to improve LSH efficiency and denoise features.
Applying PCA before clustering often yields higher silhouette scores.

7. Project Tips

Prioritise data quality: handle missing values, detect outliers, and scale appropriately.
Evaluation metrics:
- Frequent patterns: support, confidence, lift, conviction, Kulczynski, etc.
- Clustering/community detection: silhouette, Davies–Bouldin, modularity.
- Dimensionality reduction: reconstruction error, downstream model accuracy.
Visualise results with heatmaps, correlation matrices, network graphs, and 2D embeddings (PCA, t-SNE).
In practice, LSH is just a candidate generator—always follow up with precise distance checks.
Document your pipeline and tuning experiments to ensure reproducibility.