Subjects | Portfolio

Machine Learning

Mon, 20 Oct 2025 00:00:00 +0000

1. Introduction

Machine learning automatically learns patterns from data to make predictions or decisions.
Main paradigms: supervised (classification/regression), unsupervised (clustering/dimensionality reduction), reinforcement learning.
Key challenges: generalisation, data quality, computational efficiency, interpretability.

2. Formalising Inductive Machine Learning

2.1 Problem Setup

Assume input space $\mathcal{X}$, output space $\mathcal{Y}$, and an unknown data distribution $\mathcal{D}$.
Training set $S={(x_i, y_i)}_{i=1}^{n}$ is drawn i.i.d. from $\mathcal{D}$.
Goal: pick a hypothesis $h \in \mathcal{H}$ that minimises the risk $$R(h) = \mathbb{E}_{(x,y)\sim\mathcal{D}}[\ell(h(x), y)]$$ Since $\mathcal{D}$ is unknown, we minimise the empirical risk $$\hat{R}S(h) = \frac{1}{n}\sum{i=1}^{n} \ell(h(x_i), y_i)$$ possibly with regularisation (RERM).

2.2 Inductive Bias

The No Free Lunch theorem states no learner dominates across all distributions.
We encode prior knowledge as bias and learn the weights from data.

3. Limits of Learning and Generalisation

VC dimension: Measures expressive power of $\mathcal{H}$ and bounds generalisation error.
Bias–variance trade-off: Complex models have high variance; simple models incur high bias.
Overfitting: Low training error but poor unseen performance. Mitigate with regularisation, early stopping, cross-validation.
PAC learning: Describes when a concept class is learnable—Probably Approximately Correct.

4. Geometry and Nearest Neighbours

4.1 Distance-Based Learning

Represent data points as vectors in high-dimensional space and use a metric to measure similarity.
$k$-NN classification: majority vote among the $k$ closest labelled neighbours.
$k$-NN regression: predict the (possibly weighted) average of the neighbours’ targets.

4.2 Properties

Virtually zero training cost—just store the data—while inference is heavy.
Distance metric choice matters: Euclidean, Manhattan, cosine, etc.
Curse of dimensionality: distances become less informative; use dimensionality reduction or distance weighting.

4.3 Kernels and Manifolds

Define nonlinear similarities with kernel functions to capture local structure.
Manifold assumption: high-dimensional data may live on lower-dimensional surfaces—neighbourhood information remains meaningful.

5. Decision Tree Learning

5.1 Structure

Recursively split the feature space with questions. Leaves output predictions.
Splitting criteria: information gain, Gini index, chi-squared tests.

5.2 Training (ID3/C4.5/CART)

Choose the best attribute to split on.
Recurse on child nodes.
Stop when leaves are pure or the sample count hits thresholds.

Continuous features use threshold-based binary splits.

5.3 Pruning

Pre-pruning: depth limits, minimum samples per node, etc.
Post-pruning: build the full tree, then remove branches that raise validation error (e.g., reduced error pruning).

5.4 Pros and Cons

Interpretable, no need for feature scaling, handles nonlinearity.
Sensitive to data splits and prone to overfitting → use ensembles (Random Forest, Gradient Boosting) for robustness.

6. Perceptron

6.1 Linear Separator

Learns weights $w$ and bias $b$ to classify via $\text{sign}(w^\top x + b)$.
Update rule for misclassified sample $(x,y)$:
$w \leftarrow w + \eta y x$, $b \leftarrow b + \eta y$.
Perceptron convergence theorem: If data is linearly separable, convergence occurs in finite steps.

6.2 Limitations

Cannot solve nonlinearly separable problems (e.g., XOR) without kernels or multilayer networks.
Sensitive to learning rate $\eta$ and sample order.

7. Practical Considerations

Preprocessing: Handle missing values and outliers, apply scaling, encode categorical features.
Evaluation: Maintain train/validation/test splits; use k-fold CV or time-series CV as appropriate.
Regularisation: L1 encourages sparsity; L2 stabilises weights. Dropout, batch normalisation, etc., regulate neural nets.
Hyperparameter search: Grid, random, or Bayesian optimisation.
Interpretability: SHAP, LIME, and feature importance expose model behaviour.

8. Beyond Binary Classification

Multiclass: One-vs-rest, one-vs-one, softmax regression.
Multi-label: A sample may belong to multiple classes—use binary relevance, classifier chains, sigmoid outputs.
Regression: Predict continuous targets using linear regression, regression trees, Gaussian processes.
Imbalanced data: Apply class weighting, SMOTE, or cost-sensitive losses.
Structured outputs: Sequence labelling with HMM/CRF, image segmentation with U-Net, etc.

9. Linear Models

9.1 Linear Regression

Fits $y \approx w^\top x + b$ via ordinary least squares.
Regularisation: Ridge (L2), Lasso (L1), Elastic Net.

9.2 Logistic Regression

Models binary probability with the sigmoid: $$P(y=1|x) = \sigma(w^\top x + b) = \frac{1}{1 + e^{-(w^\top x + b)}}$$
Train by maximising log-likelihood or minimising cross-entropy.
Decision boundary is linear—extend with kernels or polynomial features for nonlinearity.

9.3 Softmax Regression

For $K$ classes,
$\mathrm{softmax}(z)_k = \frac{e^{z_k}}{\sum_j e^{z_j}}$.
Optimise with cross-entropy; works well with stochastic gradient descent.

9.4 Pros and Cons

Pros: explainable, fast to train, strong with high-dimensional sparse data.
Cons: limited expressiveness for complex nonlinear relationships—upgrade with kernels or deep networks.

10. Suggested Learning Workflow

Define the problem and capture domain knowledge.
Clean and explore the data; engineer informative features.
Establish baselines with simple models (linear, tree, k-NN).
Decide on validation strategy and tune hyperparameters.
Compare models on performance, interpretability, and operational cost.
Deploy, monitor for drift, and retrain periodically.

Software Engineering

Fri, 17 Oct 2025 00:00:00 +0000

1. Overview

Software engineering applies disciplined, systematic methods to build complex systems.
Goals: deliver high-quality software on predictable cost/schedule while keeping maintenance and evolution in mind.
Pillars: Process, Method, Tooling, and Quality.

2. Software Development Life Cycle (SDLC)

2.1 Waterfall (Requirements → Design → Implementation → Verification → Maintenance)

Fully sequential. Works for contract-heavy or change-resistant projects.
Pros: strong documentation, straightforward management.
Cons: slow feedback, poor at absorbing requirement changes.

2.2 Iterative & Incremental

Repeated cycles of design/build/test deliver small increments and expose risk early.
Examples: Spiral model (risk-driven), RUP, incremental delivery.

2.3 Agile

Emphasises customer collaboration and rapid iteration: Scrum, Kanban, etc.
Each sprint covers planning → development → review → retrospective.
Prioritises working software over exhaustive documents, yet minimal artefacts are still maintained.

3. Requirements Engineering

3.1 Types

Functional: behaviours and use cases the system must perform.
Non-functional: performance, security, availability, compliance—quality attributes.
Domain-specific: rules tied to particular industries or organisations.

3.2 Elicitation Techniques

Interviews, workshops, surveys, observation, prototyping.
Agile-friendly user stories (“As a … I want … so that …”) capture intent.

3.3 Analysis & Specification

Resolve conflicts, prioritise (MoSCoW, WSJF).
Document with UML use-case diagrams, structured templates, and SRS documents.
Govern changes via Change Control Boards and versioning.

4. System and Software Design

4.1 Architecture Design

Define structures to satisfy quality attributes.
Patterns: layered, client–server, microservices, event-driven.
Styles: REST, SOA, pipe-and-filter.
Artefacts: C4 model diagrams, Architecture Decision Records (ADRs).

4.2 Detailed Design

Specify modules, classes, interfaces, and data structures.
UML diagrams (class, sequence, state, activity).
Principles: SOLID, dependency injection, GoF patterns (strategy, observer, factory, etc.).

4.3 Reuse

Leverage libraries, components, frameworks.
Product line engineering derives variants from a shared asset base.

5. Project Planning and Management

5.1 Estimation

Use COCOMO II, Function Points, Story Points.
Management artefacts: WBS, Gantt charts, Critical Path Method.

5.2 Risk Management

Identify → analyse (probability/impact) → mitigate/avoid/transfer/accept.
Focus on staffing, schedule, and technical uncertainties.

5.3 Configuration Management

Track versions of code, docs, build artefacts with Git/SVN.
Adopt branching strategies (Git Flow, trunk-based) and peer reviews.

6. Implementation & Quality Techniques

6.1 Coding Standards

Naming conventions, style guides (PEP8, Google style).
Automate quality via linters, formatters, static analyzers.

6.2 Refactoring

Improve structure without changing behaviour. Remove smells (long method, large class, feature envy).
Use CI pipelines and regression tests to ensure safety.

6.3 Performance Tuning

Profile to locate bottlenecks. Apply caching, async processing, data-structure optimisations.
Reference models include Amdahl’s and Little’s laws.

7. Testing & QA

7.1 Test Levels

Unit: functions/classes, often TDD.
Integration: interactions between components.
System: end-to-end functional/non-functional checks.
Acceptance: confirm user needs (UAT).

7.2 Techniques

White-box: path, branch coverage.
Black-box: equivalence partitioning, boundary analysis.
Static reviews: peer review, walkthrough, inspection.

7.3 Automation & CI/CD

Frameworks: JUnit, pytest, Cypress.
CI tools: GitHub Actions, Jenkins for build/test/deploy automation.
Track coverage and quality gates (e.g., SonarQube).

8. Operations and Maintenance

8.1 Maintenance Types

Corrective: fix defects.
Adaptive: respond to environment changes.
Perfective: enhance performance/features.
Preventive: eliminate latent issues.

8.2 DevOps

Integrates development and operations with CI/CD, monitoring, IaC.
Tooling: Docker, Kubernetes, Terraform, Prometheus, Grafana.
Close feedback loop via logs, APM, feature flags.

9. Quality Models and Standards

ISO/IEC 25010: functionality, reliability, usability, efficiency, maintainability, portability, security, compatibility.
CMMI: organisational process maturity (levels 1–5).
ISO/IEC 12207: software life-cycle processes.
IEEE standards: templates for SRS (IEEE 830), test documentation (IEEE 829), etc.

10. Tools and Collaboration

Requirements tracking: Jira, Azure DevOps, Trello.
Documentation: Confluence, Notion, MkDocs; diagrams with PlantUML, Mermaid.
Code collaboration: GitHub, GitLab; adopt PR reviews, pair/mob programming.
Metrics: defect density, MTBF, defect discovery rate, lead time.

11. Ethics and Professionalism

Follow ACM/IEEE codes of ethics.
Address privacy, accessibility, algorithmic fairness, licence compliance.
Conduct postmortems and root-cause analyses to foster organisational learning.

Databases

Wed, 30 Apr 2025 00:00:00 +0000

1. Database Normalization

1.1 Definition

Normalization decomposes relations so that only related attributes remain together, eliminating anomalies and redundancy.

1.2 Anomalies

Insertion anomaly: need to insert unrelated data to add a new fact.
Update anomaly: modifying one duplicate tuple but not the others causes inconsistencies.
Deletion anomaly: removing a tuple unintentionally deletes valuable information.

Normalization uses functional dependencies to prevent these anomalies.

2. Functional Dependencies

2.1 Definition

In relation R, if attribute set $X$ determines $Y$, we denote $X \rightarrow Y$—tuples with the same $X$ must share the same $Y$.

2.2 Guidelines

Derive dependencies from attribute semantics and business rules, not from sample data alone.
Primary and candidate keys typically determine all other attributes.

2.3 Full Functional Dependency

$X \rightarrow Y$ but no proper subset of $X$ determines $Y$.
Indicates $Y$ depends on the entire composite key.

2.4 Partial Functional Dependency

When $Y$ depends on only part of a composite key.
Break such relations into 2NF to remove partial dependencies.

3. Normal Forms

3.1 First Normal Form (1NF)

All attribute values are atomic (no repeating groups or arrays).
Remove multi-valued attributes. If non-key attributes do not fully depend on the key, move to 2NF.

3.2 Second Normal Form (2NF)

In 1NF and every non-key attribute is fully functionally dependent on the primary key.
Eliminates partial dependencies; composite-key relations may require decomposition.

3.3 Third Normal Form (3NF)

In 2NF and no non-key attribute is transitively dependent on the primary key (no $A \rightarrow B$, $B \rightarrow C$ implying $A \rightarrow C$ among non-keys).

3.4 Boyce–Codd Normal Form (BCNF)

Stronger than 3NF: every determinant in a functional dependency is a candidate key.
Needed when multiple candidate keys exist and 3NF still allows anomalies.

Higher normal forms reduce redundancy but may increase join cost. Aim for a balance; 5NF is rarely necessary in practice.

3.5 Lossless Decomposition & Dependency Preservation

Decompose relations without losing information and retain essential dependencies.
Joining the decomposed tables must reconstruct the original data.

4. Views

4.1 Definition

A view is a virtual table defined by a SELECT query over base tables. It stores no data, providing abstraction, reuse, and security.

4.2 Syntax

CREATE VIEW view_name [(column_list)]
AS
SELECT ...
[WITH CHECK OPTION];

WITH CHECK OPTION enforces view predicates during DML operations.
ORDER BY is not allowed inside view definitions; apply ordering when querying the view.

4.3 DML Restrictions

View updates propagate to base tables only when:
1. key and NOT NULL columns are present,
2. no aggregates, DISTINCT, GROUP BY, or UNION,
3. simple single-table views.
Dropping a view leaves base tables intact, but drop dependent constraints first.

4.4 Benefits

Simplify complex queries.
Provide column-level access control.
Maintain logical independence when base schema changes.

5. Indexing

5.1 What & Why

An index is a data structure that enables fast record retrieval—most RDBMSs use B+ Trees (sometimes hashes).
Conceptually stores key-value pairs (key → record address).

5.2 How It Works

Look up the search key in the index.
Follow the pointer to fetch the record, avoiding full table scans and reducing complexity to O(log n).

5.3 Design Considerations

Best for selective predicates, frequent sorting/grouping, join keys, read-heavy workloads.
Caveats: maintenance cost on INSERT/UPDATE/DELETE, functions/casts may prevent usage, low selectivity columns yield little benefit.

5.4 Clustered vs. Nonclustered

Clustered index: table rows are stored in index order (e.g., InnoDB primary key).
Nonclustered index: separate structure points to the actual data pages.

6. Triggers

6.1 Concept

Stored routines that fire automatically in response to events (DML, DDL, login).
Enforce integrity rules, record audit logs, maintain derived values.

6.2 Types

BEFORE triggers: execute prior to the event—validate or adjust values.
AFTER triggers: run post-event—log or cascade changes.
INSTEAD OF triggers: mainly for views; intercept operations and apply custom logic.

6.3 Cautions

Complex trigger chains can hurt performance or cause recursion.
Failures roll back the entire transaction—add proper exception handling.

7. Stored Procedures

7.1 Definition

Precompiled SQL code stored in the database; accepts parameters to automate recurring tasks.
Executes on the server, reducing network trips and centralising business logic.

7.2 Example

CREATE PROCEDURE proc_name(IN p INT, OUT result VARCHAR(50))
BEGIN
 -- SQL statements
END;

Invoke with CALL proc_name(value, @out);

7.3 Pros & Cons

Pros: reusable, secure (permission-controlled), potential performance gains.
Cons: higher complexity, tricky version control and testing, DBMS vendor lock-in.

7.4 Procedure vs. Function

Functions must return a value and can appear inside SELECT queries.
Procedures may omit a return value, use OUT parameters, and manage transactions.

8. Transactions

8.1 Definition

A transaction is a logical unit of work that groups multiple SQL statements. Commit all if successful, otherwise roll back to maintain consistency.

8.2 ACID Properties

Atomicity: all-or-nothing execution.
Consistency: preserves integrity constraints.
Isolation: concurrent transactions do not interfere.
Durability: committed changes survive failures.

8.3 Isolation Levels

Read Uncommitted: allows dirty reads.
Read Committed: prevents dirty reads (default in many systems).
Repeatable Read: guarantees repeatable reads; prevents non-repeatable reads.
Serializable: equivalent to serial execution; safest but slowest.

8.4 Concurrency Control

Locks (shared/exclusive, two-phase locking), optimistic control, timestamp ordering, MVCC.

8.5 Design Tips

Keep transactions short to avoid deadlocks and lock contention.
Common pattern: BEGIN; ...; COMMIT; with ROLLBACK; on errors.

9. Production Checklist

Balance normalization and denormalization according to read/write patterns.
Maintain only necessary indexes and review usage statistics regularly.
Include logging and error handling in triggers/procedures for auditability.

Operating Systems

Wed, 30 Apr 2025 00:00:00 +0000

1. History and Concepts

1.1 Batch Processing

Early OSes executed a single job at a time with no user intervention.
Operators manually queued jobs; the next job started only after the previous finished.
Idle CPU time abounded while waiting for I/O, making computation inefficient.

1.2 Limitations of Batch

CPU idle time: Fast CPUs waited for slow I/O devices.
Slow job switching: Manual intervention delayed the transition between jobs.

1.3 Spooling (Fixes Issue 1)

Buffers I/O so computation and device work overlap.
Example: printer spooling—documents are queued on disk; printers fetch at their own pace while the CPU continues other work.
Benefit: CPU need not stall for I/O completion.
Caveat: If the program depends immediately on I/O results (e.g., read()), spooling alone cannot help.

1.4 Automatic Job Sequencing (Fixes Issue 2)

Scheduling software automatically loads the next job as soon as the previous one finishes.
Improves throughput compared with manual batching.
Requires multiprogramming to keep multiple jobs resident in memory.

Multiple jobs reside in memory; CPU switches to another while the current one waits for I/O.
FCFS plus priority-based policies balance fairness.
Time sharing: CPU time is sliced so each job sees frequent progress. Heavy context switching gives the illusion of concurrency but adds overhead.

2. OS and Computer-System Structure

2.1 Four Components

Hardware: CPU, memory, I/O devices provide raw resources.
Operating System: Arbitrates resource access among programs.
Application Programs: Decide how resources are used to satisfy user tasks.
Users: People or other systems interacting with the computer.

2.2 Bus Architecture

Early machines used a single bus—CPU, memory, I/O shared one channel—causing bottlenecks.
Hierarchical dual-bus design emerged:
- System bus: connects fast CPU/memory.
- I/O bus: connects slower peripherals.
- Northbridge: controller for high-speed devices.
- Southbridge: handles slower devices and legacy ports.

3. Execution Modes: User vs. Kernel

CPUs provide at least two privilege levels for system protection.
Each mode controls which instructions and memory regions are accessible.

3.1 Kernel Mode

Privileged; OS services, drivers, and resource managers run here with full hardware access.

3.2 User Mode

Restricted; applications cannot execute privileged instructions and must request services via the OS.

3.3 Mode Switching

System calls, interrupts, and traps flip the mode bit and jump into kernel handlers.
After handling, control returns to user mode at the interrupted point.

4. CPU Event Handling

4.1 Hardware Interrupts

Asynchronous events from devices.
Sequence: save context → jump to ISR → perform service → restore context → resume execution.
Priorities decide which interrupts pre-empt others; keep ISRs short.

4.2 Traps (Software Interrupts)

Synchronous events within a program: e.g., divide-by-zero, system calls.
Trigger a switch to kernel mode but may reuse existing context without full saving.

4.3 I/O Device Components

Bus: data, address, and control lines.
Device registers: control, status, input, output; mapped to memory for CPU access.
Controller: intermediates between devices and buses, arbitrating use.

5. Device Access Methods

5.1 I/O Instructions

CPU-specific instructions communicate directly with devices.
Non-portable because instruction sets differ.

5.2 Memory-Mapped I/O

Maps device registers into the main address space so ordinary loads/stores access hardware.
Improves portability and programming simplicity.

6. I/O Completion Techniques

6.1 Polling

CPU repeatedly checks device status.
Fast response for quick devices but wastes CPU cycles when waiting.

6.2 Interrupt-Driven I/O

CPU continues other tasks until a device interrupts to signal completion.
Saves busy-wait time but incurs context-switch overhead and may delay detection of very fast events.

Rule of thumb: Fast devices favour polling; slower ones benefit from interrupts.

7. Data Transfer Methods

7.1 Programmed I/O

CPU performs all data movement; expensive for large transfers.

7.2 Direct Memory Access (DMA)

A DMA controller transfers data between memory and devices on behalf of the CPU.
Process: CPU issues command → DMA moves data → completion interrupt notifies CPU.
Pros: frees CPU cycles; excels for high-volume transfers.
Cons: additional hardware cost and possible bus contention.

8. OS Design Considerations

8.1 General vs. Special Purpose

General-purpose OSes serve diverse workloads; special-purpose OSes target embedded, real-time, or appliance scenarios.
Balancing performance, fairness, and reliability requires deliberate trade-offs.

8.2 Layering vs. Modularity

Layering: strict top-down structure; upper layers use lower-layer services only.
Modularity: independent components communicate through explicit interfaces; freer ordering but demands dependency management.
Modern OSes blend both: layered architecture populated with modular subsystems for stability and extensibility.

9. Kernel Structures and System Calls

9.1 System Calls & Mode Switch

User code invokes an API → generates a trap → CPU enters kernel mode → trap handler executes → returns to user mode.

9.2 Parameter Passing

Registers: fastest but few parameters fit.
Memory blocks: store arguments in memory and pass the address.
Stack: push arguments in user mode; kernel pops them.

9.3 Monolithic Kernel

All kernel services share one address space.
Pros: minimal overhead for cross-service calls.
Cons: tightly coupled; a faulty module can crash the entire kernel.

9.4 Address Spaces

Define accessible address ranges. Sharing the same space allows direct referencing; separate spaces reinforce protection boundaries.

9.5 Microkernel

Decomposes services into user-space servers communicating via IPC.
Pros: low coupling, easy maintenance, unload unused services.
Cons: frequent address-space switches and IPC overhead can hurt performance.

9.6 Hybrid Kernel

Real-world kernels typically combine monolithic cores with loadable modules or user-space servers, balancing performance and flexibility.

Computer Networks

Tue, 15 Apr 2025 00:00:00 +0000

Chapter 1. Introduction

1.1 Evolution of Data Networks

Consider commercial viability, line installation cost (very high), and relatively cheaper equipment.
Evaluate whether new technologies can reuse existing infrastructure.
- Success: xDSL leveraged telephone lines for broadband.
- Failure: 1970s video phones—insufficient demand relative to infrastructure cost.
Modems convert digital ↔ analogue signals so legacy phone lines can transport data.

1.2 Development Path

Telcos gradually expanded data services while keeping the telephone backbone.
Phone number ↔ IP mapping allowed voice-over-IP on regular lines, though with lower audio quality.

1.3 Internet & ISPs

ISPs (e.g., KT, SKT) provide network access.
Hierarchical topology: local ISPs → backbone ISPs → global backbones connected via peering/transit.

Chapter 19. Host-to-Host Delivery

2.1 OSI Seven Layers (Quick Review)

Physical: signal transmission.
Data Link: MAC, error detection, hop-to-hop delivery.
Network: routing, addressing, multicast, ARP.
Transport: TCP/UDP for end-to-end reliability.
Session
Presentation
Application: HTTP, DNS, etc. (Labs typically focus on layers 4–7.)

2.2 Network Layer Roles

Internetworking: connects heterogeneous networks.
Addressing: assigns globally unique identifiers.
Routing: chooses optimal paths.
Packetizing: encapsulates upper-layer data.
Fragmenting: splits packets to respect MTU limits.

2.3 Switching

Circuit switching: dedicated path (telephone network).
Packet switching: best-effort, store-and-forward (Internet).

2.4 Datagram Service

Connectionless: each packet is routed independently.
Packets may arrive out of order; TCP reorders and handles flow control.

Chapter 19. IPv4 Addressing and Subnets

3.1 Address Classes

32-bit addresses must be unique and universally routable.
Classful ranges:
- Class A: 0xxxxxxx (0–127), NetID 7 bits, HostID 24 bits.
- Class B: 10xxxxxx (128–191), NetID 14 bits, HostID 16 bits.
- Class C: 110xxxxx (192–223), NetID 21 bits, HostID 8 bits.
- Class D: 1110xxxx (224–239), multicast.
- Class E: 1111xxxx (240–255), reserved for experiments.
Special addresses: -.0.0.0 (network), -.0.0.1 (gateway), -.255.255.255 (broadcast).

3.2 Subnetting

Partition large networks into manageable subnets to improve organisation and security.
Use bitwise AND: IP & subnet mask = subnet address.
CIDR enables flexible, classless allocation.

3.3 DHCP

Dynamically assigns IP configuration. Sequence: Discover → Offer → Request → ACK.
Mitigates address exhaustion; well-suited for mobile devices.
Combine with NAT to hide private addresses.

3.4 NAT

Translates private ↔ public addresses so many hosts share one public IP.
PAT (port address translation) differentiates sessions.
External access requires port forwarding or static public IPs.

3.5 IP Datagram

TTL decrements at each router; packet discarded at zero to avoid loops.
Fragmentation splits packets exceeding link MTU; reassembly happens at the final destination.

Chapter 20. Network Layer Protocols

4.1 ARP

Resolves IP addresses to MAC addresses within the same subnet.
Request sent via broadcast; reply delivered as unicast. Results cached.

4.2 IP

Minimum 20-byte header with version, header length, TTL, protocol, checksum, etc.
Protocol field multiplexes upper-layer protocols (TCP=6, UDP=17, OSPF=89, …).

4.3 Fragmentation & MTU

Routers split packets when downstream MTU is smaller.
DF (Don’t Fragment) flag forbids splitting, prompting ICMP errors instead.

4.4 ICMP

Reports network-layer errors and diagnostic information.
Error messages: Destination Unreachable, (deprecated) Source Quench, Time Exceeded, Parameter Problem, Redirect.
Query messages: Echo (ping), Timestamp, etc.
ICMP only reports problems—flow and congestion control live elsewhere.

4.5 IPv6 Overview

128-bit address space, simplified header, richer QoS support.
Address notation uses 16-bit hex blocks with :: compression.
Transition strategies:
1. Dual Stack (run IPv4 & IPv6 simultaneously).
2. Tunnelling (encapsulate IPv6 in IPv4).
3. Header Translation (convert headers; may lose info like flow labels).

Chapter 21. Unicast & Multicast Routing

5.1 Routing Types

Unicast: one-to-one (default Internet traffic).
Multicast: one-to-many (group members only).
Broadcast: one-to-all (limited in IPv4).
Anycast: one-to-nearest (e.g., DNS root replicas).

5.2 Overlay Networks

Logical networks built atop the physical Internet (e.g., CDNs).
Reduce load when delivering identical media streams.
New services must weigh overlay benefits against cost.

5.3 Unicast Protocols

RIP: distance vector, hop-count metric—small networks.
OSPF: link-state—large enterprise networks.
BGP: path vector—inter-AS routing on the global Internet.

5.4 Multicast Routing

Hosts join/leave groups via IGMP.
Routers build multicast trees to forward only along necessary links.
Protocols include PIM (protocol-independent multicast) variants.

Chapter 22. Security Fundamentals

6.1 HTTP Security

Basic auth: Base64 username/password—susceptible to eavesdropping.
Digest auth: adds hashing yet still vulnerable (no server authentication, MITM).
Real-world solution: HTTPS via SSL/TLS to encrypt the transport layer.

6.2 Secure Email

Requirements: confidentiality, authentication, integrity, non-repudiation.
Encryption approaches:
- Symmetric keys (fast but key distribution is hard).
- Public/private keys (PGP combines both).
Strong passphrases should be long, unpredictable, yet memorable.

6.3 Public-Key Validation

Certificate Authorities (CA): trusted third parties issue certificates binding public keys to identities. Certificates include serials, subject/issuer info, validity, signatures.
PGP Web of Trust: decentralised—users vouch for one another; keys have validity and trust scores.

6.4 Transport-Layer Security

SSL/TLS: negotiate session keys, encrypt data, authenticate endpoints. HTTPS = HTTP over TLS.
Drawback: handshake incurs public-key operations and latency.

6.5 Network-Layer Security (IPsec)

Provides tunnelling, encryption, and integrity between routers or hosts.
Protocols: ESP (encryption + authentication) and AH (authentication + integrity only).
Modes: transport (end-to-end) and tunnel (network-to-network).

6.6 VPN

Emulates a private network over the public Internet.
Comparison:
- Leased line: secure but expensive.
- Hybrid: mix of private lines and Internet.
- VPN: IPsec tunnel mode balances cost and security.

7. Summary & Study Tips

Analyse cost vs. benefit when reusing existing infrastructure versus deploying new tech.
Practise subnetting and CIDR to design addressing plans.
Compare routing protocols (RIP/OSPF/BGP) and match them to network scale.
Security spans multiple layers: application (HTTPS) → transport (TLS) → network (IPsec).
Use packet-analyser tools (Wireshark) to observe ARP, ICMP, DHCP, TLS handshakes and solidify understanding.

Algorithms

Thu, 10 Oct 2024 00:00:00 +0000

1. Overview

This note compiles the key concepts covered in an undergraduate algorithms course. Use it to refresh important ideas quickly and to understand how the individual topics are connected.

2. Sorting Algorithms

Sorting is the preparatory step for many problems. When choosing an algorithm, consider input size, data distribution, auxiliary space, and whether stability (preserving the order of equal keys) matters.

Algorithm	Average Time	Worst Time	Extra Space	Highlights
Merge Sort	O(n log n)	O(n log n)	O(n)	Stable; great for external sorting
Quick Sort	O(n log n)	O(n²)	O(log n)	Divide and conquer; pivot choice is critical
Selection Sort	O(n²)	O(n²)	O(1)	Few swaps but generally slow; unstable
Bubble Sort	O(n²)	O(n²)	O(1)	Simple; fast on nearly sorted data with early exit
Insertion Sort	O(n²)	O(n²)	O(1)	Excellent for small or nearly sorted arrays; stable
Radix Sort	O(d·n)	O(d·n)	O(n + k)	Digit-wise; bypasses comparisons
Counting/Bucket Sort	O(n + k)	O(n + k)	O(n + k)	Strong when value range is small or uniform

2.1 Merge Sort

Recursively splits the array and merges the sorted halves.
Stable because equal keys retain their relative order.
Drawbacks: needs extra memory and incurs recursion overhead.

2.2 Quick Sort

Partitions around a pivot into smaller and larger elements and recurses.
Typically the fastest in practice, but degenerates to O(n²) on sorted data unless you randomise or choose smart pivots (median-of-three, etc.).

2.3 Selection Sort

Repeatedly selects the minimum (or maximum) remaining element and swaps it into place.
Swap count is limited to O(n), but the algorithm is mainly pedagogical due to its O(n²) comparisons.

2.4 Bubble Sort

Repeatedly swaps adjacent elements that are out of order so the largest “bubbles” to the end.
Each pass places one element; adding a flag for “already sorted” speeds up nearly sorted inputs.

2.5 Insertion Sort

Inserts each element into the correct position within the already sorted prefix.
Stable and efficient for tiny datasets or partially ordered inputs.

2.6 Radix Sort

Sorts integers or fixed-length strings digit by digit: LSD (least significant digit first) or MSD (most significant digit first).
Requires a stable subroutine (e.g., counting sort) to guarantee global stability.

2.7 Counting & Bucket Sort

Counting sort achieves O(n + k) for small value range k; bucket sort achieves expected O(n) when inputs are uniformly distributed.
Both need extra memory and require advance knowledge of the key range.

3. Trees: Red-Black Trees

Red-black trees are balanced binary search trees that maintain O(log n) height by enforcing colour-based invariants.

Invariants:
(1) Every node is red or black.
(2) The root and NIL leaves are black.
(3) Red nodes have black children.
(4) Every path from a node to its descendant NIL nodes contains the same number of black nodes.
Operations:
Insert like a BST, then fix colours/rotations. Delete by replacing with a successor and resolving the “double black” case.
Why use them: Fewer rotations than AVL trees, making them practical when inserts/deletes are frequent.
Examples: std::map, std::set, Java TreeMap, Linux CFS scheduler.

4. Hash Tables

Idea: Hash a key to an array index so average lookup/insertion/deletion becomes O(1).
Collision: When multiple keys hash to the same bucket. Quality of the hash function and load factor (n/m) dictate performance.

4.1 Collision-Resolution Strategies

Separate chaining: Linked lists or dynamic arrays per bucket. Simple and flexible, but needs pointer storage.
Open addressing: Probe for the next empty slot (linear, quadratic probing, double hashing). Cache-friendly, yet suffers from clustering.
Rehashing: Enlarge the table when the load factor exceeds a threshold and reinsert all entries.

4.2 Design Tips

Aim for a hash function that distributes keys evenly, is fast to compute, and resists adversarial input.
For concurrent scenarios, look into lock-free buckets, striped locks, or partitioned tables.

5. Graph Algorithms

Graphs model networks, dependency orders, and relationships. Sparse graphs typically use adjacency lists; dense graphs favour adjacency matrices.

5.1 Topological Sort

Produces a linear order of a DAG’s prerequisites.
Implement via Kahn’s algorithm (in-degree queue) or DFS finishing times; both detect cycles when sorting fails.
Useful for course prerequisites, build systems, and task scheduling.

5.2 Bellman–Ford Algorithm

Handles graphs with negative edge weights to find single-source shortest paths.
Runs |V|−1 relaxations and checks for updates on the |V|th iteration to detect negative cycles.
Critical for routing protocols such as RIP where Dijkstra cannot be applied.

5.3 Pebble Game

Places “pebbles” on DAG nodes to study computation order and resource usage.
Helps derive lower bounds for register allocation, memory minimisation, and proof complexity.
Solving the optimal pebbling strategy is challenging beyond restricted graph classes.

5.4 Additional Highlights

DFS/BFS: Core search routines; basis for connectivity, cycle detection, and topological sort.
Strongly Connected Components: Kosaraju’s or Tarjan’s algorithms condense graphs into DAGs.
Minimum Spanning Tree: Kruskal and Prim minimise connection cost in undirected weighted graphs.

6. String Matching: Rabin–Karp

Uses a rolling hash to compare the pattern hash with substrings of the text.
Offers expected O(n + m) time; collisions trigger explicit substring comparisons.
Particularly handy for multi-pattern search after precomputing pattern hashes.

7. Study Tips

Implement algorithms and analyse both average and worst cases using your own test datasets.
Visualise algorithm steps with tools such as VisuAlgo or CS Academy.
In coding interviews or contests, inspect input bounds and memory constraints before choosing a strategy.
Suggested reading: CLRS, Sedgewick’s Algorithms, and Problem Solving with Algorithms and Data Structures by Miller & Ranum.

Data Mining

Tue, 16 Apr 2024 00:00:00 +0000

1. Overview

Data mining extracts meaningful patterns and knowledge from massive datasets.
Pipeline: data cleaning → feature construction → modelling/pattern discovery → evaluation & visualisation.
Algorithms must account for the “4 V’s” of big data: volume, variety, velocity, veracity.

2. Frequent Itemset Mining

2.1 Concepts

Identify sets of items that co-occur in transaction data.
Support: fraction of transactions containing itemset $X$
$$\mathrm{support}(X) = \frac{#\text{transactions containing }X}{#\text{all transactions}}$$
Confidence: probability that $Y$ appears when $X$ does in rule $X \rightarrow Y$.
Lift: $ \frac{P(X,Y)}{P(X)P(Y)}$ measures how much more often $X$ and $Y$ appear together than by independence.

2.2 Apriori Algorithm

Top-down search that grows larger frequent itemsets from smaller ones.
Apriori principle: if an itemset is not frequent, none of its supersets can be frequent.
Steps:
1. Generate candidate 1-itemsets and compute support.
2. Keep itemsets above the minimum support threshold.
3. Combine survivors to form $(k+1)$-item candidates.
4. Repeat until no more frequent sets remain.
Limitation: repeated data scans and huge candidate sets.

2.3 FP-Growth

Compress transactions into an FP-tree without candidate generation.
Process: order items by support → insert transactions into the tree sharing prefixes → build conditional FP-trees recursively.
Requires less I/O than Apriori and scales well to large datasets.

2.4 Association Rules

Split each frequent itemset into antecedent/consequent pairs to form rule candidates.
Filter by support, confidence, lift, etc.
Applications: recommendation, market-basket analysis, intrusion detection.

3. Similar Item Search: Locality-Sensitive Hashing (LSH)

3.1 Problem

Quickly find similar objects in very large datasets (duplicate documents, image search, record linkage).

3.2 Core Idea

Design hash families so that similar items collide with high probability.
Choose hash functions according to similarity metrics:
- Jaccard → MinHash
- Cosine → Random hyperplane
- $L_p$ distance → $p$-stable hashes

3.3 MinHash with Banding

Represent documents as sets and build MinHash signatures (minimum values of multiple hash functions).
Split the signature matrix into $b$ bands of $r$ rows; hash each band to buckets.
If two signatures match in any band, treat them as a candidate pair and compute exact similarity.
Adjust $(r, b)$ to balance false negatives (more bands) vs. false positives (more rows).

3.4 LSH Pipeline

Preprocess data into sets or vectors.
Generate LSH signatures.
Extract candidate pairs.
Compute exact distances and return top-$k$ or threshold matches.

4. Clustering

4.1 Essentials

Discover natural groupings in unlabeled data.
Objectives: maximise intra-cluster similarity and minimise inter-cluster similarity.
Distance metrics: Euclidean, cosine, Hamming, DTW, etc.

4.2 K-means

Classic algorithm for roughly spherical clusters.
Procedure: choose $K$ centres (random or k-means++), assign points to nearest centre, recompute centres, repeat until convergence.
Watch out for $K$ selection (elbow, silhouette), sensitivity to outliers, and initial seeds.

4.3 Hierarchical Clustering

Builds a dendrogram based on pairwise distances.
Agglomerative (bottom-up): start with singletons and merge clusters using linkage criteria (single, complete, average, Ward).
Divisive (top-down): start with all points and split recursively.
Cut the tree at a chosen level to obtain the desired number of clusters.

4.4 Density-Based Clustering

DBSCAN/HDBSCAN treat dense regions as clusters and sparse areas as noise.
Parameters: radius $\varepsilon$ and minimum points MinPts.
Captures arbitrarily shaped clusters and automatically flags outliers.

5. Community Detection in Graphs

5.1 Definition

Partition a graph into subgraphs with dense internal connections and sparse external ones.
Use cases: social networks, recommendation engines, biological networks.

5.2 Modularity

Measures the difference between intra-cluster and inter-cluster edge density.
Louvain/Leiden algorithms greedily maximise modularity for large graphs.

5.3 Spectral Clustering

Uses eigenvectors of the graph Laplacian to embed nodes into a low-dimensional space.
Workflow: build adjacency matrix $W$, degree matrix $D$, compute normalised Laplacian $L = D^{-1/2}(D-W)D^{-1/2}$, take the smallest $k$ eigenvectors, and run k-means.
Handles nonlinear structures and non-spherical clusters.

5.4 Comparison

Modularity optimisation is fast but subject to resolution limits.
Spectral methods are more costly (eigen decomposition) yet yield high-quality partitions.
Choose based on graph size, sparsity, and runtime constraints.

6. Dimensionality Reduction

6.1 Motivation

High-dimensional data worsens computation, visualisation, and the curse of dimensionality.
Dimensionality reduction keeps as much information as possible while compressing representation.

6.2 Principal Component Analysis (PCA)

Eigen-decompose the covariance matrix to find directions of maximal variance.
Steps: standardise data → compute covariance → eigenvalues/vectors → project onto top-$k$ components.
Linear, orthogonal transform that preserves maximum variance; sensitive to nonlinearity/outliers.

6.3 Linear Discriminant Analysis (LDA)

Maximises between-class variance while minimising within-class variance.
Supervised dimensionality reduction; boosts classification performance.

6.4 Nonlinear Techniques

t-SNE: excellent for local structure visualisation, but distorts global layout.
UMAP: faster than t-SNE with better global preservation.
Autoencoders: neural networks that learn nonlinear embeddings.

6.5 Working with LSH/Clustering

Reduce dimensionality first to improve LSH efficiency and denoise features.
Applying PCA before clustering often yields higher silhouette scores.

7. Project Tips

Prioritise data quality: handle missing values, detect outliers, and scale appropriately.
Evaluation metrics:
- Frequent patterns: support, confidence, lift, conviction, Kulczynski, etc.
- Clustering/community detection: silhouette, Davies–Bouldin, modularity.
- Dimensionality reduction: reconstruction error, downstream model accuracy.
Visualise results with heatmaps, correlation matrices, network graphs, and 2D embeddings (PCA, t-SNE).
In practice, LSH is just a candidate generator—always follow up with precise distance checks.
Document your pipeline and tuning experiments to ensure reproducibility.

Data Structures

Mon, 15 Apr 2024 00:00:00 +0000

1. Overview

Data structures organise information in memory and on disk so that algorithms can run efficiently. When solving problems, consider both the abstract data type (what operations are supported) and the concrete implementation (time/space trade-offs).

2. Arrays and Wrapper Classes

2.1 Arrays

Traits: Contiguous memory, O(1) random access by index.
Limitations: Fixed size; mid-array insert/delete costs O(n) shifts.
Use cases: Cache-friendly for numerical computing, DP tables, game state loops.

2.2 Class Wrappers

Wrapping arrays inside classes adds safety features such as bounds checking, dynamic resizing, and iterators.
std::vector (C++) and ArrayList (Java) double their capacity when full, yielding amortised O(1) append operations.

3. Stack and Queue

3.1 Stack

Last-in-first-out (LIFO). Underpins call stacks, delimiter checking, DFS backtracking.
Core ops: push, pop, top. Implementable with arrays or linked lists.

3.2 Queue

First-in-first-out (FIFO). Used in BFS, job scheduling, buffering.
Core ops: enqueue, dequeue, front. Circular arrays eliminate index wrap-around.

3.3 Deque

Inserts and deletes on both ends. Handy for sliding windows and monotonic queues.

4. Pointers and Linked Lists

4.1 Pointers

Store memory addresses and reference dynamically allocated objects.
Beware of null dereferences, dangling pointers, and double frees.

4.2 Singly Linked List

Nodes store (data, next). Inserts/deletes run in O(1); random access takes O(n).
Techniques: head insertion, tail pointer maintenance, sentinel (dummy) heads.

4.3 Doubly Linked List

Each node keeps prev and next, enabling bidirectional traversal.
Useful for LRU caches and editors. Twice as many pointer updates—test carefully.

5. List Abstract Data Type

List ADTs define ordered collections; performance depends on implementation.
Dynamic array list: O(1) indexing, O(n) mid-insert/delete.
Linked list: O(1) insert/delete (with references), O(n) search; cursor iterators help frequent traversals.

6. Recursion

Functions call themselves to break a problem into smaller cases.
Essential elements: base case, recursive relation, and state propagation.
Watch call-stack depth; tail recursion can often be transformed into iteration.

7. Trees

Hierarchical, acyclic structures branching from a root.
Terminology: depth, height, degree, subtree.
Traversals: preorder, inorder, postorder, level-order—implemented via recursion or queues.

8. Binary Search Tree (BST)

Maintains the invariant left subtree < root < right subtree.
Average O(log n) lookup/insert/delete; skewed trees degrade to O(n).
Balancing options: AVL, red-black tree, treap, splay tree—guarantee logarithmic height.

9. Priority Queue

Stores elements with priorities; retrieves the highest (or lowest) priority element.
Typically implemented with heaps: insert and extract in O(log n), peek in O(1).
Applications: Dijkstra’s algorithm, schedulers, event simulation, running medians.

10. Graphs

Comprised of vertices (V) and edges (E). Consider direction, multiedges, self-loops.
Representations:
- Adjacency list: O(V + E) space—ideal for sparse graphs.
- Adjacency matrix: O(V²) space—constant-time edge existence checks.
Traversals: DFS (stack/recursion) and BFS (queue) underpin connectivity, topological order, and cycle detection.

11. Weighted Graphs

Each edge carries a cost. Models maps, networks, logistics.
Single-source shortest paths: Dijkstra (non-negative weights), Bellman–Ford (with negatives), SPFA (practical improvements).
All-pairs shortest paths: Floyd–Warshall (O(V³)), Johnson’s algorithm (sparse graphs).
Minimum spanning tree: Kruskal (Union-Find) and Prim (priority queue).

12. Sorting (Review)

Reorders data to enable faster search or analytics.
Comparison-based: Quick, merge, heap sort—lower bound O(n log n).
Non-comparison: Counting, radix, bucket sort—use key range/digits to reach O(n + k).
Evaluate stability, in-place behaviour, memory footprint, and whether divide-and-conquer fits the scenario.

13. Searching

Linear search: Scan sequentially; works on unsorted collections; O(n).
Binary search: Halves the range on a sorted array; O(log n).
Hash lookup: Average O(1) with well-designed collisions handling.
Tree search: Balanced BSTs or B-Trees deliver O(log n); maintaining balance is vital.

14. Study Tips

Build your own complexity tables and revisit them regularly.
Step through pointer manipulations with a debugger to internalise recursion and linked structures.
Analyse input size, update frequency, and random access needs before choosing a structure.