Deep Learning | Portfolio

📁 Chapter 2: Feedforward Neural Network (FNN) Summary

Wed, 22 Oct 2025 00:00:00 +0000

1. Overview of Feedforward Neural Networks (FNN)

FNN = Feedforward Neural Network (also called Fully Connected Neural Network).
Most FNNs adopt a fully connected architecture, so the two terms are often used interchangeably.
It is another name for the Multilayer Perceptron (MLP), the most fundamental form of an artificial neural network.
Universal Approximation Theorem:
With a sufficient number of hidden neurons and an appropriate activation function, an FNN can approximate any n-dimensional continuous function to an arbitrary degree of accuracy.

2. Structure and Design of FNN

An FNN has unidirectional data flow — signals move only forward.
Each neuron performs weighted summation → activation function.
Importantly, it assumes independence among data points (no recurrent connections).

2.1 Layer Structure

Input layer: receives external data
Hidden layer(s): extracts features
Output layer: produces predictions or classifications

Usually there is one input and one output layer;
the number of hidden layers varies depending on task complexity.

2.2 Fully Connected (FC) Layers

Every neuron in one layer is connected to every neuron in the previous layer.
Even with the same input, different neurons extract different features — more neurons allow richer representation.
Weighted summation: assigns greater weights to important features.
$$ z = \sum_i w_i x_i + b $$
Activation function: introduces nonlinearity, essential for learning complex patterns (e.g., XOR).

2.3 ReLU (Rectified Linear Unit)

$$x > 0 \Rightarrow x,\quad x \le 0 \Rightarrow 0$$
Computationally efficient and stabilizes training.
Neural networks must be differentiable (or piecewise differentiable) to allow backpropagation.

2.4 Universal Approximation and Network Depth

Even a 2-layer FNN can approximate continuous functions with enough hidden neurons.
However, increasing neurons arbitrarily leads to overfitting →
stacking deeper layers can achieve the same performance with fewer neurons.

2.5 Design Checklist

Input shape
Output shape / activation function
Hidden-layer activation function
Network size (depth & width)

3. Classification vs. Regression

Classification: predicts a category (class).
Regression: predicts a continuous value.

Classification can be viewed as a discriminant function (decision boundary) or a probabilistic model (predicting class probabilities).

4. Binary Classification Model

Binary classification models predict a Bernoulli distribution.

Output activation: Sigmoid
$$\sigma(z) = \frac{1}{1 + e^{-z}}$$
Loss function: Binary Cross-Entropy, which pairs naturally with the sigmoid output.

5. Multi-class Classification Model

Multi-class classification predicts a categorical (multinomial) distribution.

Output activation: Softmax
$\mathrm{softmax}(z)_k = \frac{e^{z_k}}{\sum_j e^{z_j}}$
Loss function: Categorical Cross-Entropy

Note: Binary classification can also be expressed as a 2-dimensional softmax,
but it’s typically simplified to a 1-dimensional sigmoid model.

6. Regression Model

Regression often assumes observation noise follows a Gaussian distribution.
The model usually predicts the mean ( \mu = y(x) ) only (variance is treated as constant).
Output activation: Identity function.

7. Input Layer

If the input is an n-dimensional vector, the input layer should have n neurons.
Example: a 28×28 image is flattened into a 784-dimensional input vector.

8. Summary of Activation Functions

8.1 Sigmoid Family

Pros: differentiable everywhere, squashes values into [0, 1].
Cons: exponential computation cost, gradient saturation/vanishing,
outputs are positive → optimization path may oscillate.

8.2 Hyperbolic Tangent (tanh)

Range [-1, 1], reducing oscillation in optimization.
Still suffers from exponential computation and saturation problems.

8.3 ReLU Family

ReLU: fast, robust to saturation/vanishing.
Leaky ReLU / PReLU: assigns small slope in negative region to mitigate “dead ReLU.”
ELU: exponential negative region — stable even for large negative inputs.
Maxout: neuron output = max of several linear functions → activation is learned.
Swish (SiLU): (x \cdot \sigma(x)); modern nonlinearity with strong empirical performance.

9. Network Size (Width & Depth)

Width: number of neurons per layer
Depth: number of layers
Search strategies: Random Search is generally more efficient than exhaustive Grid Search.
Modern approaches include Neural Architecture Search (NAS) for automated structure optimization.

9.1 Scaling Strategy

As demonstrated by EfficientNet, jointly scaling width, depth, and input resolution —
called Compound Scaling — yields more efficient performance improvements.

Download PDF

📁 Chapter 3: Neural Network Learning and Optimization

Wed, 22 Oct 2025 00:00:00 +0000

Table of Contents

1. Overview of Neural Network Learning

A neural network initially has no idea what output to produce for a given input.
By repeatedly feeding training data, it gradually learns the mapping rule between input and output.
“Learning” means discovering this mapping rule.
Each neuron is composed of weighted summation and an activation function.
Stacking layers forms the overall neural network structure, and these structural choices are fixed before training.

2. Model Parameters vs. Hyperparameters

Model parameters: weights and biases.
These change continuously during training depending on data and are saved as the trained model.
Hyperparameters: learning rate, optimizer, batch size, network depth, dropout rate, etc.
These are set manually and do not change automatically through training.
If an activation function (e.g., PReLU) contains a learnable coefficient, that coefficient counts as a model parameter.

For example, in a house price prediction model, the mapping
(number of rooms, area, house type, distance to station → price)
is captured by the model parameters. Once training is complete, the model can perform inference based on those learned rules.

3. Setting Up the Optimization Problem

Learning is redefined as minimizing or maximizing an objective function.
Standard formulation:

$$ \begin{aligned} &\text{minimize} && f(\mathbf{x}) \ &\text{subject to} && g_i(\mathbf{x}) \le 0,\quad i=1,\dots,m \ &&& h_j(\mathbf{x}) = 0,\quad j=1,\dots,p \end{aligned} $$

( f(\mathbf{x}) ): objective (loss) function
( g_i, h_j ): constraint functions
Getting close to the optimal solution is called convergence. Once sufficiently close, the model is said to have converged.

4. Loss Functions for Regression and Classification

Regression Problems:
Find parameters ( \theta ) that minimize the difference between target ( t ) and prediction ( y = f_\theta(x) ).
Representative loss = Mean Squared Error (MSE).

$$ \begin{aligned} &\underset{\theta}{\text{minimize}} && \frac{1}{N} \sum_{i=1}^{N} (f(\mathbf{x}_i; \theta) - t_i)^2 \ &\text{subject to} \ &&& \text{No constraints} \end{aligned} $$
Classification Problems:
Find parameters ( \theta ) that minimize the difference between the observed distribution ( t ) and predicted distribution ( y ).
Representative loss = Cross-Entropy (CE).

$$ \begin{aligned} &\underset{\theta}{\text{minimize}} && - \frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} t_{ik} \log \hat{y}_{ik} \ &\text{subject to} \ &&& \text{No constraints} \end{aligned} $$

Once the loss function is defined, parameters are updated iteratively in the direction that minimizes the loss.

5. Core Idea of Gradient Descent

The ideal case is reaching the global minimum, but real loss surfaces are complex → in practice, we settle for a good local minimum.
Gradient descent updates parameters using the gradient of the loss function ( L(\theta) ):

$$ \theta^{(t+1)} = \theta^{(t)} - \alpha \nabla_\theta L(\theta^{(t)}) $$
( \alpha ): learning rate (step size)
( -\nabla_\theta L ): steepest descent direction
Since neural networks are composite functions, gradients for each parameter are computed via the chain rule (backpropagation).

6. Families of Optimization Algorithms

First-order methods: Gradient Descent, SGD, Momentum, AdaGrad, RMSProp, Adam, etc.
Stable even when the loss surface is non-convex.
Quasi-second-order methods: Quasi-Newton, Conjugate Gradient, Levenberg–Marquardt.
Approximate curvature information for faster convergence but require more memory.
Second-order methods: Newton’s method, interior-point methods.
Directly use curvature, fast but computationally heavy and suited to convex problems — impractical for large neural networks.

7. Routine of Gradient Descent

Perform forward propagation to compute predictions.
Evaluate the loss function.
Compute partial derivatives (gradients) of the loss w.r.t. each parameter (backpropagation).
Update weights and biases using the update rule.
Stop training once the loss change is below a certain threshold (convergence).

When parameters are fixed after convergence, the model encodes the mapping function that determines outputs for any given input.

8. Summary at a Glance

Neural network learning = discovering functional rules from data.
Regression usually uses MSE; classification uses Cross-Entropy.
Optimization = reformulating learning as a loss minimization problem and solving it numerically.
Gradient descent is the fundamental yet powerful first-order optimization method.
Backpropagation is essential for applying gradient descent to neural networks.

Download PDF