Neural Architectures Decoded: FFNN, RNN, and Transformers

Deterministic layers

Feedforward networks — stacked linear transforms with nonlinear hinges

Imagine stacking transparent sheets on a projector. Each sheet applies a linear transform (rotate, scale) followed by a squishing function (nonlinearity). After enough sheets, the projection can draw almost any boundary in the input space. Backpropagation finds the optimal sheet orientations. The universal approximation theorem guarantees this works in principle; careful regularization makes it work in practice.

$$ h^{(l)} = \sigma!\left(W^{(l)},h^{(l-1)} + b^{(l)}\right) $$

Layer l applies weights W, shifts with bi, then squishes through σ (ReLU keeps positives, zeros negatives). Animate )} h^{(L-1)} + b^{(L)}\right) $$

Final layer converts raw logit scores into a probability distribution over classes — animate ). Deep networks extrapolate (more layers compose higher-order abstractions). ResNet skip connections let you go very deep without vanishing gradients.

• Regularization toolkit Stability: Dropout (ensemble of thinned networks), weight decay (L2 Bayesian prior), batch normalization (normalizes internal activations). Combine based on dataset size: small data → heavy regularization.
• Decision boundary animation Visual: Add layers one by one to a 2D scatter plot — watch the linear decision boundary curve into complex manifolds. This is the best single visualization for explaining universal approximation to newcomers.

Temporal memory

Recurrent networks — a hidden state that reads the past

The RNN insight: feed the network its own previous output. This creates a rolling memory vector that compresses all prior context into a fixed-size representation. The downside: gradients traveling backward through T steps multiply a weight matrix T times, causing them to vanish (shrink to zero) or explode (grow unboundedly). LSTM and GRU solve this with differentiable gates — learned switches that control what to remember, forget, and output. Visualize the hidden state ,x_t + W_{hh},h_{t-1} + b_h\right) $$

Vanilla RNN: new hidden state blends current input and previous state through tanh (−1 to +1). Animate h_t ); if > 1 it → ∞ (exploding). Heatmap each factor across time to show where gradients collapse.

• LSTM gates Gated: Input, forget, and output gates provide differentiable memory control; they learn what to retain vs. erase.

• GRU trade-off Lean: Fewer gates, faster inference, slightly less expressive but easier to tune.

• Exploding gradients → clip at 1.0 or switch to gated cells.

• Teacher forcing shortens convergence but hides scheduled sampling debt.

• Profile sequential latency; batching timesteps amortizes framework overhead.

Parallel sequence understanding

Transformers — every token votes on every other token

The Transformer breakthrough: replace sequential state with a single operation that lets any two tokens interact directly, regardless of distance. Each token generates three vectors — a Query (what am I looking for?), a Key (what do I offer?), and a Value (what information to pass?). The attention score between two tokens is their query-key dot product, scaled and softmaxed. The result is fully parallelizable during training and scales predictably with compute — which is why every frontier model today is Transformer-based.

$$ \text{Attn}(Q,K,V) = \text{softmax}!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V $$

Scale by √d_k to prevent softmax from saturating in high dimensions. Animate , columns = key tokens, cell brightness = attention weight.

$$ \text{FFN}(x) = \text{GELU}(xW_1 + b_1),W_2 + b_2 $$

Per-token feedforward block (applied identically to each position) keeps local nonlinear capacity. Typically 4× wider than d_model.

$$ \text{PE}(pos, 2i) = \sin!\left(\frac{pos}{10000^{2i/d}}\right) $$

Sinusoidal positional encoding injects sequence order without learned parameters. Different frequencies encode short- vs. long-range position — animate , each in a d_k = d_model/H subspace. Different heads specialize — some capture syntactic dependencies, others semantic ones. Animate each head , reuse previously computed Key and Value vectors. Without the cache, each token step recomputes the entire sequence — latency scales ). With the cache, each step is O(n).

• Chinchilla scaling law Scaling: Compute-optimal training: N parameters require ~20N training tokens. Smaller, well-trained models consistently outperform larger undertrained ones. Plot ): fixed-input → FFNN, sequential/streaming → RNN, relational/long-context → Transformer. Map your problem to exactly one branch, then optimize within it.

• FFNN Static: Best for tabular or fixed-size signals with limited context. Simple, deterministic, cheap.

• RNN Sequential: Streaming or small-sequence problems needing temporal awareness without huge hardware.

• Transformer Context-rich: Large context, transfer learning, multimodal modeling. Heavy but unmatched flexibility.

Metric	Value	Notes
Latency target	< 10 ms	Prefer FFNN/GRU or quantized attention chunks.
Context window	64 → 128k tokens	Transformers shine ).

---

Related posts:

• llm-finetuning-and-moe
• federated-learning-deep-dive
• devops-mlops-fundamentals