Deep Learning

Deep learning studies neural networks as trainable function approximators: layers define the computation, losses define the target, gradients move parameters, and architecture choices determine what structure is easy to learn. Read this section as a mechanism-first path rather than a catalogue.

Knowledge map

Fundamentals and training mechanics come first because every architecture reuses them; architectures then enable representation learning, generative models, and the engineering needed to train at scale.

flowchart TD
  NN[Neural Network Fundamentals] --> BP[Backpropagation]
  BP --> Train[Optimizers, Init, Normalization, Regularization]
  Train --> Arch[Architectures: CNN, RNN, LSTM, Attention, Transformers]
  Arch --> Repr[Representation and Self-Supervised Learning]
  Repr --> Gen[Generative Models]
  Train --> Eng[Engineering: Frameworks, Mixed Precision, Distributed]

Reading path

Read fundamentals and training mechanics, then architectures, representation learning, generative models, and engineering.

1. Neural Network Fundamentals: how affine layers, nonlinearities, losses, and optimizers combine into a trainable model.
2. Multilayer Perceptrons: dense feed-forward networks that transform features without recurrent state.
3. Backpropagation: reverse-mode chain-rule differentiation through a computational graph.
4. Vanishing and Exploding Gradients: why deep chain-rule products can shrink or blow up.
5. Activation Functions: nonlinearities that control expressiveness and gradient flow.
6. Loss Functions: differentiable objectives for regression and classification.
7. Optimizers: SGD, momentum, and Adam-style rules that turn gradients into updates.
8. Initialization: starting weight scales that keep activations and gradients usable.
9. Normalization: batch and layer standardization with trainable affine recovery.
10. Regularization: dropout, weight penalties, and other ways to reduce memorization.
11. Residual Connections: skip paths that let deep blocks learn corrections.
12. Convolutional Neural Networks: shared local filters for grids and images.
13. Recurrent Neural Networks: stateful sequence models with shared temporal transitions.
14. LSTM and GRU: gated recurrent cells for longer-range credit assignment.
15. Attention: content-based weighted routing between positions or modalities.
16. Transformers: attention, residual, normalization, and feed-forward blocks for parallel sequence modeling.
17. Representation Learning: learned feature spaces for prediction, retrieval, and transfer.
18. Autoencoders: encoder-decoder models that learn latent codes by reconstruction.
19. Self-Supervised Learning: pretext objectives generated from unlabeled data.
20. Contrastive Learning: embedding objectives that separate positives from negatives.
21. Transfer Learning: reusing pretrained features on a new task.
22. Fine-Tuning: selectively updating pretrained parameters or adapters.
23. Multimodal Learning: aligning and fusing text, image, audio, and video.
24. Generative Adversarial Networks: generator-discriminator games for sharp implicit generation.
25. PyTorch: dynamic-tape tensor programming and explicit training loops.
26. TensorFlow and Keras: high-level model APIs and production workflows.
27. Mixed Precision: lower-precision arithmetic with scaling and FP32 safeguards.
28. Distributed Training: synchronized or partitioned training across devices and machines.

Connections

• Mathematical Foundations supplies the gradients and linear algebra behind training.
• Natural Language Processing, Computer Vision, and Generative AI specialize these architectures to their modalities.

Nav

Learning path — Deep learning

Neural Network Fundamentals →