Week 3: Deep Learning for Structured and Engineering Data
Lecture
Status: Lecture notes and slides are available.
Overview
The first half explains how neural networks use engineering measurements to make predictions and how training improves those predictions. It introduces inputs and targets, hidden layers, activation functions, loss, gradients, backpropagation, and model evaluation. Short PyTorch examples connect the concepts to code without assuming a particular format for later practical work.
The second half examines different modeling choices: representation learning and self-supervision, local and global models, foundation models, federated training, and mixture of experts. These address different questions and can sometimes be combined; they are not stages in a progression.
Materials
| Material | Where |
|---|---|
| Lecture notes | read |
| Slides | open (view in browser) |
| Lab | - |
| Papers | - |
References
[1] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. Available at https://www.deeplearningbook.org
[2] C. M. Bishop, H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. DOI: 10.1007/978-3-031-45468-4. Available at https://www.bishopbook.com
[3] P. Montero-Manso, R. J. Hyndman, “Principles and algorithms for forecasting groups of time series: Locality and globality,” International Journal of Forecasting, vol. 37, no. 4, pp. 1632 to 1653, 2021. DOI: 10.1016/j.ijforecast.2021.03.004
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, et al., “On the Opportunities and Risks of Foundation Models,” arXiv preprint, 2021. arXiv:2108.07258
[5] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, B. Aguera y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR vol. 54, pp. 1273 to 1282, 2017.
[6] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, “Adaptive Mixtures of Local Experts,” Neural Computation, vol. 3, no. 1, pp. 79 to 87, 1991. DOI: 10.1162/neco.1991.3.1.79
[7] PyTorch contributors. PyTorch tutorials and Automatic differentiation with torch.autograd. The PyTorch logo used in the slides comes from the official documentation.
[8] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” International Conference on Learning Representations (ICLR), 2014. Paper.