Introduction to the mathematics of neural networks
September 18, 2019, 1:30 to 3:00 PM. B-1508
Introduction to neural networks. Weights, biases, activation function, loss function.
Matrix viewpoint: vectors and affine transformations.
Optimization, stochastic gradient descent.
Derivation of the backpropagation equations.
A video and the slides of the presentation.
Note: Unfortunately, the slides do not appear in the video, so interested viewers are encouraged to consult them alongside the recording.
Universality of neural networks: the Cybenko and Hornik theorems
October 2, 2019, 1:30 to 3:00 PM. B-1510
Universality: theorems of Cybenko, Hornik
Hahn-Banach theorems, Riesz representation.
Elements of measure theory.
Proof of Cybenko's theorem for sigmoids on spaces of continuous functions.
A video.
A proof that bounded sigmoidal functions are discriminatory.
The role of depth in the expressive power of neural networks
October 16, 2019, 1:30 to 3:00 PM. B-3420.
Revisiting universality for neural networks.
Heuristics on the density of neural networks in the space of bounded functions.
Expressive power: depth vs. width.
Theorems of Telgarsky, Eldan-Shamir.
The video.
Introduction to geometric deep learning
November 6, 2019, 1:30 to 3:00 PM. B-1510.
Image classification and convolutional networks.
Preliminaries: linear algebra, Fourier theory, convolution, and spectral theory of the Laplacian.
Spectral theory on undirected graphs.
SGCNN: Spectral Graph Convolutional Networks.
The video.
Spectral dimensionality reduction
November 20, 2019, 1:30 to 3:00 PM. B-1510.
The manifold hypothesis.
Riemannian manifolds and Euclidean embeddings.
Variational characterization of eigenvalues of symmetric operators.
Principal component analysis.
Belkin-Niyogi algorithms for spectral dimensionality reduction.
The video and the Belkin-Niyogi paper.
Winter/Spring 2020
Initial hyperparameter distribution and training of deep neural networks.
To be determined.
Norm propagation and failure modes of learning.
Random deep networks.
Elements of probability theory: measures, Markov chains, martingales, Doob's convergence theorem.
Initialization of random networks, hyperparameter variance: Hanin's theorems.
Regularization and optimization of neural networks
To be determined.
Activation functions: sigmoids, cross-entropy, and softmax.
Overfitting and regularization techniques: weight decay, L2 and L1, dropout.
Hessian descent, dynamic gradient descent.
Back to the home page.