2019-03-13 03:03:21 8 Comments

I am looking for a book that goes through the mathematical aspects of neural networks, from simple forward passage of multilayer perceptron in matrix form or differentiation of activation functions, to back propagation in CNN or RNN (to mention some of the topics).

Do you know any book that goes in depth into this theory? I've had a look at a couple (such as Pattern Recognition and Machine Learning by Bishop) but still have not found a rigorous one (with exercises would be a plus). Do you have any suggestions?

### Related Questions

#### Sponsored Content

#### 0 Answered Questions

### What are some current best practices for function approximation using neural networks?

**2018-08-13 18:26:19****tkw954****90**View**1**Score**0**Answer- Tags: interpolation neural-networks artificial-intelligence

#### 0 Answered Questions

### Neural Networks and the Chain Rule

**2018-07-01 16:25:54****NicNic8****155**View**5**Score**0**Answer- Tags: optimization machine-learning neural-networks

#### 0 Answered Questions

### Understanding the use of convolution in neural networks.

**2018-05-11 10:11:49****Mr. President****67**View**3**Score**0**Answer- Tags: algorithms convolution machine-learning neural-networks

#### 0 Answered Questions

### Deriving diagonal approximation of Hessian in a neural network

**2018-04-15 20:57:27****ps_****211**View**1**Score**0**Answer- Tags: machine-learning neural-networks hessian-matrix

#### 1 Answered Questions

### [SOLVED] Parameterized families of functions, other than neural networks, that are useful for machine learning

**2017-05-29 20:52:45****eternalGoldenBraid****102**View**1**Score**1**Answer- Tags: machine-learning neural-networks

#### 0 Answered Questions

### delay time from mathematical perspective

**2014-09-29 20:24:04****CroCo****34**View**0**Score**0**Answer- Tags: book-recommendation

#### 0 Answered Questions

### Modern Mathematical Theory for Neural Networks, Cellular Automata, Neuroscience

**2013-08-04 12:06:50****ask****721**View**4**Score**0**Answer- Tags: machine-learning neural-networks artificial-intelligence cellular-automata

#### 0 Answered Questions

### Two-layer Perceptron for XOR

**2013-11-21 22:23:30****Wouter****1182**View**5**Score**0**Answer- Tags: computational-mathematics neural-networks

#### 2 Answered Questions

### [SOLVED] Reference Request: Neural Networks as Mathematical Models

**2014-12-24 10:55:09****Dal****246**View**3**Score**2**Answer- Tags: reference-request book-recommendation machine-learning biology neural-networks

#### 1 Answered Questions

### [SOLVED] Sources to learn and understand advanced probability in ML models

**2015-10-08 11:10:33****Celdor****673**View**2**Score**1**Answer- Tags: probability book-recommendation machine-learning bayesian-network

## 5 comments

## @Jim Stuttard 2019-03-24 11:41:20

Not a book but maybe of some interest for a current perspective:

Backprop as Functor: A compositional perspective on supervised learning Brendan Fong David I. Spivak Remy Tuyeras (2018) gives a category theoretic structural framework based on the algorithm:

https://arxiv.org/pdf/1711.10455.pdf

This is further discussed by David Spivak (2019) via:

https://www.reddit.com/r/math/comments/ahrar7/lectures_in_applied_category_theory_mit_2019/

## @pcp 2019-03-14 09:56:38

One of my favorite books on theoretical aspects of neural networks is Anthony and Bartlett's book: "Neural Network Learning Theoretical Foundations".

This book studies neural networks in the context of statistical learning theory. You will find loads of estimates of VC dimensions of sets of networks and all that fun stuff.

I should say that this book does not go into detail on CNNs and RNNs.

## @Josef Knecht 2019-03-14 19:43:35

Gilbert Strang (of MIT OCW Linear Algebra lectures and

Introduction to Linear Algebrafame) has a new textbook on linear algebra for deep learning, Linear Algebra and Learning from Data.It's got a decent course in linear algebra, some statistics & optimization, the calculus needed for stochastic gradient descent, and then applies them all to neural network models.

## @Shamisen Expert 2019-03-13 03:33:59

For MLPs, there is a rigorous derivation in the optimization textbook by Edwin Chong and Zak. Although it is notation heavy as all things related to neural networks must be.

This book is for some reason freely available online. See page 219 of https://eng.uok.ac.ir/mfathi/Courses/Advanced%20Eng%20Math/An%20Introduction%20to%20Optimization-%20E.%20Chong,%20S.%20Zak.pdf

I think there is essentially no good mathematical textbook on convolutional neural networks or RNN in existence. People essentially just base their intuition off of MLPs. But it is not hard to create a mathematically rigorous derivation of forward and backward propagation of CNN or RNN.

## @Rahul 2019-03-13 12:11:21

"This book is for some reason freely available online." That is

probablya copyright violation by the webpage owner eng.uok.ac.ir/mfathi. But I won't tell anyone if you won't ;)## @Jair Taylor 2019-03-13 03:11:25

I'd recommend Deep Learning by Goodfellow, Bengio and Courville. I don't know if I'd call it "purely mathematical", but it covers a good amount of math background in the first few chapters. No exercises, though.

## @Eli 2019-03-13 03:17:54

Thank you - I've actually had a look at that one too, but while it is good in introducing the main mathematical tools needed for NN, I found it a bit lacking when it came to properly develop the model mathematically.