Exploding and vanishing gradients in deep neural networks: the effect of residual connections

The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection is explained in this context. Specifically, a characterization of Liapunov exponents due to Furstenberg and Kifer is exploited in order to make a precise statement about the Liapunov spectrum and the effect of residual connections on it. --- Source: arXiv:2606.17013v1 - http://arxiv.org/abs/2606.17013v1 PDF: https://arx...