Two hidden layer suffice: the caterpillar net

Every continuous piecewise linear (CPWL) function is a finite signed sum of maxima of affine functions, making maximum representation a central problem in the study of the expressiveness of ReLU networks. A now-disproved conjecture predicted logarithmic depth for maxima. Inspired by the proof by Brunck et al. (2026) that two hidden layers suffice for general CPWL functions, we give the Caterpillar Net, a nonrecursive construction for the maximum of $N$ inputs with two hidden layers of widths $O(N^2)$ and $2^{O(N\log N)}$. The construction has a direct geometric interpretation.

Source: arXiv:2610.26795v1 - http://arxiv.org/abs/2610.26795v1 PDF: https://arxiv.org/pdf/2610.26795v1 Original Link: http://arxiv.org/abs/2610.26795v1