The Vapnik-Chervonenkis Dimensions of Different Neural Network Architectures

The Vapnik-Chervonenkis dimension, VC dimension in short, is a measure of expressivity or
richness of a set of functions. In this thesis, we explore this concept in relation to different neural
network architectures that use sigmoid activation functions. More specifically, we will take a
look at classical multilayered feed-forward neural networks and at two NeuralODE architectures,
namely Liquid Time Constant (LTC) networks and Continuous-Time Recurrent Neural Networks
(CT-RNNs). In the latter two, the output of the network is computed by numerically solving an
ordinary differential equation.
For these networks, we derived upper bounds on the VC dimension, depending on the number
of neurons, and in case of the recurrent models (LTC and CT-RNN), discretization steps. This
was done through a method involving the number of components of the zero-set of functions
that are dependent on the network parameters. Here various techniques relating to topology and
geometrical analysis were used. We find a very strong dependence of the VC dimension bound
on the number of neurons and a sizeable dependence on the number of discretization steps. The
recurrent models had a higher bound than the classical network for the same number of neurons,
which is partly due to the recurrent models having more parameters than the classical network.