Research to help provide mathematical basis for designing dynamic neural networks

Neural networks, or mathematical models of massively parallel nonlinear systems inspired by biological neural networks, form the backbone of everything from ridesharing apps to suggested products on Amazon. While researchers know that neural networks are often successful in detecting patterns and trends in ever-changing data sets, they don’t perfectly understand why.

Researchers at the University of Illinois Urbana-Champaign and Massachusetts Institute of Technology are seeking to develop new mathematical models that will provide new foundations for analyzing and designing neural networks by conceptualizing their architectures as continuous dynamical systems. The team also aims to help designers better understand the tradeoffs between expressive power – a neural net’s ability to approximate complicated functions of their inputs – and complexity.

“We hope to take the guesswork out of building and training neural network systems,” said Maxim Raginsky, an associate professor of electrical and computer engineering and researcher in UIUC’s Coordinated Science Lab. Raginsky is the project’s principal investigator. “This research will push the boundaries of the theory and practice of deep learning, building on interactions between electrical engineering, mathematics, statistics, and theoretical computer science.”

This is no small task, as the model will need to account for millions of parameters and heavy computational demands. Researchers will do this by formalizing the structure of networks on a set of two axes, where the horizontal axis represents “depth” as a continuum of layers and the vertical axis represents “width” as a continuum of neurons in each layer.

The project consists of three primary research thrusts. The first will focus on continuous models of neural dynamical systems, which will leverage depth as a continuous parameter akin to time in differential equations. The second will look at discretization schemes, which can help researchers better understand fundamental tradeoffs between finite depth and width to provide guidance for the design of dynamic neural networks. And, finally, the team will work to develop algorithms for designing and training neural nets that will capitalize on the findings of the first two theoretical thrusts. The overall goal of the project is to provide principled explanations of why deep neural nets can generalize and perform well even without being explicitly programmed to do so.

The project, “Analysis and Geometry of Neural Dynamical Systems,” is funded by the National Science Foundation at $417,461 for three years. Raginsky’s co-principal investigator is CSL’s Mohamed-Ali Belabbas, an associate professor of electrical and computer engineering at UIUC. This is a collaborative project with MIT, led by Philippe Rigollet, a professor of mathematics.

“We expect this project to be radical in its approach to conceptualizing deep neural nets as continuous dynamical systems,” Raginsky said. “Continuous abstractions like this have been the key to success in many disciplines throughout the twentieth century, including probability, optimization, and control, and we hope to see our work play the same transformative role for understanding and designing machine learning algorithms in the context of deep neural nets.”