Assistant Professor, ECE, CS, Neuroscience at UIUC | Theoretical neuroscience, neural dynamics, spiking networks | Recruiting PhDs & Postdocs
Joined June 2020
- Tweets58
- Following944
- Followers612
- Likes1K
Rainer Engelken retweeted
What do control theory and neuroscience have in common? Shared origins in cybernetics — and, lately, “poor cousin” status next to ML. To reconnect them, we’ve been organizing workshops at this interface, most recently at the Flatiron Institute. Thanks to all who joined!
Belated update: I joined UIUC ECE as an Assistant Professor. Our lab works at the intersection of theoretical neuroscience, machine learning, and dynamical systems, with a focus on learning and spiking networks. I had to miss #COSYNE2026 for visa reasons.
ALT Poincare section of chaotic spiking network. Colors indicate local Lyapunov exponents.
We are recruiting PhD students and postdocs at UIUC ECE interested in learning dynamics, spiking networks, and dynamical systems. Details: rainerengelken.github.io/joi…
Rainer Engelken retweeted
The #SNUFA24 final program is out and the event is next Tue-Wed! If you love spiking neural networks, click on the link below to check it out and register (free).
snufa.net/2024/
I thank S. Goedeke, J. Liedtke, R.-M. Memmesheimer, M. Monteforte, A. Palmigiano, M. Puelma Touzel, A. Schmidt, M. Schottdorf, and F. Wolf for fruitful discussions. Moreover, I thank L.F. Abbott, S. Migirditch, A. Palmigiano, Y. Park, and D. Sussillo for comments on a draft.
12/n
SparseProp doesn't work (yet) for stochastic input most multivariate neuron models. Advantageous scaling of O(log(N)) only for sparse networks (Fixed number of synapses per neuron). A more fundamental limitations: it is not clear how to calculate surrogate gradients.
11/n
Excited to present this afternoon another #NeurIPS paper on #SparseProp, which is a novel event-based algorithm to simulate and train spiking neural networks, reducing computational cost from N to log(N) per network spike for sparse spiking networks:
openreview.net/forum?id=yzZb…
1/n
I gave a talk about some of the core idea of SparseProp at #JuliaCon some years ago, but this was an earlier version of Julia and didn't include heterogeneous networks, Chebychev polynomials and network training. youtube.com/watch?v=cLLQcshW…
10/n
Example code for networks of QIF, LIF and EIF neurons in julia for SparseProp is available at github.com/RainerEngelken/Sp…
#JuliaLang @JuliaLanguage
9/n
SparseProp can also be used for event-based training of spiking networks, the backward pass has the same complexity as the forward pass. Caveat: Many people use surrogate gradients and it is not clear to me how to do that in event-based simulations. Any ideas? @hisspikeness
8/n
For neurons where the next spike time can't be obtained analytically, e.g. exponential integrate-and-fire, SparseProp is still an option. Instead of the analytical phase response curve (PRC), the numerical PRC can be approximated precisely using Chebychev polynomials.
7/n
SparseProp also works for heterogeneous networks where different neurons have, for instance, different external inputs or different membrane time constants. In that case, the binary heap is based on the time to the next spike.
6/n
The two tricks: First, go to a moving reference frame, where only threshold and reset are updated at every spike instead of every N neuron states. Second: Use a binary heap to find the next neuron that spikes next.
5/n
For instance, conventionally, running a network of a million neurons for 100 seconds simulation time would take more than 3 years. Using the tricks, I was able to reduce this to less than one hour.
4/n
However, conventional event-based simulation scale with O(N) per network spike, which makes them tedious for large nets. SparseProp uses two tricks to reduce this computational cost from O(N) to O(log(N)) per network spike in sparse networks.
3/n
In event-based simulations of spiking networks, the network dynamics is solved analytically between network spikes. This allows numerically exact simulations, which is convenient, whenever you care about accuracy or if you want to avoid checking different step size Δt.
2/n
I am thrilled to present tomorrow my #NeurIPS paper on #GradientFlossing, which is a novel approach to tackling the exploding/vanishing gradient problem based on concepts from dynamical systems theory (Lyapunov exponents):
openreview.net/forum?id=jEQR…
1/n
Also, a shoutout to related work by @MannyDePresso , @memming ,@mptouzel, @SuryaGanguli , @poolio and Minmin Chen. I also thank J. Stone, L.F. Abbott, M. Ding, O. Marshall, S. Goedeke, S. Lippl, M.P. Puelma-Touzel, J. Lindner, and reviewers for feedback on the manuscript.
14/n
I'd be grateful for feedback and suggestions on which other dynamical systems to apply gradient flossing to. Poster #720 tomorrow morning. 15/15