Posts Tagged ‘neuroscience’
Leaky Integrate-and-Fire
About one year ago I attended a conference where one professor presented the work of her team on simulating real neurons. I was curious about how body neurons can be modeled and I have taken the opportunity of a long train journey to browse a bit.
I stared my search looking for “Spiking neural network”. The most surprising thing I learned is that real neurons seems do nothing most of the time. Suddenly they can fire a spike to alert other neurons, but usually they seem idle. The frequency at which they fire is quite low, and this is probably the reason why the brain consumes so few energy. If you search on google you will quickly see that this frequency is usually less than 1kHz, usually much less. By converse, the “artificial neurons” we use in deep learning models fire continuously and synchronously at gigahertz rates: every parameter need to be evaluated at each clock signal, the output can be zero but we need to recalculate the output of each neuron in the network.
Biological neurons seems modeled as a Capacitor with a reset circuit close to it. Each time an input spike is received, the capacitor charge increases. When the charge is high enough, a spike is emitted and the charge is reset. A more realistic model, the leaky integrate-and-fire, puts also a resistance in parallel of the capacitor. It’s role is to discharge slowly the capacitor, to clear the effect of old spikes received too long ago. Input spikes are significant only if they are received in a short time interval.
On this tutorial you will see a good presentation that describes the models and the formulas: https://compneuro.neuromatch.io/tutorials/W2D3_BiologicalNeuronModels/student/W2D3_Tutorial1.html
A biological neuron is composed of dendrides, which are root like structures that receives the inputs from other neurons. The dendrides enters the body of the neuron cell, from which an axon exits: the axon is the output connector that will propagate the spike to thousand of other neurons. Axons and dendrides do not touch directly, they are connected via synapses.
The leaky integrate-and-fire model consists in this formula:
The output current of the neuron depends on 2 components: the leaky part is on the left (with the R internal resistor) and the input driven part with the C constant. u_rest is the tension at rest of the output, when we have not enough input spikes.
The presentation is suggesting some other interesting points.
The spikes are all the same: the shape is the same, in 1-2 ms the spike ends. Seems that it is just important that the spike happened, there is not a concept of high or low spike. By converse artificial neurons have an output that is a real number, maybe normalized but with variability, not just 1 or 0.
There exists more sophisticated models that can explain another phenomenon called adaptation. Suppose you present the same constant input to a neuron, with the leaky integrate-and-fire model the neuron will charge, emit a spike, recharge, emit another spike… So it will generate an output sequence with a specific frequency. Seems that nature do not likes this behavior: a constant signal do not bring much information, real neurons will emit spikes, but with a frequency that decreases. The spikes will be there but less and less frequent.
But how do biological neuron learn? with artificial neurons you have algorithms that uses the gradient and adapt the neuron’s coefficients to learn producing the correct output. I ask myself if a model exists to explain how the dendrides and axon change in response to all these spikes. Maybe a good topic to explore in another train journey.
World AI Cannes Festival 2024
The exhibit has been held between February 8th and February 10th in Cannes, French Riviera
While 2023 edition was highly democratized, this time the organization decided to allow access to the interesting speeches only to paying tickets. The company I work for has also bought less ticket so, sadly, I did not profit as much as I wanted of this event. The organization also changed the mobile app they had the last year: in 2023 was so easy to find what was going on across the exhibit, and this time it has become a mess: check each single room to see what was planned… I hope in 2025 they will change their minds.
One interesting speech I saw was “NeuroAI: Advancing Artificial Intelligence through Brain-Inspired Innovations”. The speakers described us some interesting research paths about building more efficient AI hardware. The human brain consumes a ridiculous amount of energy, comparable to a candle light, by converse the modern hardware is really energy-intensive, requiring hundreds of watts to operate. To make the things worst, the current trend is just to build bigger models using more hardware and more training data. It will be impossible to keep this trend, as the energy demanded will quickly become incredibly huge: 85 TWh by 2027 according to this scientific american article
More efficient hardware could probably mimic what the brain is doing: natural neurons fires at some incredibly low rate, just 30Hz. Neurons receive signals, and aggregate them producing an output only once a certain threshold is reached. The output spike does not last for long time and does not consumes that much energy.
It is possible to use some transistors to implement a similar behavior, this could become an unit that can be cloned million of times realizing in hardware an huge array of neurons. Training these devices wold then mean let them learn the correct thresholds and input connections. By converse current architectures require to continously compute values for each neuron output even if the output is not required.
One more energy-intensive task is moving the data around: current architectures have specialized memory units and computation units. The required data has to be moved in and out from the memory to the GPU and back. This requires a lot of energy.
IBM did some experiments, and seems using this neuromorphic patter gives some advantages over traditional LSTM architecture. You can visit Stanislaw Wozniak page to find some references.
I then followed some few other presentations, and some commercial demos. AMD’s one about realtime video processing was both appealing and frightening. You can buy now devices that can process video input streams at up of 90 frames per seconds. You can buy a software stack that allows you to view this camera flow and decide that on some video areas you want to perform some actions: for instance you put a camera watching a car access gate and decide that you use a region of the view to search for a car registration number. You could stack these operation on the remote device or turn you attention on a central system that now just receives events such has the car xx123yy has passed here at 12:05.
The guy setting up this stack of operation does not need to be a rocket scientist, a nice ui can help him setting up his/her video driven application. Moreover you could react to events and change the pipeline of some other cameras to perform other operations. In Singapore parkings facilities they are using this to detect free parkings and guide VIP customers to theirs places… or maybe to avoid non vip customers to pick reserved slots. So far so good, but what does prevent some malicious entity from having cams looking for a specif guy, and being reprogrammed to detect his activity while moving around a building? Do we have to fear that a cam at work place starts detecting how many times we go to the toilets? Unfortunately it does not seems anymore sci-fi… Field Proven Smart City Video Analytics Solution
Giovanni Bricconi 
