Note: Another message to the Skeptics Guide to the Universe that's disappeared into a black hole. This is a about a news item about a new fast neural net experiment, and my reply in order to bring some reality to the AI hype.
---
Just finished the latest great episode and wanted to give some context and insight into the fast simulation of neural networks featured in Science or Fiction (item 1).
The technique isn't new, it's based on prior work back when computers were even slower. It's essentially a way to move some data generating/processing from the CPU into the functional GPU space because GPUs are designed to blast through things like rastrums really fast with anonymous functions really really fast, so if you can move your data into procedural buffers directly in the GPU you have a big speed improvement on your hand. To understand why there's some basic software development concepts we need to dig into;
A typical (and often the first way people learn) way of doing software development is what's known as imperative programming where you as a developer prescribe the logical flow of your program. Most software, including traditional simulation software, are being done this way to various degrees so that we can do complex network operations and have more code reuse (and this is important), but there's always room for improvement. It's essentially a program that does this, then than, then if this, then that, otherwise that, then this, then do that, then do this, otherwise do this other.
A method that's faster and less error prone is known as functional programming where the logic of the program is described through passing functions around instead of the data itself. It's in many ways better, but much harder to wrap your head around. Most pieces of software is a mix of the two (and more) methodologies, but the more you can muster functionally (especially if you design your passable functions well), the better (in my opinion). Instead of pushing large amounts of data around with boolean logic and human intuition, a functional model of processing is using a mathematical language to operate over complex data, which is far superior for optimisations. We no longer say "do this, then that, otherwise this", you describe a formula that's compatible with your data model so you don't have to check that you're within bounds; your formula will only operate within bounds.
The news item is a mix of a) going from imperative to functional logic, and b) a JIT* way creating synaptic weights instead of pushing that data around. Essentially you spend a lot less time pushing lots of data around for processing by pushing small and efficient functions around instead.
The neural net part of the item is kinda less important. First, neural net simulations are only as good or interesting as your understanding of the neurals and synapses you're trying to simulate**. This particular simulation is based on 1 specific model using GeNN (model available upon request), so its application to other models or even simulations is not really know, but you can assume it will be (as the procedural technique is for everything, only the synaptic weights are neural net simulation specific).
So what we can say now is that if you have a modern NVIDIA GPU (so, not all GPUs, not all brands), you can simulate this model much faster. It's a good progressive step forward, but as a research project still quite limited in what it can do.
Hope any of that were helpful and made some sense.
Cheers,
Alex
* JIT - Just-in-time; compiling smaller pieces of code just before it needs to be run, instead of the more traditional "compile everything before you even run the program."
** https://sheltered-objections.blogspot.com/2021/02/i-have-been-listening-regularly-for-few.html
---
Just finished the latest great episode and wanted to give some context and insight into the fast simulation of neural networks featured in Science or Fiction (item 1).
The technique isn't new, it's based on prior work back when computers were even slower. It's essentially a way to move some data generating/processing from the CPU into the functional GPU space because GPUs are designed to blast through things like rastrums really fast with anonymous functions really really fast, so if you can move your data into procedural buffers directly in the GPU you have a big speed improvement on your hand. To understand why there's some basic software development concepts we need to dig into;
A typical (and often the first way people learn) way of doing software development is what's known as imperative programming where you as a developer prescribe the logical flow of your program. Most software, including traditional simulation software, are being done this way to various degrees so that we can do complex network operations and have more code reuse (and this is important), but there's always room for improvement. It's essentially a program that does this, then than, then if this, then that, otherwise that, then this, then do that, then do this, otherwise do this other.
A method that's faster and less error prone is known as functional programming where the logic of the program is described through passing functions around instead of the data itself. It's in many ways better, but much harder to wrap your head around. Most pieces of software is a mix of the two (and more) methodologies, but the more you can muster functionally (especially if you design your passable functions well), the better (in my opinion). Instead of pushing large amounts of data around with boolean logic and human intuition, a functional model of processing is using a mathematical language to operate over complex data, which is far superior for optimisations. We no longer say "do this, then that, otherwise this", you describe a formula that's compatible with your data model so you don't have to check that you're within bounds; your formula will only operate within bounds.
The news item is a mix of a) going from imperative to functional logic, and b) a JIT* way creating synaptic weights instead of pushing that data around. Essentially you spend a lot less time pushing lots of data around for processing by pushing small and efficient functions around instead.
The neural net part of the item is kinda less important. First, neural net simulations are only as good or interesting as your understanding of the neurals and synapses you're trying to simulate**. This particular simulation is based on 1 specific model using GeNN (model available upon request), so its application to other models or even simulations is not really know, but you can assume it will be (as the procedural technique is for everything, only the synaptic weights are neural net simulation specific).
So what we can say now is that if you have a modern NVIDIA GPU (so, not all GPUs, not all brands), you can simulate this model much faster. It's a good progressive step forward, but as a research project still quite limited in what it can do.
Hope any of that were helpful and made some sense.
Cheers,
Alex
* JIT - Just-in-time; compiling smaller pieces of code just before it needs to be run, instead of the more traditional "compile everything before you even run the program."
** https://sheltered-objections.blogspot.com/2021/02/i-have-been-listening-regularly-for-few.html
Comments
Post a Comment
All are allowed to comment here, I don't discriminate against anyone's opinion (ie. I delete nothing, except spam and bad personal attacks). Don't be too rude, try to stay polite, but above all, engage using your best arguments, especially towards other commenters (they may not laugh it off as easily as I do). And allow some air between paragraphs and rebuts. Don't get off the lawn. Have fun. Enjoy.