Skip to main content

Fast supercomputers in your bedroom!

Note: Another message to the Skeptics Guide to the Universe that's disappeared into a black hole. This is a about a news item about a new fast neural net experiment, and my reply in order to bring some reality to the AI hype.

---


Just finished the latest great episode and wanted to give some context and insight into the fast simulation of neural networks featured in Science or Fiction (item 1).

The technique isn't new, it's based on prior work back when computers were even slower. It's essentially a way to move some data generating/processing from the CPU into the functional GPU space because GPUs are designed to blast through things like rastrums really fast with anonymous functions really really fast, so if you can move your data into procedural buffers directly in the GPU you have a big speed improvement on your hand. To understand why there's some basic software development concepts we need to dig into;

A typical (and often the first way people learn) way of doing software development is what's known as imperative programming where you as a developer prescribe the logical flow of your program. Most software, including traditional simulation software, are being done this way to various degrees so that we can do complex network operations and have more code reuse (and this is important), but there's always room for improvement. It's essentially a program that does this, then than, then if this, then that, otherwise that, then this, then do that, then do this, otherwise do this other.

A method that's faster and less error prone is known as functional programming where the logic of the program is described through passing functions around instead of the data itself. It's in many ways better, but much harder to wrap your head around. Most pieces of software is a mix of the two (and more) methodologies, but the more you can muster functionally (especially if you design your passable functions well), the better (in my opinion). Instead of pushing large amounts of data around with boolean logic and human intuition, a functional model of processing is using a mathematical language to operate over complex data, which is far superior for optimisations. We no longer say "do this, then that, otherwise this", you describe a formula that's compatible with your data model so you don't have to check that you're within bounds; your formula will only operate within bounds.

The news item is a mix of a) going from imperative to functional logic, and b) a JIT* way creating synaptic weights instead of pushing that data around. Essentially you spend a lot less time pushing lots of data around for processing by pushing small and efficient functions around instead.

The neural net part of the item is kinda less important. First, neural net simulations are only as good or interesting as your understanding of the neurals and synapses you're trying to simulate**. This particular simulation is based on 1 specific model using GeNN (model available upon request), so its application to other models or even simulations is not really know, but you can assume it will be (as the procedural technique is for everything, only the synaptic weights are neural net simulation specific).

So what we can say now is that if you have a modern NVIDIA GPU (so, not all GPUs, not all brands), you can simulate this model much faster. It's a good progressive step forward, but as a research project still quite limited in what it can do.

Hope any of that were helpful and made some sense.


Cheers,
Alex


* JIT - Just-in-time; compiling smaller pieces of code just before it needs to be run, instead of the more traditional "compile everything before you even run the program."

** https://sheltered-objections.blogspot.com/2021/02/i-have-been-listening-regularly-for-few.html

Comments

Popular posts from this blog

Hyper AI : Sam Harris and his techno-bros on Artificial Intelligence

TL;DR: Don't be seduced by Hollywood bullshit! Engage those critical thinking skills Bias and background Recently someone asked Sam Harris in his " Ask me anything #1 " episode about his latest views on Artificial Intelligence  ( AI for short ), and they weren't all positive. You see, he's lately been invited along to a conference on the matter of threats to human existence, and AI feature very, very high on the list of potential human extinguishers. He lists Elon Musk as his friend and inviter, and within the conference refers to " people close to development of AI " that all agree with the following; AI's will get smarter or more advanced than the human intelligence, will be able to modify and improve their own code, and come to some negative conclusion about us puny humans, whereas the next logical step is, of course, " Exterminate the humans! " I might be paraphrasing. Harris didn't list who these people " close to the

The Skeptics guide to Artificial Intelligence

I have been listening regularly for a few years to the Skeptics Guide to the Universe , and I like it a lot. But sometimes you've got an itch than just won't easily scratch away, and this is one of those times. I should note that sent this to the website through their contact form, but I've never been lucky getting anything through (or any response) from doing so several times in the past, which is why I post it here so it doesn't go to waste. I don't actually know if anything goes through there, or I'm ignored, or marked as spam, or they're too busy with my pedestrian requests, could be anything. Normally I wouldn't care, but I wrote this rather lengthy letter about something I care deeply about (and hopefully also can teach someone out there something about the neuance of the topic) and decided to share it. My Skeptics guide to Artificial Intelligence is written with a love for the show. ------------- So. You know how you guys sometimes riff over the

Dune (2021) review by a Dune fanatic

 So, I watched the new Dune movie yesterday, and my mind is conflicted, stirred, perplexed and bewildered, so in order to sort myself out, I decided to write this review. As a film nut I'm going to compare Dune (2021) by Dennis Villeneuve to Dune (1984) by David Lynch ( and definitly not the Alan Smithee version ), and skip the Dune miniseries as I only watched it bitsy and half-heartedly way too long ago to matter now. There will be tons of spoilers, opinions and other dangers ahead. Now, other people have delved into the usual stuff about visuals ( they are lovely ), special effects ( they are great ), cinematography ( it is awesome ), sound ( it is fantastic ) and so forth, but I wanted to focus less on that and more on the story, the point of the book, the characters themselves, and dig into the nitty gritty of how these films cater to a serious Dune nerd like myself. Storytelling Now, the story of the book and how it is told has often been described as un-filmable, especially