Philosophical Discussion of Issues in Software

12th May 2026
GPUs: From 3D graphics to neural networks - How multi-dimensional arrays of numbers were repurposed.

First there was Pong (early 1970s), the first commercial video game. The first time computer software was used to animate lines and dots on a screen for fun. (Actually it wasn't software, as such. It was hard-wired.)

As the games industry progressed, through the likes of Space Invaders and Pac-Man, the graphics got more sophisticated but the way they were drawn was still basically the same. A 2D array of numbers representing a 2D array of pixels on the screen. But by about 1980 we come to games like Atari Battlezone, using 3D wire-frame graphics.

Now we're not just drawing sprites, where each pixel in the sprite corresponds to a pixel on the screen. Those pixels on the screen represent lines connecting points in an imaginary 3D space. We now have to separate the world (3D space) from a particular view of that world (2D screen) - how the world is and how we know/see it. Ontology and Epistemology rear their ugly philosophical heads.

The part of the software which draws that 3D world now has (1) a representation of those X,Y,Z coordinates - vectors - and (2) a way to translate those sets of 3D vectors into 2D screen positions, so they can be drawn. And to be able to draw those vectors from the viewpoint of anywhere in that 3D space we need to be able to tranform them - rotate, translate, etc. To do that we need to multiply them by transformation matrices. So we've reached the stage where vectors and matrices need to be represented in the software, and the computer needs to be able to do the standard things you do with vectors and matrices - multiply them together and suchlike.

In The Beginning Was The Pixel...

As those games get more complicated, their software needs to represent more of those vectors and matrices and do more math[s] on them. The CPU in the computer did all of that calculating. But around the end of the 20th Century NVIDIA decided to invent a processing unit that could specialize in just those kinds of calculations, leaving the CPU to do other things. It's specifically for handling these kinds of graphics, so it's called a Graphics Processing Unit (GPU). It does all that 3D vector/matrix maths quickly by using an increasing number of processors working in parallel.

Lots of Matrix Mathematics

Speaking of things happening in parallel, something else was happening while these increasingly sophisticated computer games were being written. The development of Artificial Intelligence, as with so many human inventions, took its inspiration from Nature - from biological neural networks. Lots of inter-connected nodes (neurons). The decision as to whether a given neuron "fires" (sends a signal to the neuron(s) to which it's connect in the next layer) depends on a sum of the weighted input signals. Lots of multiplications and additions to do for lots of inputs to lots of neurons. That means vectors, matrices and tensors (matrices which can have more than 2 dimensions) to keep track of all those numbers and calculation. So it turns out that the underlying mathematics is similar to that of 3D graphics even though the overlying application is very different. On the face of it, 3D graphics and artificial brains are quite different things. Yet mathematically they're similar enough that the same calculation hardware - GPUs - can be used to represent them.

This often turns out to be the way of things with mathematics. Because the world as a whole tends to follow patterns and rules, and since mathematics is all about describing patterns/regularities/logical consistencies, parts of that world that seem very different can often turn out to be describable by similar maths. Take, as another example, financial derivatives and fluid diffusion. The pricing of financial derivatives (stock options at al) turns out to work in the same way as the diffusion of a gas down a tube.

It's a funny old world.

Any thoughts?

If you have any interesting comments on this or other sections of the site, feel free to contact me.