What's behind matrix multiplication?

Tivadar Danka small portrait Tivadar Danka
Matrix multiplication

Understanding math will make you a better engineer.

So, I am writing the best and most comprehensive book about it.

Matrix multiplication is not easy to understand.

Even looking at the definition used to make me sweat, let alone trying to comprehend the pattern. Yet, there is a stunningly simple explanation behind it.

Let's pull back the curtain!

First, the raw definition. This is how the product of AA and BB is given. Not the easiest (or most pleasant) to look at.

Matrix multiplication

We are going to unwrap this. Here is a quick visualization before the technical details. The element in the ii-th row and jj-th column of ABAB is the dot product of AA's ii-th row and BB's jj-th column.

Matrix multiplication visualized

Now, let's look at a special case: multiplying the matrix AA with a (column) vector whose first component is 11, and the rest is 00. Let's name this special vector e1e_1. Turns out that the product of AA and e1e_1 is the first column of AA.

Matrix-vector product

Similarly, multiplying AA with a (column) vector whose second component is 11 and the rest is 00 yields the second column of AA.

That's a pattern!

Matrix-vector product

By the same logic, we conclude that AA times eke_k equals the kk-th column of AA.

Matrix-vector product

This sounds a bit algebra-y, so let's see this idea in geometric terms. Yes, you heard right: geometric terms.

Matrices represent linear transformations. You know, those that stretch, skew, rotate, flip, or otherwise linearly distort the space. The images of basis vectors form the columns of the matrix.

We can visualize this in two dimensions.

A linear transformation in two dimensions

Moreover, we can look at a matrix-vector product as a linear combination of the column vectors. Make a mental note of this, because it is important.

Matrix-vector product

(If unwrapping the matrix-vector product seems too complex, I got you. The computation below is the same as in the above, only in vectorized form.)

Matrix-vector product

Now, about the matrix product formula. From a geometric perspective, the product ABAB is the same as first applying BB, then AA to our underlying space.

Product of linear transformations

Recall that matrix-vector products are linear combinations of column vectors. With this in mind, we see that the first column of ABAB is the linear combination of AA's columns. (With coefficients from the first column of BB.)

First column of the matrix product

We can collapse the linear combination into a single vector, resulting in a formula for the first column of ABAB. This is straight from the mysterious matrix product formula.

First column of the matrix product

The same logic can be applied, thus giving an explicit formula to calculate the elements of a matrix product.

Matrix product

Linear algebra is powerful exactly because it abstracts away the complexity of manipulating data structures like vectors and matrices. Instead of explicitly dealing with arrays and convoluted sums, we can use simple expressions ABAB.

That's a huge deal.

Peter Lax sums it up perfectly: "So what is gained by abstraction? First of all, the freedom to use a single symbol for an array; this way we can think of vectors as basic building blocks, unencumbered by components."

Having a deep understanding of math will make you a better engineer.

I want to help you with this, so I am writing a comprehensive book that takes you from high school math to the advanced stuff.
Join me on this journey and let's do this together!