Every implicit time step you will ever take, every pressure projection in a fluid solver, every steady-state heat calculation ends in the same line of code: solve . Your first linear algebra course taught you the mechanics of that line: write the augmented matrix, eliminate, back-substitute. It probably never told you what the equation is asking, or why some systems have one answer, some have none, and some have infinitely many. In 1.2 Matrices as Transformations a matrix became a machine that moves space. This lesson reads through that picture, and the two most important subspaces in linear algebra, the column space and the kernel, fall right out.
Transcript
- You have probably solved hundreds of linear systems by Gaussian elimination. Today we find out what those systems were actually asking.
- Last lesson, a matrix became a machine that moves every vector in the plane. So the equation A x equals b is really a question: which input vector does the transformation A send to the output b?
- For most matrices the answer is easy: exactly one input. The interesting cases, the ones your old course never explained, are when there is no answer at all, or infinitely many.
- Here is a matrix that will cause trouble. Its first column, one and a half, is where i hat lands. Its second column, two and one, is where j hat lands.
- Notice anything? The second column is exactly twice the first. Both landing spots lie on the same line.
- Watch what A does to the whole plane. Every grid line, every vector, gets crushed onto a single line.
- That line is every output A can possibly produce. It is called the column space, because it is the span of the columns. Here it is only one dimensional.
- Now pick a target b that is off the line, say one, three. No input can ever land there, because nothing A outputs leaves the green line. A x equals b has no solution.
- That is the first big idea. A x equals b is solvable exactly when b lies in the column space of A.
- Let's rewind. Here is a second question: which inputs get sent to zero? Take the line of vectors along negative two, one.
- Apply A, and every single one of them collapses onto the origin. This whole line is squashed to zero.
- The set of everything A sends to zero is called the kernel, or null space. You find it by solving A x equals zero. Here, x plus two y equals zero.
- Now choose b on the column space: four, two. One input that works is four, zero, because four times the first column is four, two. Call it a particular solution, x p.
- But we can add any kernel vector to x p. A sends the kernel part to zero, so the sum still lands on b. Slide along, and you trace a whole line of solutions, parallel to the kernel.
- Apply A to that entire line, and all of it lands on the single point b. Every one of those inputs answers our question.
- So the general solution is a particular solution plus anything in the kernel. Remember this shape. Linear differential equations have exactly the same structure.
- Count dimensions. The column space is a line, so the rank of A is one. The kernel is a line, so the nullity is one. One plus one is two, the dimension of the input space.
- That is the rank nullity theorem. Every input dimension is either kept, and shows up in the output, or crushed into the kernel. Nothing is lost without a trace.
- Compare a healthy matrix. Its columns point in different directions, so the plane is stretched and sheared but never flattened.
- Now the kernel is just the zero vector, the rank is two, and every b has exactly one solution. That means the transformation can be undone: B has an inverse.
- And here is what elimination was doing all along. Row reduce the system with b equal to four, two. The second row becomes all zeros: zero equals zero, no contradiction.
- The first column has a pivot. The second has none, so y is a free variable. Set y equal to t, and x equals four minus two t. That is the line we just saw.
- With b equal to one, three, the same steps give zero equals five halves. That impossible row is elimination's way of saying b is not in the column space.
- One last thing, and it matters for fluids. An incompressible fluid solver finds the pressure by solving a giant linear system, one unknown per grid cell.
- But only pressure differences push the fluid. Raise every cell by the same amount and the differences, and the forces, do not change.
- So the constant vector, all ones, is in the kernel of the pressure matrix. The matrix is singular, pressure is only defined up to a constant, and real solvers pin one cell or subtract the mean.
- To recap. The column space holds every reachable output, so it decides whether a solution exists. The kernel holds everything crushed to zero, so it decides how many.
- Next time: a single number, the determinant, that tells you whether a matrix flattens space at all.
is a question about a transformation
Recall the two readings of a matrix–vector product from the previous lessons. Column by column,
so is a linear combination of the columns with weights taken from . Geometrically, is a function: it takes an input vector and returns an output .
So the equation asks one question in two languages:
- Transformation language: which input vector(s) does send to the output ?
- Combination language: which weights build out of the columns of ?
You met the second version in 1.1 Vectors, Linear Combinations & Span, when we found coefficients of a linear combination by solving a little system. Now we can ask the two follow-up questions every scientist cares about:
- Existence. Is there any input that lands on ?
- Uniqueness. If so, is there exactly one, or many?
The column space answers the first question, and the kernel answers the second.
Column space: every output you can reach
The column space of is the set of all possible outputs:
It is the span of the columns, so it is always a flat subspace through the origin of the output space. This gives us the first theorem of the lesson almost for free:
Take the matrix from the video,
Its second column is exactly twice its first column . Every output is , a multiple of one vector. So the column space is the line through the origin in the direction . The entire plane gets crushed onto that line. Then:
- lies on the line, so is solvable.
- does not (it isn’t a multiple of ), so has no solution. No input, however clever, can land off the line.
When the columns of a 2×2 matrix are independent, the column space is all of and every is reachable.
Kernel: everything that gets squashed to zero
The second subspace lives in the input space. The kernel (or null space) of is everything sends to the zero vector:
The system is called the homogeneous system. It always has at least the trivial solution , because a linear map sends the origin to the origin. The kernel is interesting when it contains more than that.
For our singular , means , i.e. . That is the line
Every vector on this line collapses onto the origin. You can check one directly: .
Why is the kernel always a subspace? If and , then by linearity . So combinations of kernel vectors stay in the kernel, which makes it a flat subspace through the origin: a point, a line, a plane, and so on.
General solution = particular + homogeneous
Here is why the kernel controls uniqueness. Suppose is one solution of (a particular solution), and is any vector in the kernel. Then
So adding a kernel vector to a solution gives another solution. Conversely, if and are two solutions, their difference satisfies , so it lies in the kernel. Together:
For our example with : works, because . The general solution is
a line parallel to the kernel. In the video, applying to this whole line collapses it onto the single point . You could just as well have picked or . Different particular solutions just slide the starting point along the same line.
So a linear system has zero, one, or infinitely many solutions, never exactly two: either (none), or the solution set is a shifted copy of the kernel (a single point if the kernel is trivial, otherwise a line, plane, …).
Things to try:
- Start with the lesson example. Drag around the right panel. It turns red (no solution) off the green column space and green when it’s on it, and then the pink solution line appears on the left, parallel to the red kernel. The sim marks the solution closest to the origin as , but any point on the pink line would do.
- Drag along the pink line on the left. Its image on the right doesn’t move at all. That is ” sends the whole line to ” in action.
- Switch to Rank 2: invertible. The kernel shrinks to a dot, the column space fills the plane, and the solution set becomes a single point for every .
- Pick any matrix and press Make column 2 = 2 × column 1. Watch the transformed grid on the right fold up into a line and the readout’s rank drop from 2 to 1 while the nullity rises from 0 to 1.
Elimination, revisited
Now let’s redo the mechanics you already know, but read every step geometrically. Row operations are:
- swap two rows,
- multiply a row by a nonzero number,
- add a multiple of one row to another.
Each is reversible, so row operations never change the solution set of (and in particular never change the kernel). What they do change is how easy the system is to read. They do change the column space in general, which is why we read column-space information off the original columns, as you’ll see below.
Echelon forms, pivots and free variables
A matrix is in row echelon form (REF) when all zero rows are at the bottom and each row’s first nonzero entry (its pivot) is strictly to the right of the pivot above it. It is in reduced row echelon form (RREF) when, in addition, every pivot equals 1 and is the only nonzero entry in its column. The RREF of a matrix is unique; REFs are not.
- A pivot column contains a pivot. Its variable is a pivot (basic) variable.
- A column with no pivot gives a free variable: you can set it to anything, and the pivot variables adjust.
The rank of is the number of pivots.
Worked example: a 3×4 system done carefully
Solve with
Step 1: eliminate below the first pivot. Use and on the augmented matrix:
Step 2: look for the next pivot. Column 2 has nothing below row 1, so it has no pivot: will be free. The next pivot is the 1 in column 3, row 2. Eliminate below it with :
This is already the RREF (the pivots are 1 and each pivot column is otherwise zero). The bottom row reads : no contradiction, so the system is consistent.
Step 3: read it off. Pivots sit in columns 1 and 3, so the rank is 2. The free variables are and . The two nonzero rows say
so and . In vector form:
Step 4: check. is column 1 plus column 3: . ✓ And , . ✓
The general solution is exactly “particular + homogeneous”: setting the free variables to zero gives , and the coefficients of and are the kernel vectors.
Reading the kernel straight from the RREF
The recipe in Step 3 always works. For the right-hand side is zero, so the particular part vanishes and you get:
They are automatically independent: each has a 1 in its own free slot where the others have 0. So
This dimension is called the nullity of .
Reading the column space and solvability
The pivot columns of the original matrix form a basis for . Here that’s columns 1 and 3: and . The other columns are combinations of them, and the kernel vectors record the recipes: says column 2 = 2·column 1, and says column 4 = 3·column 1 + 2·column 3. So is a plane in .
Which are in that plane? Run the same elimination on a general right-hand side . The last row becomes
so the system is solvable exactly when . That is the equation of the column-space plane. Our passes: . A right-hand side like would produce the impossible row .
Rank and the rank–nullity theorem
The rank of is the dimension of its column space, which equals the number of pivots:
Every column of is either a pivot column or a free column, so counting columns gives:
Geometric meaning. The input space has dimensions’ worth of directions. keeps some of them, which show up as the column space, and crushes the rest into the kernel. No dimension is lost without a trace. In the 2×2 example, one direction (along ) is crushed and one survives as the column-space line: . In the 3×4 example: .
Transcript
- The same ideas work in three dimensions. Let's watch a three by three matrix flatten space.
- This matrix should look familiar from lesson one point one. Its third column is the sum of the first two, so the three columns only span a plane.
- Here is a lattice filling a block of space. Apply the matrix, and the whole block is pressed flat into that plane: the column space.
- The column space is the plane z equals x plus y. Any target b off this plane has no solution. The rank is two.
- Rewind. One direction must be lost: rank two plus nullity one makes three. The kernel is the line along negative one, negative one, one.
- Apply the matrix once more. The block flattens, and the entire kernel line collapses to the origin.
- Three dimensions in, two dimensions out, one dimension crushed. That is rank nullity in a picture.
The video’s matrix is the dependent example from 1.1 Vectors, Linear Combinations & Span: columns , and their sum . Three input dimensions go in, a two-dimensional plane () comes out, and the line spanned by is crushed to zero: .
Some consequences you can now predict without computing anything:
- Wide matrices (, more unknowns than equations): rank , so the nullity is at least . A consistent wide system always has infinitely many solutions.
- Tall matrices (, more equations than unknowns): rank , so the column space is a proper subspace of . Some right-hand sides have no solution. That’s why overdetermined data-fitting problems use least squares instead.
- Square matrices: rank ⇔ nullity 0 ⇔ every has exactly one solution. One property gives you all three.
Inverse matrices
A square matrix is invertible if there is a matrix with
Geometrically, is the transformation that undoes . That’s only possible if never merges two inputs into one output. If with , then wouldn’t know where to send that output. Since means , a map can be undone exactly when its kernel is trivial. Combined with rank–nullity, this gives the central equivalence of the subject:
The 2×2 inverse
For ,
You can check it by multiplying: . The number is the 2×2 determinant, the same independence test from lesson 1.1. When it’s zero the columns are parallel, the plane is flattened, and no inverse exists. For our singular example, , as expected.
Example. The healthy matrix from the video, , has , so
Check the first row of : and . ✓
Gauss–Jordan for larger matrices
To invert an matrix, row reduce the wide matrix . If reduces to , the right half becomes :
Why does this work? The row operations amount to multiplying on the left by some matrix with . Applying the same to gives . Equivalently, you are solving the systems at once, and their solutions are the columns of . If a pivot is missing, is singular and Gauss–Jordan tells you so.
Worked example. Invert . This is the video’s rank-2 matrix with its corner entry changed from 2 to 3, which breaks the dependence.
So . Spot check: row 3 of times column 3 of is . ✓ With the original corner entry 2, the same elimination would have produced the row on the left: no third pivot, no inverse.
Fluids: the pressure solve and its kernel
Here is the biggest you’ll meet in this course. An incompressible fluid solver (Stam’s Stable Fluids, and the 11.10 The Projection Method for Incompressible Flow) first moves the velocity field forward, which leaves it slightly compressible. Then it finds a pressure whose gradient removes the divergence, by solving the Poisson equation
Discretized on a grid with one unknown per cell, the Laplacian becomes a matrix (you’ll build the operator in 2.11 The Laplacian & Vector Identities and the discretization in 10.1 Finite Differences: Discretizing Derivatives). In 1D with spacing , the row for cell is the stencil . Solid walls impose a Neumann condition (no flow through the wall, so ), which modifies the end rows. For four cells:
Every row sums to zero, so . The constant vector is in the kernel. Physically, only pressure differences push fluid. Raise the pressure in every cell by the same amount, and every force stays the same. Everything in this lesson now applies:
- Pressure is only defined up to a constant. If solves , so does . That’s particular + homogeneous.
- is singular (rank , nullity 1), so
np.linalg.solvewill fail or return garbage. - Not every right-hand side is allowed. Adding up all the rows of gives zero, so the entries of always sum to zero. A solution can exist only if . Physically, the net divergence in a closed box must be zero: you can’t pump fluid into a sealed container. Rank–nullity shows this is also sufficient. The column space has dimension and sits inside the -dimensional set of vectors whose entries sum to zero, so the two are equal.
Code it yourself
First, use SymPy’s exact arithmetic to confirm the worked example. Then you’ll write the algorithm.
Now use row reduction to produce the full solution set of a singular system, or detect that there isn’t one.
Practice problems
Let .
- Describe the column space and the kernel geometrically, and give a basis for each.
- Find the general solution of .
- Does have a solution?
- Verify rank–nullity.
Show worked solution
- Column 2 is times column 1, so , a line through the origin with slope . For the kernel, reduces to the single equation (the second row is times the first), so and .
- is in the column space. Row reduce: gives . So , i.e. . With free: . Check: ✓, and ✓.
- The same row operation gives the last row , which reads . No solution: is not a multiple of .
- Rank 1 (one pivot) + nullity 1 (one free variable) = 2 = number of columns. ✓
Let .
- Find the RREF of , its rank, and a basis for .
- For which is solvable? Describe this set geometrically.
- Find the general solution for .
Show worked solution
- and give rows and . Then zeros the last row, and clears above the second pivot:
Pivots in columns 1 and 3, so the rank is 2. is free. Setting gives , , so . Check: ✓. Nullity 1, and ✓. 2. Track the right-hand side through the same operations. Row 2 becomes , row 3 becomes , and then row 3 becomes . The system is solvable iff . That’s a plane through the origin in , the column space, spanned by the pivot columns and . (Both satisfy ✓.) 3. satisfies ✓. Reducing the augmented matrix: rows become , , , and then gives . So and :
Check: ✓.
- Compute the inverse of and use it to solve .
- For which values of is invertible? For each value where it isn’t, find its kernel and column space.
Show worked solution
-
, so . Check: ✓. Then . Check: , ✓.
-
, which is zero exactly when . So is invertible for all .
- : . The kernel is given by , so it’s . The column space is .
- : . The kernel is given by , so it’s . The column space is .
In both cases rank 1 + nullity 1 = 2. (Because is symmetric, its kernel comes out perpendicular to its column space. That’s no accident, as you’ll see when we study symmetric matrices.)
- Can a matrix have a trivial kernel? Can a matrix have ? Justify both answers with rank–nullity.
- For the 4-cell Neumann Laplacian in the fluids section (take ), compute by hand and find the rank.
- Show that is solvable if and only if .
- Solve , giving the general solution, and then the solution with mean pressure zero.
Show worked solution
- No. A matrix has at most 3 pivots (one per row), so its rank is at most 3 and its nullity is . No again for the matrix: its rank is at most 3 (one pivot per column), so is at most a 3-dimensional subspace of and can’t be all of it.
- reads , , , . The first gives , the second then , and the third (the fourth is then automatic). So , the nullity is 1, and the rank is .
- (⇒) Each column of sums to zero (equivalently, the rows add up to the zero row), so for any the entries of sum to . So if , then . (⇐) The set is a 3-dimensional subspace (one linear condition in ). We just showed , and . A 3-dimensional subspace inside another 3-dimensional subspace must equal it, so .
- ✓. Pin . Row 1: gives . Row 2: gives . Row 3: gives . Row 4 checks: ✓. General solution: . The mean of is , so the zero-mean solution is : . Physically, fluid enters at one end and leaves at the other, so the pressure ramps linearly to push it through.
Looking ahead
You now have the two subspaces that organize the rest of the module. In 1.4 Determinants as Volume Scaling a single number, , will detect at a glance whether a square matrix flattens space, i.e. whether its kernel is nontrivial. Then in 1.5 Eigenvectors & Eigenvalues the eigenvectors of for eigenvalue will turn out to be exactly the kernel of : finding eigenvectors is a null-space computation, and you just wrote the code for it.
Key Takeaways
- asks which input sends to , or equivalently which combination of the columns builds .
- The column space is every reachable output (the span of the columns). The system is solvable iff .
- The kernel is everything crushes to . It decides uniqueness: the general solution is with , the same structure as linear ODEs.
- Row reduction to RREF exposes everything: pivots count the rank, free variables generate a kernel basis, and a row means .
- Rank–nullity: . Every input dimension is either kept or crushed.
- A square matrix is invertible ⇔ trivial kernel ⇔ full rank ⇔ columns form a basis. Use for 2×2 and Gauss–Jordan on for larger matrices, but solve systems in code with
solve, notinv. - The fluid pressure matrix has the constant vector in its kernel. Pressure is defined only up to a constant, the right-hand side must have zero net divergence, and solvers pin a cell or subtract the mean.