Run any explicit time stepper long enough and one of two things happens: the numbers settle down, or they explode. In your Simulations course you saw forward Euler blow up on a stiff problem when the step was too big, and you probably learned a rule like “keep below ”. This lesson explains where that comes from. A linear update applies the same matrix over and over, and in the last three lessons we learned to see a matrix as a transformation (1.2 Matrices as Transformations), to find its kernel (1.3 Linear Systems, Kernel & Column Space), and to measure how it squashes space (1.4 Determinants as Volume Scaling). Now we put those together to find the few special directions that a matrix doesn’t turn: its eigenvectors. Along those directions the matrix is just multiplication by a number, the eigenvalue, and those numbers decide growth, decay, oscillation and numerical stability for everything else in this course.
If eigenvectors never made sense to you before, that’s normal. Most courses start with the formula. We’ll start with the picture.
Transcript
- Eigenvectors and eigenvalues. If these words never quite clicked for you, this is the video where they should. We start with pictures, not formulas.
- Here is a matrix A, thought of as a transformation of the whole plane. Its columns tell you where i hat and j hat land, and the grid lines follow along.
- Pick a single vector, x, and draw the line through it, its span. Now apply A, and keep your eye on that line.
- The arrow has been knocked off its own line. That is what almost every vector does: a matrix rotates it as well as stretching it.
- Let's do the same thing to a whole fan of vectors at once, each one drawn with its own span line.
- Apply A. Watch the lines, not the arrows. Nearly every arrow swings away from the line it started on.
- But two directions are special. Vectors on these two lines stay on their own line. They do not rotate at all. They only get stretched or shrunk.
- These are the eigenvectors of A. Along the direction two, one, every vector is stretched to twice its length. Along minus one, one, every vector shrinks to half. Those scale factors, two and one half, are the eigenvalues.
- In symbols, A v equals lambda v. Applying the matrix to an eigenvector gives back the same vector, just multiplied by the number lambda.
- The value of lambda tells you exactly what happens along its line. Here is a different matrix, B, with eigenvectors along the two diagonals.
- Apply B. Along one, one, lambda is one and a half, a stretch. Along one, minus one, lambda is minus one: the arrow flips to point the opposite way, but it never leaves its line.
- So, lambda bigger than one stretches. Lambda between zero and one shrinks. A negative lambda flips. And lambda equal to zero crushes that whole line to the origin.
- Not every matrix has eigenvectors you can draw. A rotation by ninety degrees turns every single vector off its line.
- So a rotation has no real eigenvectors at all. Its eigenvalues turn out to be complex numbers, and in lesson one point seven we will see that complex eigenvalues mean rotation.
- A shear is stingy in a different way. It slides each row of the grid sideways. Only the horizontal direction stays on its line, with lambda equal to one.
- The eigenvalue one is repeated, it appears twice, but there is only one line of eigenvectors. A matrix like this, which runs short of eigenvectors, is called defective.
- At the other extreme, two times the identity stretches everything by two. Every vector stays on its line, so every nonzero vector is an eigenvector, all with lambda equal to two.
- Eigenvectors also explain what happens when you apply a matrix over and over, which is exactly what a time-stepping simulation does. Back to our matrix A, and its two eigen-lines.
- Start with a vector x, close to the shrinking direction. Then compute A x, A squared x, A cubed x, and so on.
- Every step, the part of x along the green line is cut in half, and the part along the amber line is doubled. Very quickly, the amber part wins, and the iterates line up with the dominant eigenvector.
- Now rescale to the same length after every step, so nothing runs off the screen. From any starting direction, the arrow swings onto the dominant eigenvector.
- This is power iteration: multiply, normalize, repeat. It is how computers find the largest eigenvalue of matrices with millions of rows, and it is why a time stepper's biggest eigenvalue decides whether it blows up.
- Recap. An eigenvector is a direction the matrix only scales, never turns. The eigenvalue is the scale factor: stretch, shrink, or flip. In the next video, we compute them by hand.
Directions a matrix doesn’t turn
Take a matrix and a nonzero vector . Draw the line through the origin along , its span. Now apply . In general points somewhere else entirely: the matrix has rotated the vector off its line as well as stretching it. For example, with
and is nowhere near the line through .
But try : . That vector stayed on its own line and simply doubled. Try : . Same line again, now shrunk to half. Every multiple of also doubles and every multiple of also halves, because . So the matrix has two invariant directions: lines that it maps onto themselves.
Two details matter. First, is excluded, because for every , which tells us nothing. Second, is allowed. It means : the whole line is crushed to the origin, so is in the kernel. (“Eigen” is German for “own” or “characteristic”: these are the matrix’s own directions.)
The value of tells you exactly what happens along its line:
| Eigenvalue | What does to the eigen-line |
|---|---|
| stretches it | |
| shrinks it toward the origin | |
| leaves every point on it fixed | |
| crushes it to the origin | |
| flips each vector to the other side of the origin, scaled by |
Some matrices let you read off eigenvectors just by thinking about the geometry:
- Diagonal matrix : it scales the -axis by 3 and flips the -axis. Eigenvectors () and ().
- Projection onto a line : vectors on are untouched (), and vectors perpendicular to are sent to zero ().
- Reflection across a line : vectors on stay put (), and vectors perpendicular to flip ().
- Rotation by 90°: every nonzero vector is turned off its line. There are no real eigenvectors at all.
Now hunt for eigenvectors yourself.
Things to try:
- With the General preset, drag slowly around. The readout shows the angle between and . Find the two directions where it drops to . Check that the readout shows along and along .
- Tick Sweep x around the circle and watch the sweep slow down and light up near each eigen-direction. Notice that the eigen-lines are where the circle’s image (the ellipse) touches the dashed line through .
- Switch to Stretch + flip. One eigenvector comes out pointing the opposite way: that’s .
- Try Rotation 90°: you can’t win. Try Shear: only one line works. Try Scalar 2I: everything works.
- Pick any preset with real eigenvalues and press Run 20 steps. Hold that thought until the power-iteration section below.
From geometry to algebra: det(A − λI) = 0
Dragging vectors around is great for intuition, but we need a way to compute eigenvectors. Here is the whole derivation, and it uses only things you already know.
Transcript
- In the last video we saw eigenvectors as directions a matrix only stretches. Now let's find them by hand. The key tool is the determinant.
- Start from the definition. We want a nonzero vector v and a number lambda with A v equal to lambda v.
- Move everything to one side. To factor out v, write lambda v as lambda times the identity matrix times v. So the matrix A minus lambda I sends v to zero.
- So v is a nonzero vector in the kernel of A minus lambda I. A nonzero kernel means the matrix squashes the plane flat, and from lesson one point four, that happens exactly when the determinant is zero.
- Watch it happen with the matrix four, one, two, three. On the left, the unit square after applying A minus lambda I. On the right, its signed area, the determinant, plotted against lambda.
- As lambda grows, the parallelogram gets thinner. At lambda equals two it collapses onto a line. The determinant is zero, so lambda equals two is an eigenvalue.
- Past two, the area turns negative: the orientation has flipped. Then at lambda equals five it collapses again. A second eigenvalue.
- For every other lambda, the matrix is invertible, and only the zero vector maps to zero. The eigenvalues are exactly where this curve crosses zero.
- That curve is the characteristic polynomial. Compute it: four minus lambda times three minus lambda, minus one times two. That's lambda squared minus seven lambda plus ten, which factors as lambda minus two times lambda minus five.
- Each eigenvalue gives a kernel to find. For lambda equals five, A minus five I has rows minus one, one and two, minus two. Both rows say y equals x, so the eigenvector is one, one.
- For lambda equals two, both rows say two x plus y equals zero, so the eigenvector is one, minus two.
- Always check. A times one, one is five, five, which is five times one, one. And A times one, minus two is two, minus four, which is two times one, minus two.
- Two free checks. The eigenvalues add up to the trace, the sum of the diagonal: five plus two equals four plus three. And they multiply to the determinant: five times two equals twelve minus two.
- In fact, every two by two characteristic polynomial is lambda squared, minus the trace times lambda, plus the determinant. Two numbers you can read off the matrix decide both eigenvalues.
- Why do simulation people care so much? Take any iteration, x n plus one equals A x n. Write the start in eigenvector coordinates. Then each step simply multiplies each component by its own eigenvalue.
- So if every eigenvalue has size less than one, the iteration decays to zero. If even one eigenvalue has size bigger than one, it blows up.
- Here is one you already know. Forward Euler on y prime equals lambda y gives y n plus one equals one plus h lambda, times y n. The number one plus h lambda is the eigenvalue of the time stepper.
- With lambda equal to minus ten and a step of five hundredths, the multiplier is one half, and the numerical solution decays, just like the true one.
- With a step of one quarter, the multiplier is minus one and a half. Its size is bigger than one, and the numerical solution oscillates and explodes, even though the true solution decays.
- For a system, Euler multiplies by the matrix I plus h A, whose eigenvalues are one plus h lambda for each eigenvalue lambda of A. Stability means every one of them stays inside the unit circle.
- A fluid solver plays the same game with a matrix that has millions of rows. The diffusion step's eigenvalues reach about minus four nu over delta x squared, and keeping them all stable forces the time step below delta x squared over two nu. Stability limits like the CFL condition are eigenvalue bounds.
- And for the differential equation x prime equals A x, every eigenvector hands you a solution for free: e to the lambda t, times v. That idea is the engine of module seven.
- Recap. Solve the determinant of A minus lambda I equals zero to get the eigenvalues. For each one, find the kernel of A minus lambda I to get the eigenvectors. Check with the trace and the determinant. Next time, we use eigenvectors as a basis, and the matrix becomes pure scaling.
Start from and move everything to one side. To factor out , write as , where is the identity matrix (you can’t subtract a number from a matrix, but you can subtract ):
So is a nonzero vector in the kernel of . From 1.3 Linear Systems, Kernel & Column Space, a square matrix has a nonzero kernel exactly when it is not invertible, and from 1.4 Determinants as Volume Scaling, that happens exactly when its determinant is zero (the matrix squashes space flat). Therefore:
The characteristic polynomial
Expanding always produces a polynomial in , called the characteristic polynomial . For an matrix it has degree , so an matrix has at most distinct eigenvalues. Counting complex roots and repeats, it has exactly (by the fundamental theorem of algebra). The number of times appears as a root is its algebraic multiplicity.
For a matrix, it’s worth deriving once and for all:
The coefficient is the trace (sum of the diagonal) and is the determinant, so
The discriminant sorts every matrix into three cases:
- : two distinct real eigenvalues, two eigen-lines;
- : one repeated real eigenvalue (see the section on repeated eigenvalues below);
- : a complex-conjugate pair, no real eigenvectors, and the matrix rotates. That’s 1.7 Complex Eigenvalues & Rotation.
Worked example: a 2×2 with distinct real eigenvalues
Find the eigenvalues and eigenvectors of .
Step 1: characteristic polynomial.
Step 2: eigenvalues. and .
Step 3: an eigenvector for each, from the kernel. For :
Both rows say the same thing (, or ). That’s no accident: we chose to make this matrix singular, so its rows must be dependent. The kernel is the line , so (or any nonzero multiple).
For :
Step 4: check. ✓ and ✓. Also and ✓.
Eigenspaces are kernels
For a given eigenvalue , the set of all solutions of is
the eigenspace of . It contains every eigenvector for , plus the zero vector. Because it’s a kernel, it’s automatically a subspace: a line, a plane, or bigger, always through the origin. Its dimension, the number of free variables when you row-reduce , is the geometric multiplicity of : how many independent eigen-directions share that eigenvalue.
This is why lesson 1.3 spent so long on kernels. Finding eigenvectors is finding a kernel, one eigenvalue at a time, using exactly the elimination you already know.
Worked example: a 3×3 triangular matrix
Find all eigenvalues and eigenvectors of
Eigenvalues. is still upper triangular, and the determinant of a triangular matrix is the product of its diagonal (expand down the first column, repeatedly):
So . For any triangular matrix, the eigenvalues are the diagonal entries.
. . Row 3 gives , then row 2 gives , and is free. So .
. . Rows 2 and 3 give , and row 1 gives . So .
. . Now is free. Row 2 gives , and row 1 gives . Taking clears the fractions: .
Check. ✓. Trace: ✓. Determinant: , the product of the diagonal ✓.
Repeated eigenvalues: when eigenvectors run short
Distinct eigenvalues always give independent eigenvectors, one line each. A repeated root is where it gets interesting. Compare two matrices that have the same characteristic polynomial, for one and for the other, with very different geometry.
The shear . It’s triangular, so twice (algebraic multiplicity 2). The eigenspace is
just the -axis: geometric multiplicity 1. Geometrically, a shear slides every horizontal row sideways by an amount proportional to its height, so only the horizontal direction stays on its line. A matrix with fewer independent eigenvectors than its size is called defective.
The scalar matrix . Here twice, and is the zero matrix, whose kernel is all of . Every nonzero vector is an eigenvector, so the geometric multiplicity is 2.
In general, for every eigenvalue,
When equality holds for every eigenvalue, the eigenvectors together form a basis of . That’s the situation 1.6 Change of Basis & Diagonalization exploits. When it fails (a defective matrix), the missing directions show up in differential equations as solutions like (7.4 Complex & Repeated Eigenvalues), and in time steppers as transient growth even when every .
Trace and determinant: free checks
The two checks we used above are theorems. If are the roots of , then . Matching coefficients with gives:
The same holds for matrices, counting repeated and complex eigenvalues: the trace is the sum of the eigenvalues and the determinant is their product. Two consequences you’ll use constantly:
- exactly when some eigenvalue is . This is the same statement as ” has a nonzero kernel”: the kernel is the eigenspace for .
- For a matrix, the pair determines both eigenvalues. In 7.3 Phase Portraits & the Trace–Determinant Plane this becomes the trace–determinant plane, a single picture that classifies every 2D linear system of ODEs.
The determinant-as-product fact also matches the geometry of lesson 1.4. If stretches one eigen-direction by and another by , then areas scale by .
Rotations: no real eigenvectors (a preview)
For a rotation by , , we get trace and det , so , which has no real roots. That matches the picture: a rotation turns every vector off its line. The roots are . Complex eigenvalues aren’t a failure. They’re the algebra’s way of saying “this matrix rotates”, and their size and angle tell you how fast it spirals. That’s the whole story of 1.7 Complex Eigenvalues & Rotation, and it’s what vortex shedding behind a cylinder looks like to a linear-stability analysis.
Iterating a matrix: eigenvalues decide growth and decay
Here’s why eigenvalues matter so much for simulation. Suppose (, say) has eigenvectors that form a basis, and you run the linear update
Write the starting state in eigenvector coordinates, . Applying multiplies each piece by its own eigenvalue, and it keeps doing so:
A tangle of coupled updates has turned into two independent scalar sequences. So:
- if every , then (decay);
- if any (and the start has a component along it), then blows up along that eigenvector;
- a negative makes that component flip sign every step. That’s the sawtooth oscillation you’ve seen in unstable simulations.
The largest is called the spectral radius . For a matrix with a full set of eigenvectors, the iteration is stable exactly when .
Euler’s method is an eigenvalue problem
Apply forward Euler to the test equation :
The number is the “eigenvalue” of the time stepper. The numerical solution decays, like the true solution for , only if . For real that means . That’s the rule you memorized in your Simulations course.
Now a system . Euler gives . If , then
so has the same eigenvectors as , with eigenvalues . Forward Euler is stable only if for every eigenvalue of . One very negative eigenvalue, a fast-decaying mode you don’t even care about, forces a tiny step for the whole simulation. That’s stiffness, which you’ll analyze properly in 4.5 Stability Regions & Stiffness.
Solutions of x’ = Ax
Differential equations get the same gift. Try a solution of the form for :
These are equal exactly when . Every eigenpair hands you a solution: a straight-line motion along that grows () or decays () exponentially. Superposing them solves every such system with a full set of eigenvectors. That’s the engine of 7.2 Linear Systems & the Eigenvalue Method.
Just above the limit, the most negative eigenvalue gives a multiplier slightly below . Its eigenvector, the zig-zag grid mode, grows by a few percent per step and swamps everything after a few hundred steps. That’s the checkerboard blow-up every fluid programmer has seen.
Power iteration
The expansion also gives us an algorithm. Suppose one eigenvalue is strictly largest in size, . Factor out :
Every ratio , so those terms die off, and the direction of converges to , the dominant eigenvector. (We need . A random start has that with probability 1.) To keep the numbers from overflowing or underflowing, rescale to unit length each step:
You watched this in the first video, and you can run it in the sim with Run 20 steps: with the General preset ( and , ratio ) it locks on within a few steps. With the Rotation preset it wanders forever, because there is no dominant real eigenvector. With the Reflection preset (, equal in size) it bounces between two directions and never settles.
Power iteration only needs matrix–vector products, so it works on matrices far too big to store densely. That’s why it, and its descendants (Lanczos, Arnoldi), are used to estimate the largest eigenvalue of a discretized PDE operator before picking a time step, and to rank web pages (PageRank is power iteration on the web’s link matrix). Library routines like np.linalg.eig use the more sophisticated QR algorithm, which is power iteration applied to many vectors at once.
Now implement it yourself, along with the trace–determinant shortcut for eigenvalues.
Practice problems
Find the eigenvalues and eigenvectors of . Verify the trace and determinant checks, and describe what does geometrically.
Show worked solution
and , so , and .
- : gives , so . Check: ✓.
- : gives , so . Check: ✓.
Checks: ✓ and ✓.
Geometrically, stretches the diagonal by a factor of 3 and flips the anti-diagonal . The negative determinant confirms that orientation is reversed. Notice that the two eigenvectors are perpendicular. That’s always true for symmetric matrices (the spectral theorem, 1.6 Change of Basis & Diagonalization).
Let .
- Find the eigenvalues and their algebraic multiplicities.
- Find each eigenspace and its geometric multiplicity.
- Is defective? Can its eigenvectors form a basis of ?
Show worked solution
-
is lower triangular, so the eigenvalues are its diagonal entries: (algebraic multiplicity 2) and (multiplicity 1). Indeed .
-
: . Row 2 gives and row 3 gives , while is free. So , geometric multiplicity 1.
: . Row 1 gives , then row 2 gives , and is free. So , geometric multiplicity 1.
-
Yes, it’s defective: has algebraic multiplicity 2 but geometric multiplicity 1. Altogether there are only two independent eigenvectors, and , which can’t span . The top-left block is a shear (in the -plane, scaled by 2), and shears are the classic defective matrices.
Two connected rooms exchange heat according to with .
- Find the eigenvalues and eigenvectors of , and write down two solutions of the form .
- Forward Euler gives . What is the largest step for which it is stable?
- Describe what the numerical solution does with .
Show worked solution
-
and , so , and .
- : gives . This is the mode where both rooms cool together.
- : gives . This is the mode where a temperature difference evens out, three times faster.
Solutions: and . Both decay.
-
has eigenvalues and . We need , so , and , so . The stricter condition wins: . The fast mode sets the limit, even though it is the one that disappears first in the true solution.
-
At the multipliers are and . The component decays nicely, but the component is multiplied by every step: the two room temperatures see-saw around each other with growing amplitude, and the simulation blows up. That’s the signature of an eigenvalue just outside the stability region.
In the simple shear flow , where layers of fluid slide past each other like cards in a deck, the strain-rate tensor is
Find its eigenvalues and eigenvectors. A small circular blob of dye is released in the flow. Along which directions is it stretched and squeezed in the first instant?
Show worked solution
and , so and .
- : gives .
- : gives .
So the principal strain directions are the two diagonals at . The blob is initially stretched along at rate and squeezed along at the same rate. The trace is zero, so its area is unchanged, as it must be in an incompressible flow. (The rest of the motion is a rotation, which is why the ellipse also turns over time.)
Suppose with .
- Show that , and more generally .
- Show that for any number .
- Show that is invertible if and only if is not an eigenvalue, and that in that case .
- Backward Euler for is . Use parts 2 and 3 to find the eigenvalues of , and explain why backward Euler is stable for every when all eigenvalues of are real and negative.
Show worked solution
- . Repeating the argument (induction) gives . Same eigenvector, eigenvalue raised to the -th power. This is the formula behind .
- . Shifting a matrix by shifts every eigenvalue by and keeps the eigenvectors. (Euler’s is this idea combined with scaling by .)
- is an eigenvalue some has is not invertible (lesson 1.3). If is invertible, then , and applying to gives , so .
- By part 2, has eigenvalues with the same eigenvectors. When and , , which is never zero, so the matrix is invertible. By part 3, has eigenvalues , which lie strictly between 0 and 1 for every . All components decay, whatever the step size. That’s why implicit methods tame stiff problems, and it’s the same reason Stam’s implicit diffusion step never blows up.
Key Takeaways
- An eigenvector of is a nonzero vector whose direction doesn’t change: . The eigenvalue is the stretch factor along that line. stretches, shrinks, flips, and crushes the line into the kernel.
- . A nonzero solution needs a nontrivial kernel, so the eigenvalues are exactly the roots of the characteristic polynomial .
- Each eigenspace is a kernel, , found by ordinary row reduction. Triangular matrices have their eigenvalues on the diagonal.
- A repeated eigenvalue may come with a full eigenspace (: every vector) or only one line (a shear: defective). Geometric multiplicity is always at most the algebraic multiplicity.
- Trace = sum and determinant = product of the eigenvalues. For a matrix, . A negative discriminant means complex eigenvalues and rotation.
- For , each eigen-component is multiplied by every step, so eigenvalues decide growth or decay. Forward Euler’s multiplier makes stability, stiffness and the CFL condition all eigenvalue conditions.
- Power iteration (multiply, normalize, repeat) converges to the dominant eigenvector at rate , and the Rayleigh quotient estimates .
- For , every eigenpair gives a solution . Next, 1.6 Change of Basis & Diagonalization uses a whole basis of eigenvectors to turn a matrix into pure scaling.