Part 4: Eigenvalues and Eigenvectors

• Download English-US transcript (PDF)

GILBERT STRANG: Moving now to the second half of linear algebra. It's about eigenvalues and eigenvectors. The first half, I just had a matrix. I solved equations. The second half, you'll see the point of eigenvalues and eigenvectors as a new way to look deeper into the matrix to see what's important there.

OK, so what are they? This is a big equation, S time x. So S is our matrix. And I've called it S because I'm taking it to be a symmetric matrix. What's on one side of the diagonal is also on the other side of the diagonal. So those have the beautiful properties. Those are the kings of linear algebra.

Now, about eigenvectors x and eigenvalues lambda. So what does that equation, Sx equal lambda x, tell me? That says that I have a special vector x. When I multiply it by S, my matrix, I stay in the same direction as the original x. It might get multiplied by 2. Lambda could be 2. It might get multiplied by 0. Lambda there could even be 0. It might get multiplied by minus 2, whatever.

But it's along the same line. So that's like taking a matrix and discovering inside it something that stays on a line. That means that it's really a sort of one dimensional problem if we're looking along that eigenvector. And that makes computations infinitely easier. The hard part of a matrix is all the connections between different rows and columns. So eigenvectors are the guys that stay in that same direction.

And y is another eigenvector. It has its own eigenvalue. It got multiplied by alpha where Sx multiplied the x by some other number lambda. So there's our couple of eigenvectors. And the beautiful fact is that because S is symmetric, those two eigenvectors are perpendicular. They are orthogonal, as it says up there. So symmetric matrices are really the best because their eigenvectors are perpendicular. And we have a bunch of one dimensional problems.

And here, I've included a proof. You want a proof that the eigenvectors are perpendicular? So what does perpendicular mean? It means that x transpose times y, the dot product is 0. The angle is 90 degrees. The cosine is 1. OK.

How to show the cosine might be there. How to show that? Yeah, proof. This is just you can tune out for two minutes if you hate proofs. OK, I start with what I know.

What I know is in that box. Sx is lambda x. That's one eigenvector. That tells me the eigenvector y. This tells me the eigenvalues are different. And that tells me the matrix is symmetric. I'm just going to juggle those four facts. And I'll end up with x transpose y equals 0. That's orthogonality. OK.

So I'll just do it quickly, too quickly. So I take this first thing, and I transpose it, turn it into row vectors. And then when I transpose it, that transpose means I flip rows and columns. But for as symmetric matrix, no different. So S transpose is the same as S.

And then I look at this one, and I multiply that by x transpose, both sides by x transpose. And what I end up with is recognizing that lambda times that dot product equals alpha times that dot product. But lambda is different from alpha. So the only way lambda times that number could equal alpha times that number is that number has to be 0. And that's the answer. OK, so that's the proof that used exactly every fact we knew. End of proof. Main point to remember, eigenvectors are perpendicular when the matrix is symmetric. OK.

In that case, now, you always want to express these facts as from multiplying matrices. That says everything in a few symbols where I had to use all those words on the previous slide. So that's the result that I'm shooting for, that a symmetric matrix-- just focus on that box.

A symmetric matrix can be broken up into its eigenvectors. Those are in Q. Its eigenvalues. Those are the lambdas. Those are the numbers lambda 1 to lambda n on the diagonal of lambda. And then the transpose, so the eigenvectors are now rows in Q transpose.

That's just perfect. Perfect. Every symmetric matrix is an orthogonal matrix times a diagonal matrix times the transpose of the orthogonal matrix. Yeah, that's called the spectral theorem. And you could say it's up there with the most important facts in linear algebra and in wider mathematics. Yeah, so that's the fact that controls what we do here.

Oh, now I have to say what's the situation if the matrix is not symmetric. Now I am not going to get perpendicular eigenvectors. That was a symmetric thing mostly. But I'll get eigenvectors.

So I'll get Ax equal lambda x. The first one won't be perpendicular to the second one. The matrix A, it has to be square, or this doesn't make sense. So eigenvalues and eigenvectors are the way to break up a square matrix and find this diagonal matrix lambda with the eigenvalues, lambda 1, lambda 2, to lambda n. That's the purpose.

And eigenvectors are perpendicular when it's a symmetric matrix. Otherwise, I just have x and its inverse matrix but no symmetry. OK. So that's the quick expression, another factorization of eigenvalues in lambda. Diagonal, just numbers. And eigenvectors in the columns of x.

And now I'm not going to-- oh, I was going to say I'm not going to solve all the problems of applied math. But that's what these are for. Let's just see what's special here about these eigenvectors.

Suppose I multiply again by A. I Start with Ax equal lambda x. Now I'm going to multiply both sides by A. That'll tell me something about eigenvalues of A squared. Because when I multiply by A-- so let me start with A squared now times x, which means A times Ax. A times Ax. But Ax is lambda x. So I have A times lambda x.

And I pull out that number lambda. And I still have a 1Ax. And that's also still lambda x. You see I'm just talking around in a little circle here, just using Ax equal lambda x a couple of times.

And the result is-- do you see what that means, that result? That means that the eigenvalue for A squared, same eigenvector x. The eigenvalue is lambda squared. And if I add A cubed, the eigenvalue would come out lambda cubed. And if I have a to the-- yeah, yeah. So if I had A to the n times, n multiplies-- so when would you have A to a high power?

That's a interesting matrix. Take a matrix and square it, cube it, take high powers of it. The eigenvectors don't change. That's the great thing. That's the whole point of eigenvectors. They don't change. And the eigenvalues just get taken to the high power.

So for example, we could ask the question, when, if I multiply a matrix by itself over and over and over again, when do I approach 0? Well, if these numbers are below 1. So eigenvectors, eigenvalues gives you something that you just could not see by those column operations or L times U. This is looking deeper. OK.

And OK, and then you'll see we have almost already seen with least squares, this combination A transpose A. So remember A is a rectangular matrix, m by n. I multiply it by its transpose. When I transpose it, I have n by m. And when I multiply them together, I get n by n. So A transpose A is, for theory, is a great matrix, A transpose times A. It's symmetric.

Yeah, let's just see what we have about A. It's square for sure. Oh, yeah. This tells me that it's symmetric. And you remember why. I'm always looking for symmetric matrices because they have those orthogonal eigenvectors. They're the beautiful ones for eigenvectors.

And A transpose A, automatically symmetric. You just you're multiplying something by its adjoint, its transpose, and the result is that this matrix is symmetric. And maybe there's even more about A transpose A. Yes. What is that?

Here is a final-- I always say certain matrices are important, but these are the winners. They are symmetric matrices. If I want beautiful matrices, make them symmetric and make the eigenvalues positive. Or non-negative allows 0. So I can either say positive definite when the eigenvalues are positive, or I can say non-negative, which allows 0. And so I have greater than or equal to 0.

I just want to say that bringing all the pieces of linear algebra come together in these matrices. And we're seeing the eigenvalue part of it. And here, I've mentioned something called the energy. So that's a physical quantity that also is greater or equal to 0.

So that's A transpose A is the matrix that I'm going to use in the final part of this video to achieve the greatest factorization. Q lambda, Q transpose was fantastic. But for a non-square matrix, it's not. For a non-square matrix, they don't even have eigenvalues and eigenvectors.

But data comes in non-square matrices. Data is about like we have a bunch of diseases and a bunch of patients or a bunch of medicines. And the number of medicines is not equal the number of patients or diseases. Those are different numbers.

So the matrices that we see in data are rectangular. And eigenvalues don't make sense for those. And singular values take the place of eigenvalues. So singular values, and my hope is that linear algebra courses, 18.06 for sure, will always reach, after you explain eigenvalues that everybody agrees is important, get singular values into the course because they really have come on as the big things to do in data. So that would be the last part of this summary video for 2020 vision of linear algebra is to get singular values in there. OK, that's coming next.

Free Downloads

Video

• Internet Archive (MP4 - 32NB)

Caption

• English-US (SRT)