Lecture 12A: Neural Nets

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at [email protected].

PATRICK WINSTON: It was in 2010, yes, that's right. It was in 2010. We were having our annual discussion about what we would dump fro 6034 in order to make room for some other stuff. And we almost killed off neural nets. That might seem strange because our heads are stuffed with neurons. If you open up your skull and pluck them all out, you don't think anymore. So it would seem that neural nets would be a fundamental and unassailable topic.

But many of us felt that the neural models of the day weren't much in the way of faithful models of what actually goes on inside our heads. And besides that, nobody had ever made a neural net that was worth a darn for doing anything. So we almost killed it off. But then we said, well, everybody would feel cheated if they take a course in artificial intelligence, don't learn anything about neural nets, and then they'll go off and invent them themselves. And they'll waste all sorts of time. So we kept the subject in.

Then two years later, Jeff Hinton from the University of Toronto stunned the world with some neural network he had done on recognizing and classifying pictures. And he published a paper from which I am now going to show you a couple of examples. Jeff's neural net, by the way, had 60 million parameters in it. And its purpose was to determine which of 1,000 categories best characterized a picture.

So there it is. There's a sample of things that the Toronto neural net was able to recognize or make mistakes on. I'm going to blow that up a little bit. I think I'm going to look particularly at the example labeled container ship. So what you see here is that the program returned its best estimate of what it was ranked, first five, according to the likelihood, probability, or the certainty that it felt that a particular class was characteristic of the picture.

And so you can see this one is extremely confident that it's a container ship. It also was fairly moved by the idea that it might be a lifeboat. Now, I'm not sure about you, but I don't think this looks much like a lifeboat. But it does look like a container ship. So if I look at only the best choice, it looks pretty good.

Here are the other things they did pretty well, got the right answer is the first choice-- is this first choice. So over on the left, you see that it's decided that the picture is a picture of a mite. The mite is not anywhere near the center of the picture, but somehow it managed to find it-- the container ship again. There is a motor scooter, a couple of people sitting on it. But it correctly characterized the picture as a motor scooter. And then on the right, a Leopard. And everything else is a cat of some sort. So it seems to be doing pretty well. In fact, it does do pretty well.

But anyone who does this kind of work has an obligation to show you some of the stuff that doesn't work so well on or doesn't get quite right. And so these pictures also occurred in Hinton's paper. So the first one is characterized as a grill. But the right answer was supposed to be convertible. Oh, no, yes, yeah, right answer was convertible. In the second case, the characterization is of a mushroom. And the alleged right answer is agaric. Is that pronounced right? It turns out that's a kind of mushroom-- so no problem there.

In the next case, it said it was a cherry. But it was supposed to be a dalmatian. Now, I think a dalmatian is a perfectly legitimate answer for that particular picture-- so hard to fault it for that. And the last case, the correct answer was not in any of the top five. I'm not sure if you've ever seen a Madagascar cap. But that's a picture of one. And it's interesting to compare that with the first choice of the program, the squirrel monkey. This is the two side by side.

So in a way, it's not surprising that it thought that the Madagascar cat was a picture of a squirrel monkey-- so pretty impressive. It blew away the competition. It did so much better the second place wasn't even close. And for the first time, it demonstrated that a neural net could actually do something. And since that time, in the three years since that time, there's been an enormous amount of effort put into neural net technology, which some say is the answer.

So what we're going to do today and tomorrow is have a look at this stuff and ask ourselves why it works, when it might not work, what needs to be done, what has been done, and all those kinds of questions will emerge.

So I guess the first thing to do is think about what it is that we are being inspired by. We're being inspired by those things that are inside our head-- all 10 to the 11th of them. And so if we take one of those 10 to the 11th and look at it, you know from 700 something or other approximately what a neuron looks like. And by the way, I'm going to teach you in this lecture how to answer questions about neurobiology with an 80% probability that you will give the same answer as a neurobiologist. So let's go.

So here's a neuron. It's got a cell body. And there is a nucleus. And then out here is a long thingamajigger which divides maybe a little bit, but not much. And we call that the axon.

So then over here, we've got this much more branching type of structure that looks maybe a little bit like so. Maybe like that-- and this stuff branches a whole lot. And that part is called the dendritic tree.

Now, there are a couple of things we can note about this is that these guys are connected axon to dendrite. So over here, they'll be a so-called pre-synaptic thickening. And over here will be some other neuron's dendrite. And likewise, over here some other neuron's axon is coming in here and hitting the dendrite of our the one that occupies most of our picture.

So if there is enough stimulation from this side in the axonal tree, or the dendritic tree, then a spike will go down that axon. It acts like a transmission line. And then after that happens, the neuron will go quiet for a while as it's recovering its strength. That's called the refractory period.

Now, if we look at that connection in a little more detail, this little piece right here sort of looks like this. Here's the axon coming in. It's got a whole bunch of little vesicles in it. And then there's a dendrite over here. And when the axon is stimulated, it dumps all these vesicles into this inner synaptic space. For a long time, it wasn't known whether those things were actually separated. I think it was Raamon and Cahal who demonstrated that one neuron is actually not part of the next one. They're actually separated by these synaptic gaps.

So there it is. How can we model, that sort of thing? Well, here's what's usually done. Here's what is done in the neural net literature. First of all, we've got some kind of binary input, because these things either fire or they don't fire. So it's an all-or-none kind of situation. So over here, we have some kind of input value. We'll call it x1. And is either a 0 or 1.

So it comes in here. And then it gets multiplied times some kind of weight. We'll call it w1. So this part here is modeling this synaptic connection. It may be more or less strong. And if it's more strong, this weight goes up. And if it's less strong, this weight goes down. So that reflects the influence of the synapse on whether or not the whole axon decides it's stimulated.

Then we got other inputs down here-- x sub n, also 0 or 1. It's also multiplied by a weight. We'll call that w sub n. And now, we have to somehow represent the way in which these inputs are collected together-- how they have collective force. And we're going to model that very, very simply just by saying, OK, we'll run it through a summer like so.

But then we have to decide if the collective influence of all those inputs is sufficient to make the neuron fire. So we're going to do that by running this guy through a threshold box like so. Here is what the box looks like in terms of the relationship between input and the output. And what you can see here is that nothing happens until the input exceeds some threshold t.

If that happens, then the output z is a 1. Otherwise, it's a 0. So binary, binary out-- we model the synaptic weights by these multipliers. We model the cumulative effect of all that input to the neuron by a summer. We decide if it's going to be an all-or-none 1 by running it through this threshold box and seeing if the sum of the products add up to more than the threshold. If so, we get a 1.

So what, in the end, are we in fact modeling? Well, with this model, we have number 1, all or none-- number 2, cumulative influence-- number 3, oh, I, suppose synaptic weight. But that's not all that there might be to model in a real neuron. We might want to deal with the refractory period.

In these biological models that we build neural nets out of, we might want to model axonal bifurcation. We do get some division in the axon of the neuron. And it turns out that that pulse will either go down one branch or the other. And which branch it goes down depends on electrical activity in the vicinity of the division. So these things might actually be a fantastic coincidence detectors. But we're not modeling that. We don't know how it works. So axonal bifurcation might be modeled.

We might also have a look at time patterns. See, what we don't know is we don't know if the timing of the arrival of these pulses in the dendritic tree has anything to do with what that neuron is going to recognize-- so a lot of unknowns here. And now, I'm going to show you how to answer a question about neurobiology with 80% probability you'll get it right. Just say, we don't know. And that will be with 80% probability what the neurobiologist would say.

So this is a model inspired by what goes on in our heads. But it's far from clear if what we're modeling is the essence of why those guys make possible what we can do. Nevertheless, that's where we're going to start. That's where we're going to go. So we've got this model of what a neuron does.

So what about what does a collection of these neurons do? Well, we can think of your skull as a big box full of neurons. Maybe a better way to think of this is that your head is full of neurons. And they in turn are full of weights and thresholds like so. So into this box come a variety of inputs x1 through xm. And these find their way to the inside of this gaggle of neurons. And out here come a bunch of outputs c1 through zn. And there a whole bunch of these maybe like so. And there are a lot of inputs like so.

And somehow these inputs through the influence of the weights of the thresholds come out as a set of outputs. So we can write that down a little fancier by just saying that z is a vector, which is a function of, certainly the input vector, but also the weight vector and the threshold vector. So that's all a neural net is. And when we train a neural net, all we're going to be able to do is adjust those weights and thresholds so that what we get out is what we want. So a neural net is a function approximator. It's good to think about that. It's a function approximator.

So maybe we've got some sample data that gives us an output vector that's desired as another function of the input, forgetting about what the weights and the thresholds are. That's what we want to get out. And so how well we're doing can be figured out by comparing the desired value with the actual value.

So we might think then that we can get a handle on how well we're doing by constructing some performance function, which is determined by the desired vector and the input vector-- sorry, the desired vector and the actual output vector for some particular input or for some set of inputs. And the question is what should that function be? How should we measure performance given that we have what we want out here and what we actually got out here?

Well, one simple thing to do is just to measure the magnitude of the difference. That makes sense. But of course, that would give us a performance function that is a function of the distance between those vectors would look like this. But this turns out to be mathematically inconvenient in the end. So how do you think we're going to turn it up a little bit?

AUDIENCE: Normalize it?

PATRICK WINSTON: What's that?

AUDIENCE: Normalize it?

PATRICK WINSTON: Well, I don't know. How about just we square it? And that way we're going to go from this little sharp point down there to something that looks more like that. So it's best when the difference is 0, of course. And it gets worse as you move away from 0. But what we're trying to do here is we're trying to get to a minimum value. And I hope you'll forgive me. I just don't like the direction we're going here, because I like to think in terms of improvement as going uphill instead of down hill.

So I'm going to dress this up one more step-- put a minus sign out there. And then our performance function looks like this. It's always negative. And the best value it can possibly be is zero. So that's what we're going to use just because I am who I am. And it doesn't matter, right? Still, you're trying to either minimize or maximize some performance function.

OK, so what do we got to do? I guess what we could do is we could treat this thing-- well, we already know what to do. I'm not even sure why we're devoting our lecture to this, because it's clear that what we're trying to do is we're trying to take our weights and our thresholds and adjust them so as to maximize performance. So we can make a little contour map here with a simple neural net with just two weights in it. And maybe it looks like this-- contour map.

And at any given time we've got a particular w1 and particular w2. And we're trying to find a better w1 and w2. So here we are right now. And there's the contour map. And it's a 6034. So what do we do?

AUDIENCE: Climb.

PATRICK WINSTON: Simple matter of hill climbing, right? So we'll take a step in every direction. If we take a step in that direction, not so hot. That actually goes pretty bad. These two are really ugly. Ah, but that one-- that one takes us up the hill a little bit. So we're done, except that I just mentioned that Hinton's neural net had 60 million parameters in it. So we're not going to hill climb with 60 million parameters because it explodes exponentially in the number of weights you've got to deal with-- the number of steps you can take.

So this approach is computationally intractable. Fortunately, you've all taken 1801 or the equivalent thereof. So you have a better idea. Instead of just taking a step in every direction, what we're going to do is we're going to take some partial derivatives. And we're going to see what they suggest to us in terms of how we're going to get around in space.

So we might have a partial of that performance function up there with respect to w1. And we might also take a partial derivative of that guy with respect to w2. And these will tell us how much improvement we're getting by making a little movement in those directions, right? How much a change is given that we're just going right along the axis.

So maybe what we ought to do is if this guy is much bigger than this guy, it would suggest we mostly want to move in this direction, or to put it in 1801 terms, what we're going to do is we're going to follow the gradient. And so the change in the w vector is going to equal to this partial derivative times i plus this partial derivative times j. So what we're going to end up doing in this particular case by following that formula is moving off in that direction right up to the steepest part of the hill.

And how much we move is a question. So let's just have a rate constant R that decides how big our step is going to be. And now you think we were done. Well, too bad for our side. We're not done. There's a reason why we can't use-- create ascent, or in the case that I've drawn our gradient, descent if we take the performance function the other way. Why can't we use it?

AUDIENCE: Local maxima.

PATRICK WINSTON: The remark is local maxima. And that is certainly true. But it's not our first obstacle. Why doesn't gradient ascent work?

AUDIENCE: So you're using a step function.

PATRICK WINSTON: Ah, there's something wrong with our function. That's right. It's non-linear, but rather, it's discontinuous. So gradient ascent requires a continuous space, continuous surface. So too bad our side. It isn't. So what to do? Well, nobody knew what to do for 25 years. People were screwing around with training neural nets for 25 years before Paul Werbos sadly at Harvard in 1974 gave us the answer.

And now I want to tell you what the answer is. The first part of the answer is those thresholds are annoying. They're just extra baggage to deal with. What we really like instead of c being a function of xw and t was we'd like c prime to be a function f prime of x and the weights. But we've got to account for the threshold somehow. So here's how you do that.

What you do is you say let us add another input to this neuron. And it's going to have a weight w0. And it's going to be connected to an input that's always minus 1. You with me so far? Now what we're going to do is we're going to say, let w0 equal t. What does that do to the movement of the threshold? What it does is it takes that threshold and moves it back to 0. So this little trick here takes this pink threshold and redoes it so that the new threshold box looks like this.

Think about it. If this is t, and this is minus 1, then this is minus t. And so this thing ought to fire if everything's over-- if the sum is over 0. So it makes sense. And it gets rid of the threshold thing for us. So now we can just think about weights. But still, we've got that step function there. And that's not good.

So what we're going to do is we're going to smooth that guy out. So this is trick number two. Instead of a step function, we're going to have this thing we lovingly call a sigmoid function, because it's kind of from an s-type shape. And the function we're going to use is this one-- one, well, better make it a little bit different-- 1 over 1 plus e to the minus whatever the input is. Let's call the input alpha. Does that makes sense?

Is alpha is 0, then it's 1 over 1 plus 1 plus one half. If alpha is extremely big, then even the minus alpha is extremely small. And it becomes one. It goes up to an asymptotic value of one here. On the other hand, if alpha is extremely negative, than the minus alpha is extremely positive. And it goes to 0 asymptotically.

So we got the right look to that function. It's a very convenient function. Did God say that neurons ought to be-- that threshold ought to work like that? No, God didn't say so. Who said so? The math says so. It has the right shape and look and the math. And it turns out to have the right math, as you'll see in a moment.

So let's see. Where are we? We decided that what we'd like to do is take these partial derivatives. We know that it was awkward to have those thresholds. So we got rid of them. And we noted that it was impossible to have the step function. So we got rid of it. Now, we're a situation where we can actually take those partial derivatives, and see if it gives us a way of training the neural net so as to bring the actual output into alignment with what we desire.

So to deal with that, we're going to have to work with the world's simplest neural net. Now, if we've got one neuron, it's not a net. But if we've got two-word neurons, we've got a net. And it turns out that's the world's simplest neuron. So we're going to look at it-- not 60 million parameters, but just a few, actually, just two parameters.

So let's draw it out. We've got input x. That goes into a multiplier. And it gets multiplied times w1. And that goes into a sigmoid box like so. We'll call this p1, by the way, product number one. Out here comes y. Y gets multiplied times another weight. We'll call that w2. The neck produces another product which we'll call p2. And that goes into a sigmoid box.

And then that comes out as z. And z is the number that we use to determine how well we're doing. And our performance function p is going to be one half minus one half, because I like things are going in a direction, times the difference between the desired output and the actual output squared.

So now let's decide what those partial derivatives are going to be. Let me do it over here. So what are we trying to compute? Partial of the performance function p with respect to w2. OK. Well, let's see. We're trying to figure out how much this wiggles when we wiggle that.

But you know it goes through this variable p2. And so maybe what we could do is figure out how much this wiggles-- how much z wiggles when we wiggle p2 and then how much p2 wiggles when we wiggle w2. I just multiplied those together. I forget. What's that called? N180-- something or other.

AUDIENCE: The chain rule

PATRICK WINSTON: The chain rule. So what we're going to do is we're going to rewrite that partial derivative using chain rule. And all it's doing is saying that there's an intermediate variable. And we can compute how much that end wiggles with respect how much that end wiggles by multiplying how much the other guys wiggle. Let me write it down. It makes more sense in mathematics.

So that's going to be able to the partial of p with respect to z times the partial of z with respect to p2. Keep me on track here. Partial of z with respect to w2. Now, I'm going to do something for which I will hate myself. I'm going to erase something on the board. I don't like to do that. But you know what I'm going to do, don't you?

I'm going to say this is true by the chain rule. But look, I can take this guy here and screw around with it with the chain rule too. And in fact, what I'm going to do is I'm going to replace that with partial of z with respect to p2 and partial of p2 with respect to w2. So I didn't erase it after all. But you can see what I'm going to do next. Now, I'm going to do same thing with the other partial derivative. But this time, instead of writing down and writing over, I'm just going to expand it all out in one go, I think.

So partial of p with respect to w1 is equal to the partial of p with respect to z, the partial of z with respect to p2, the partial of p2 with respect to what? Y? Partial of y with respect to p1-- partial of p1 with respect to w1. So that's going like a zipper down that string of variables expanding each by using the chain rule until we got to the end. So there are some expressions that provide those partial derivatives.

But now, if you'll forgive me, it was convenient to write them out that way. That matched the intuition in my head. But I'm just going to turn them around. It's just a product. I'm just going to turn them around. So partial p2, partial w2, times partial of z, partial p2, times the partial of p with respect to z-- same thing. And now, this one. Keep me on track, because if there's a mutation here, it will be fatal.

Partial of p1-- partial of w1, partial of y, partial p1, partial of p2, partial of y, partial of z. There's a partial of p2, partial of a performance function with respect to z. Now, all we have to do is figure out what those partials are. And we have solved this simple neural net. So it's going to be easy.

Where is my board space? Let's see, partial of p2 with respect to-- what? That's the product. The partial of z-- the performance function with respect to z. Oh, now I can see why I wrote it down this way. Let's see. It's going to be d minus e. We can do that one in our head.

What about the partial of p2 with respect to w2. Well, p2 is equal to y times w2, so that's easy. That's just y. Now, all we have to do is figure out the partial of z with respect to p2. Oh, crap, it's going through this threshold box. So I don't know exactly what that partial derivative is. So we'll have to figure that out, right? Because the function relating them is this guy here.

And so we have to figure out the partial of that with respect to alpha. All right, so we got to do it. There's no way around it. So we have to destroy something. OK, we're going to destroy our neuron. So the function we're dealing with is, we'll call it beta, equal to 1 over 1 plus e to the minus alpha. And what we want is the derivative with respect to alpha of beta. And that's equal to d by d alpha of-- you know, I can never remember those quotient formulas.

So I am going to rewrite it a little different way. I am going to write it as 1 minus e to the minus alpha to the minus 1, because I can't remember the formula for differentiating a quotient. OK, so let's differentiate it. So that's equal to 1 minus e to the minus alpha to the minus 2. And we got that minus comes out of that part of it. Then we got to differentiate the inside of that expression.

And when we differentiate the inside of that expression, we get e to the minus alpha.

AUDIENCE: Dr. Winston--

PATRICK WINSTON: Yeah?

AUDIENCE: That should be 1 plus.

PATRICK WINSTON: Oh, sorry, thank you. That was one of those fatal mistakes you just prevented. So that's 1 plus. That's 1 plus here too. OK, so we've differentiated that. We've turned that into a minus 2. We brought the minus sign outside. Then we're differentiating the inside. The derivative and the exponential is an exponential. Then we got to differentiate that guy. And that just helps us get rid of the minus sign we introduced. So that's the derivative.

I'm not sure how much that helps except that I'm going to perform a parlor trick here and rewrite that expression thusly. We want to say that's going to be e to the minus alpha over 1 plus e to the minus alpha times 1 over 1 plus e to the minus alpha. That OK? I've got a lot of nodding heads here. So I think I'm on safe ground.

But now, I'm going to perform another parlor trick. I am going to add 1, which means I also have to subtract 1. All right? That's legitimate isn't it? So now, I can rewrite this as 1 plus e to the minus alpha over 1 plus e to the minus alpha minus 1 over 1 plus e to the minus alpha times 1 over 1 plus e to the minus alpha. Any high school kid could do that. I think I'm on safe ground.

Oh, wait, this is beta. This is beta.

AUDIENCE: That's the wrong side.

PATRICK WINSTON: Oh, sorry, wrong side. Better make this beta and this 1. Any high school kid could do it. OK, so what we've got then is that this is equal to 1 minus beta times beta. That's the derivative. And that's weird because the derivative of the output with respect to the input is given exclusively in terms of the output. It's strange. It doesn't really matter. But it's a curiosity.

And what we get out of this is that partial derivative there-- that's equal to well, the output is p2. No, the output is z. So it's z time 1 minus e. So whenever we see the derivative of one of these sigmoids with respect to its input, we can just write the output times one minus alpha, and we've got it. So that's why it's mathematically convenient. It's mathematically convenient because when we do this differentiation, we get a very simple expression in terms of the output.

We get a very simple expression. That's all we really need. So would you like to see a demonstration? It's a demonstration of the world's smallest neural net in action. Where is neural nets? Here we go.

So there's our neural net. And what we're going to do is we're going to train it to do absolutely nothing. What we're going to do is train it to make the output the same as the input. Not what I'd call a fantastic leap of intelligence. But let's see what happens.

Wow! Nothing's happening. Well, it finally got to the point where the maximum error, not the performance, but the maximum error went below a threshold that I had previously determined. So if you look at the input here and compare that with the desired output on the far right, you see it produces an output, which compared with the desired output, is pretty close.

So we can test the other way like so. And we can see that the desired output is pretty close to the actual output in that case too. And it took 694 iterations to get that done. Let's try it again.

To 823-- of course, this is all a consequence of just starting off with random weights. By the way, if you started with all the weights being the same, what would happen? Nothing because it would always stay the same. So you've got to put some randomization in in the beginning. So it took a long time. Maybe the problem is our rate constant is too small. So let's crank up the rate counts a little bit and see what happens. That was pretty fast. Let's see if it was a consequence of random chance. Run. No, it's pretty fast there-- 57 iterations-- third try-- 67. So it looks like at my initial rate constant was too small.

So if 0.5 was not as good as 5.0, why don't we crank it up to 50 and see what happens. Oh, in this case, 124-- let's try it again. Ah, in this case 117-- so it's actually gotten worse. And not only has it gotten worse. You'll see there's a little a bit of instability showing up as it courses along its way toward a solution. So what it looks like is that if you've got a rate constant that's too small, it takes forever. If you've get a rate constant that's too big, it can of jump too far, as in my diagram which is somewhere underneath the board, you can go all the way across the hill and get to the other side.

So you have to be careful about the rate constant. So what you really want to do is you want your rate constant to vary with what is happening as you progress toward an optimal performance. So if your performance is going down when you make the jump, you know you've got a rate constant that's too big. If your performance is going up when you make a jump, maybe you want to increase-- bump it up a little bit until it doesn't look so good. So is that all there is to it? Well, not quite, because this is the world's simplest neural net. And maybe we ought to look at the world's second simplest neural net.

Now, let's call this-- well, let's call this x. What we're going to do is we're going to have a second input. And I don't know. Maybe this is screwy. I'm just going to use color coding here to differentiate between the two inputs and the stuff they go through. Maybe I'll call this z2 and this z1 and this x1 and x2.

Now, if I do that-- if I've got two inputs and two outputs, then my performance function is going to have two numbers in it-- the two desired values and the two actual values. And I'm going to have two inputs. But it's the same stuff. I just repeat what I did in white, only I make it orange.

Oh, but what happens if-- what happens if I do this? Say put little cross connections in there. So these two streams are going to interact. And then there might be some-- this y can go into another multiplier here and go into a summer here. And likewise, this y can go up here and into a multiplier like so. And there are weights all over the place like so.

This guy goes up in here. And now what happens? Now, we've got a disaster on our hands, because there are all kinds of paths through this network. And you can imagine that if this was not just two neurons deep, but three neurons deep, what I would find is expressions that look like that. But you could go this way, and then down through, and out here. Or you could go this way and then back up through here. So it looks like there is an exponentially growing number of paths through that network. And so we're back to an exponential blowup. And it won't work.

Yeah, it won't work except that we need to let the math sing to us a little bit. And we need to look at the picture. And the reason I turned this guy around was actually because from a point of view of letting the math sing to us, this piece here is the same as this piece here. So part of what we needed to do to calculate the partial derivative with respect to w1 has already been done when we calculated the partial derivative with respect to w2.

And not only that, if we calculated the partial wit respect to these green w's at both levels, what we would discover is that sort of repetition occurs over and over again. And now, I'm going to try to give you an intuitive idea of what's going on here rather than just write down the math and salute it. And here's a way to think about it from an intuitive point of view.

Whatever happens to this performance function that's back of these p's here, the stuff over there can influence p only by going through, and influence performance only going through this column of p's. And there's a fixed number of those. So it depends on the width, not the depth of the network. So the influence of that stuff back there on p is going to end up going through these guys. And it's going to end up being so that we're going to discover that a lot of what we need to compute in one column has already been computed in the column on the right.

So it isn't going to explode exponentially, because the influence-- let me say it one more time. The influences of changes of changes in p on the performance is all we care about when we come back to this part of the network, because this stuff cannot influence the performance except by going through this column of p's. So it's not going to blow up exponentially. We're going to be able to reuse a lot of the computation. So it's the reuse principle.

Have we ever seen the reuse principle at work before. Not exactly. But you remember that little business about the extended list? We know that we've seen-- we know we've seen something before. So we can stop computing. It's like that. We're going to be able to reuse the computation. We've already done it to prevent an exponential blowup. By the way, for those of you who know about fast Fourier transform-- same kind of idea-- reuse of partial results.

So in the end, what can we say about this stuff? In the end, what we can say is that it's linear in depth. That is to say if we increase the number of layers to so-called depth, then we're going to increase the amount of computation necessary in a linear way, because the computation we need in any column is going to be fixed. What about how it goes with respect to the width?

Well, with respect to the width, any neuron here can be connected to any neuron in the next row. So the amount of work we're going to have to do will be proportional to the number of connections. So with respect to width, it's going to be w-squared. But the fact is that in the end, this stuff is readily computed. And this, phenomenally enough, was overlooked for 25 years. So what is it in the end? In the end, it's an extremely simple idea. All great ideas are simple. How come there aren't more of them? Well, because frequently, that simplicity involves finding a couple of tricks and making a couple of observations.

So usually, we humans are hardly ever go beyond one trick or one observation. But if you cascade a few together, sometimes something miraculous falls out that looks in retrospect extremely simple. So that's why we got the reuse principle at work-- and our reuse computation. In this case, the miracle was a consequence of two tricks plus an observation. And the overall idea is all great ideas are simple and easy to overlook for a quarter century.

Free Downloads

Video

• iTunes U (MP4 - 113MB)
• Internet Archive (MP4 - 113MB)

Caption

• English-US (SRT)