Lecture 6: Probability Part 2

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: We'll begin this time by looking at some probability distribution that you should be familiar with from this perspective, starting with a Gaussian distribution for one variable.

We're focused on a variable that takes real values in the interval minus infinity to infinity and the Gaussian has the form exponential that is centered around some value, let's call it lambda, and has fluctuations around this value parameterized by sigma. And the integral of this p over the interval should be normalized to unity, giving you this hopefully very family of form.

Now, if you want to characterize the characteristic function, all we need to do is to Fourier transform this. So I have the integral dx e the minus ikx. So this-- let's remind you alternatively was the expectation value of e to the minus ikx. minus x minus lambda squared over 2 sigma squared, which is the probably distribution.

And you should know what the answer to that is, but I will remind. You can change variables to x minus lambda [INAUDIBLE] y. So from here we will get the factor of 2 to the minus ik lambda. You have then the integral over y minus y squared over 2 sigma squared.

And then what we need to do is to complete this square over here. And you can do that, essentially, by adding and subtracting a minus k squared sigma squared over 2. So that if I change variable to y plus ik sigma squared, let's call that z, then I have outside the integral e to the minus ik lambda minus k squared sigma squared over 2. And the remainder I can write as a full square.

And this is just a normalized Gaussian integral that comes to 1. So as you well know, a Fourier transform of a Gaussian is itself a Gaussian, and that's what we've established. E to the minus ik lambda minus k squared sigma squared over 2m.

And if I haven't made a mistake when I said k equals to 0, the answer should be 1 because k equals to 0 expectation value of 1 just amounts to normalization.

Now what we said was that a more interesting function is obtained by taking the log of this. So from here we go to the log of [INAUDIBLE] of k. Log of [INAUDIBLE] of k is very simple for the Gaussian.

And what we had said was that by definition this log of the characteristic function generates cumulants through the series minus ik to the power of n over n factorial, the end cumulant.

So looking at this, we can immediately see that the Guassian is characterized by a first cumulant, which is the coefficient of minus ik. It's lambda. It is characterized by a second cumulant, which is the coefficient of minus ik squared.

This, you know, is the variance. And we can explicitly see that the coefficient of minus ik squared over 2 factorial is simply sigma squared. So this is reputation.

But one thing that is interesting is that our series has now terminates, which means that if I were to look at the third cumulant, if I were to look at the fourth cumulant, and so forth, for the Guassian, they're all 0. So the Gaussian is the distribution that is completely characterized just by its first and second cumulants, all the rest being 0.

So, now our last time we developed some kind of a graphical method. We said that I can graphically describe the first cumulant as a bag with one point in it. Second cumulant with something that has two points in it. A third cumulant with three three, fourth cumulant with four and four points and so forth.

This is just rewriting. Now, the interesting thing was that we said that the various moments we could express graphically. So that, for example, the second moment is either this or this, which then graphically is the same thing as lambda squared plus sigma squared because this is indicated by sigma squared.

Now, x cubed you would say is either three things by themselves or put two of them together and then one separate. And this I could do in three different ways. And in general, for a general distribution, I would have had another term, which is a triangle. But the triangle is 0.

So for the Gaussian, this terminates here. I have lambda cubed plus 3 lambda sigma squared. If I want to calculate x to the fourth, maybe the old way of doing it would have been too multiply the Gaussian distribution against x to the fourth and try to do the integration. And you would ultimately be able to do that rearranging things and looking at the various powers of the Gaussian integrated from minus infinity to infinity.

But you can do it graphically. You can say, OK. It's either this or I can have-- well, I cannot put one aside and three together, because that doesn't exist. I could have two together and two not together. And this I can do in six different ways. You can convince yourself of that. What I could do two pairs, which I can do three different ways because I can either do one, two; one, three; one, four and then the other is satisfied.

So this is lambda to the fourth plus 6 lambda squared sigma squared plus 3 sigma to the fourth. And you can keep going and doing different things. OK. Question? Yeah?

AUDIENCE: Is the second-- [INAUDIBLE].

PROFESSOR: There?

AUDIENCE: Because-- so you said that the second cumulative--

PROFESSOR: Oh.

AUDIENCE: --x sqaured. Yes.

PROFESSOR: Yes. So that's the wrong--

AUDIENCE: [INAUDIBLE].

PROFESSOR: The coefficient of k squared is the second cumulant. The additional 2 was a mistake. OK. Anything else? All right.

Let's take a look at couple of other distributions, this time discrete. So the binomial distribution is repeat a binary random variable.

And what does this mean? It means two outcomes that's binary, let's call them A and B. And if I have a coin that's head or tails, it's binary. Two possibilities. And I can assign probabilities to the two outcomes to PA and PB, which has to be 1 minus PA.

And the question is if you repeat this binary random variables n times, what is the probability of NA outcomes of A?

And I forget to say something important, so I write it in red. This should be independent. That is the outcome of a coin toss at, say, the fifth time should not influence the sixth time and future times.

OK. So this is easy. The probability to have NA occurrences of A in N trials. So it has to be index yn. Is that within the n times that I tossed, A came up NA times. So it has to be proportional to the probability of A independently multiplied by itself NA times.

But if I have exactly NA occurrences of A, all the other times I had B occurring. So I have the probability of B for the remainder, which is N minus NA.

Now this is the probability for a specific occurrence, like the first NA times that I threw the coin I will get A. The remaining times I would get B. But the order is not important and the number of ways that I can shuffle the order and have a total of NA out of N times is the binomial factor.

Fine. Again, well known. Let's look at its characteristic function. So p tilde, which is now a function of k, is expectation value of e to the minus ik NA, which means that I have to weigh e minus ik NA against the probability of occurrences of NA times, which is this binomial factor PA to the power of NA, PB to the power of N minus NA. And of course, I have to sum over all possible values of NA that go all the way from 0 to N.

So what we have here is something which is this combination, PA e to the minus ik raised to the power of NA. PB raised to the complement N minus NA multiplied by the binomial factor summed over all possible values. So this is just the definition of the binomial expansion of PA e to the minus ik plus PB raised to the power of N.

And again, let's check. If I set k equals to 0, I have PA plus PB, which is 1 raised to the power of N. So things are OK. So this is the characteristic function.

At this stage, the only thing that I will note about this is that if I look at the characteristic function I will get N times. So this is-- actually, let's make sure that we maintain the index N. So this is the characteristic function appropriate to N trials. And what I get is that up to factor of N, I will get the characteristic function that would be appropriate to one trial.

So what that means if I were to look at powers of k, the expectation value of some cumulant, if I go to repeat things N times-- so this carries an index N. It is going to be simply N times what I would have had in a single trial.

So for a single trial, you really have two outcomes-- 0 or 1 occurrences of this object. So for a binary variable, you can really easily compute these quantities and then you can calculate the corresponding ones for N trials simply multiplying by N. And we will see that this is characteristic, essentially, of anything that is repeated N times, not just the binomial.

So this form that you have N independent objects, you would get N times what you would have for one object is generally valid and actually something that we will build a lot of statistical mechanics on because we are interested in the [INAUDIBLE]. So we will see that shortly.

But rather than following this, let's look at a third distribution that is closely related, which is the Poisson. And the question that we are asking is-- we have an interval. And the question is, what is the probability of m events in an interval from 0 to T?

And I kind of expressed it this way because prototypical Poisson distribution is, let's say, the radioactivity. And you can be waiting for some time interval from 0 to 1 minute and asking within that time interval, what's the probability that you will see m radioactive decay events?

So what is the probability if two things happen? One is that the probability of 1 and only 1 event in interval dt is alpha dt as dt goes to 0.

OK? So basically if you look at this over 1 minute, the chances are that you will see so many events, so many radioactivities. If you shorten the interval, the chances that you would see events would become less and less.

If you make your event infinitesimal, most of the time nothing would happen with very small probability that vanishes. As the size of the interval goes to 0, you will see 1 event. So this is one condition.

And the second condition is events in different intervals are independent. And since I wrote independent in red up there, let me write it in red here because it sort of harks back to the same condition.

And so this is the question. What is this probability? And the to get the answer, what we do is to subdivide our big interval into N, which is big T divided by the small dt subintervals.

So basically, originally let's say on the time axis, we were covering a distance that went from 0 to big T and we were asking what happens here. So what we are doing now is we are sort of dividing this interval to lots of subintervals, the size of each one of them being dt. And therefore, the total number is big T over dt. And ultimately, clearly, I want to sit dt going to 0 so that this condition is satisfied.

So also because of the second condition, each one of these will independently tell me whether or not I have an event. And so if I want to count the total number of events, I have to add things that are occurring in different intervals. And we can see that this problem now became identical to that problem because each one of these intervals has two possible outcomes-- nothing happens with probability 1 minus alpha dt, something happens with probability alpha dt.

So no event is probability 1 minus alpha dt. One event means probability alpha dt. So this is a binomial process.

So we can calculate, for example, the characteristic function. And I will indicate that we are looking at some interval of size T and parameterized by this straight alpha, we'll see that only the product will occur. So this is this before Fourier variable.

We said that it's a binary, so it is one of the probabilities plus e to the minus ik plus the other probability raised to the power of N. Now we just substitute the probabilities that we have over here.

So the probability of not having an event is 1 minus alpha dt. The probability of having an event is alpha dt. So alpha dt is going to appear here as e to the minus ik minus 1. So alpha st, e to the minus ik, is from here. From PB I will get 1 minus alpha dt.

And I bunched together the two terms that are proportional to alpha dt. And then I have to raise to the power of N, which is T divided by dt. And this whole prescription is valid in the limit where dt is going to 0.

So what you have is 1 plus an infinitesimal raised to a huge power. And this limiting procedure is equivalent to taking the exponential. So basically this is the same thing as exponential of what is here multiplied by what is here. The dt's cancel each other out and the answer is alpha T e to the minus ik minus 1.

So the characteristic function for this process that we described is simply given by this form. You say, wait. I didn't ask for the characteristic function. I wanted the probability.

Well, I say, OK. Characteristic function is simply the Fourier transform. So let me Fourier transform back, and I would say that the probability along not the Fourier axis but the actual axis is open by the inverse Fourier process.

So I have to do an integral dk over 2 pi e to the ikx times the characteristic function. And the characteristic function is e to minus-- what was it? e to the alpha T. E to the minus ik k minus 1.

Well, there is an equal to minus alpha T that I can simply take outside the integration. I have the integration over k e to that ikx. And then what I will do is I have this factor of e to the something in the-- e to the alpha T to the minus ik. I will use the expansion of the exponential.

So the expansion of the exponential is a sum over m running from 0 to infinity. The exponent raised to the m-th power. So I have alpha T raised to the m-th power, e to the minus ik raised to the m-th power divided m factor here.

So now I reorder the sum and the integration. The sum is over m, the integration is over k. I can reorder them. So on the things that go outside I have a sum over m running from 0 to infinity e to the minus alpha T, alpha T to the power of m divided by m factorial. Then I have the integral over k over 2 pi e to the ik.

Well, I had the x here and I have e to the minus ikm here. So I have x minus m.

And then I say, OK, this is an integral that I recognize. The integral of e to the ik times something is simply a delta function. So this whole thing is a delta function that's says, oh, x has to be an integer. Because I kind of did something that maybe, in retrospect, you would have said why are you doing this. Because along how many times things have occurred, they have either occurred 0 times, 1 times, 2 decays, 3 decays. I don't have 2.5 decays.

So I treated x as a continuous variable, but the mathematics was really clever enough to say that, no. The only places that you can have are really integer values. And the probability that you have some particular value integer m is simply what we have over here, e to the minus alpha T alpha T to the power of m divided by m factorial, which is the Poisson distribution. OK.

But fine. So this is the Poisson distribution, but really we go through the root of the characteristic function in order to use this machinery that we developed earlier for cumulants et cetera.

So let's look at the cumulant generating function. So I have to take the log of the function that I had calculated there. It is nicely in the exponential, so I get alpha T e to the minus ik minus 1.

So now I can make an expansion of this in powers of k so I can expand the exponential. The first term vanishes because this starts with one. So really I have alpha T sum running from 1 to infinity or N running from 1 to infinity minus ik to the power of n over n factorial.

So my task for identifying the cumulants is to look at the expansion of this log and read off powers of minus ikn to the power of n factorial. So what do we see? We see that the first cumulant of the Poisson is alpha T, but all the coefficients are the same thing.

The expectation value-- sorry. The second cumulant is alpha T. The third cumulant, the fourth cumulant, all the other cumulants are also alpha T.

So the average number of decays that you see in the interval is simply alpha T. But there are fluctuations, and if somebody should, for example, ask you what's the average number cubed of events, you would say, OK. I'm going to use the relationship between moments and cumulants. I can either have three first objects or I can put one of them separate in three different factions.

But this is a case where the triangle is allowed, so diagrammatically all three are possible. And so the answer for the first term is alpha T cubed. For the second term, it is a factor of 3. Both this variance and the mean give me a factor of alpha T, so I will get alpha T squared. And the third term, which is the third cumulant, is also alpha T. So the answer is simply of this form.

Again, m is an integer. Alpha T is dimensionless. So there is no dimension problem by it having different powers. OK. Any questions?

All right. So that's what I wanted to say about one variable. Now let's go and look at corresponding definitions when you have multiple variables.

So for many random variables, the set of possible outcomes, let's say, has variables x1, x2. Let's be precise. Let's end it a xn.

And if these are distributed, each one of them continuously over the interval, to each point we can characterize some kind of a probability density. So this entity is called the joint probability density function.

And its definition would be to look at probability of outcome in some interval that is between, say, x1, x1 plus dx1 in one, x2, x2 plus dx2 in the second variable. xn plus dxn in the last variable. So you sort of look at the particular point in this multi-dimensional space that you are interested.

You build a little cube around it. You ask, what's the probability to being that cube? And then you divide by the volume of that cube.

So dx1, dx2, dxn, which is the same thing that you would be doing in constructing any density and by ultimately taking the limit that all of the x's go to 0. All right. So this is the joint probability distribution.

You can construct, now, the joint characteristic function. Now how do you do that? Well, again, just like you would do for your transform with multiple variables. So you would go for each variable to a conjugate variable.

So x1 would go to k1. x2 would go to k2. xn would go to kn. And this would mathematically amount to calculating the expectation value of e to the minus i k1x1 x1, k2x2 and so forth, which you would obtain by integrating over all of these variables, e to the minus ik alpha x alpha, against the probability of x1 through xn.

Question?

AUDIENCE: It's a bit hard to read. It's getting really small.

PROFESSOR: OK.

[LAUGHTER]

PROFESSOR: But it's just multi-dimensional integral. OK? All right.

So this is, as the case of one, I think the problem is not the size but the angle I see. I can't do much for that. You have to move to the center. OK.

So what we can look at now is joint moment. So you can-- when we had one variable, we could look at something like the expectation value of x to the m. That would be the m-th moment.

But if you have two variable, we can raise x1 to some other, x2 to another power, and actually xn to another power. So this is a joint moment.

Now the thing is, that the same way that moments for one variable could be generated by expanding the characteristic function, if I were to expand this function in powers of k, you can see that in meeting the expectation value, I will get various powers of x1 to some power, x2 to some power, et cetera. So by appropriate expansion of that function, I can generate all of-- read off all of these moments.

Now, a more common way of generating the Taylor series expansion is through derivatives. So what I can do is I can take a derivative with respect to, say, ik1. If I take a derivative with respect to ik1 here, what happens is I will bring down a factor of minus x alpha. So actually let me put the minus so it becomes a factor of x alpha.

And if I integrate x alpha against this, I will be generating the expectation value of x alpha provided that ultimately I set all of the k's to 0. So I will calculate the derivative of this function with respect to all of these arguments. At the end of the day, I will set k equals to 0. That will give me the expectation value of x1.

But I don't want x1, I want x1 raised to the power of m1. So I do this. Each time I take a derivative with respect to minus ik, I will bring down the factor of the corresponding x. And I can do this with multiple different things. So d by the ik2 raised to the power of m2 minus d by the ikn, the whole thing raised to the power of mn.

So I can either take this function of multiple variables-- k1 through kn-- and expand it and read off the appropriate powers of k1k2. Or I can say that the terms in this expansion are generated through taking appropriate derivative. Yes?

AUDIENCE: Is there any reason why you're choosing to take a derivative with respect to ikj instead of simply putting the i in the numerator? Or are there-- are there things that I'm not--

PROFESSOR: No. No. There is no reason. So you're saying why didn't I write this as this like this? i divided?

AUDIENCE: Yeah.

PROFESSOR: I think I just visually saw that it was kind of more that way. But it's exactly the same thing. Yes. OK. All right.

Now the interesting object, of course, to us is more the joint cumulants. So how do we generate joint cumulants? Well previously, essentially we had a bunch of objects for one variable that was some moment. And in order to make them cumulants, we just put a sub C here. So we do that and we are done.

But what did operationally happen was that we did the expansion rather than for the characteristic function for the log of the characteristic function. So all I need to do is to do precisely this set of derivatives applied rather than to the joint characteristic function to the log of the joint characteristic function. And at the end, set all of the case to 0. OK?

So by looking at these two definitions and the expansion of the log, for example, you can calculate various things. Like, for example, x1x2 with a C is the expectation value of x1x2. This joint moment minus x1x2, just as you would have thought, would be the appropriate generalization of the variance. And this is the covariance. And you can construct appropriate extensions. OK.

Now we made a lot of use of the relationship between moments and cumulants. We just-- so the idea, really, was that the essence of a probability distribution is characterized in the cumulants. Moments kind of depend on how you look at things.

The essence is in the cumulants, but sometimes the moments are more usefully computed, and there was a relationship between moments and cumulants, we can generalize that graphical relation to the case joint moments and joint cumulants. So graphical relation applies as long as points are labeled by appropriate or by corresponding variable.

So suppose I wanted to calculate some kind of a moment that is x1 squared. Let's say x2, x3. This may generate for me many diagrams, so let's stop from here.

So what I can do is I can have points that I label 1, 1, and 2. And have them separate from each other. Or I can start pairing them together. So one possibility is that I put the 1's together and the 2 starts separately.

Another possibility is that I can group the 1 and the 2 together. And then the other 1 starts separately. But I had a choice of two ways to do this, so this comes-- this diagram with an overall factor of 2.

And then there's the possibility to put all of them in the same bag. And so mathematically, that means that the third-- this particular joint moment is obtained by taking average of x1 squared x2 average, which is the first term. The second term is the variance of x1. And then multiplied by x2.

The third term is twice the covariance of x1 and x2 times the mean of x1. And the final term is just the third cumulant. So again, you would need to compute these, presumably, from the law of the characteristic function and then you would be done.

Couple of other definitions. One of them is an unconditional probability. So very soon we will be talking about, say, probabilities appropriate to the gas in this room. And the particles in the gas in this room will be characterized where they are, some position vector q, and how fast they are moving, some momentum vector p. And there would be some kind of a probability density associated with finding a particle with some momentum at some location in space.

But sometimes I say, well, I really don't care about where the particles are, I just want to know how fast they are moving. So what I really care is the probability that I have a particle moving with some momentum p, irrespective of where it is.

Then all I need to do is to integrate over the position the joint probability distribution. And the check that this is correct is that if I first do not integrate this over p, this would be integrated over the entire space and the joint probabilities appropriately normalized so that the joint integration will give me one. So this is a correct normalized probability.

And more generally, if I'm interested in, say, a bunch of coordinates x1 through xs, out of a larger list of coordinates that spans x1 through xs all the way to something else, all I need to do to get unconditional probability is to integrate over the variables that I'm not interested. Again, check is that it's a good properly normalized.

Now, this is to be contrasted with the conditional probability. The conditional probability, let's say we would be interested in calculating the pressure that is exerted on the board. The pressure is exerted by the particles that impinge on the board and then go away, so I'm interested in the momentum of particles right at the board, not anywhere else in space.

So if I'm interested in the momentum of particles at the particular location, which could in principle depend on location-- so now q is a parameter p is the variable, but the probability distribution could depend on q.

How do we obtain this? This, again. is going to be proportional to the probability that I will find a particle both at this location with momentum p. So I need to have that. But it's not exactly that there's a normalization involved. And the way to get normalization is to note that if I integrate this probability over its variable p but not over the parameter q, the answer should be 1.

So this is going to be, if I apply it to the right-hand side, the integral over p of p of p and q, which we recognize as an example of an unconditional probability to find something at position 1. So the normalization is going to be this so that the ratio is 1.

So most generally, we find that the probability to have some subset of variables, given that the location of the other variables in the list are somewhat fixed, is given by the joint probability of all of the variables x1 through xn divided by the unconditional probability that that applies to the parameters of our fixed. And this is called Bayes' theorem.

By the way, if variables are independent, which actually does apply to the case of the particles in this room as far as their momentum and position is concerned, then the joint probability is going to be the product of one that is appropriate to the position and one that is appropriate to the momentum.

And if you have this independence, then what you'll find is that there is no difference between conditional and unconditional probabilities. And when you go through this procedure, you will find that all the joint cumulants-- but not the joint moments, naturally-- all the joint cumulants will be 0.

OK. Any questions? Yes?

AUDIENCE: Could you explain how the condition of p--

PROFESSOR: How this was obtained? Or the one above?

AUDIENCE: Yeah. The condition you applied that the integral is 1.

PROFESSOR: OK. So first of all, what I want to look at is the probability that is appropriate to one random variable at the fixed value of all the other random variables. Like you say, in general I should specify the probability as a function of momentum and position throughout space. But I'm really interested only at this point. I don't really care about other points.

However, the answer may depend whether I'm looking at here or I'm looking at here. So the answer for the probability of momentum is parametrized by q. On the other hand, I say that I know the probability over the entire space to be a disposition with the momentum p as given by this joint probability.

But if I just set that equal to this, the answer is not correct because the way that this quantity is normalized is if I first integrate over all possible values of its variable, p. The answer should be 1, irrespective of what q is.

So I can define a conditional probability for momentum here, a conditional probability for momentum there. In both cases the momentum would be the variable it. And integrating over all possible values of momentum should give me one for a properly normalized probability distribution.

AUDIENCE: [INAUDIBLE].

PROFESSOR: Given that q is something. So q could be some-- now here q can be regarded as some parameter. So the condition is that this integration should give me 1.

I said that on physical grounds, I expect this conditional probability to be the joint probability up to some normalization that I don't know.

OK. So what is that normalization? The whole answer should be 1. What I have to do is an integration over momentum of the joint probability. I have said that an integration over some set of variables of a joint probability will give me the unconditional probability for all the others. So integrating over all momentum of this joint probability will give me the unconditional probability for position.

So the normalization of 1 is the unconditional probability for position divided by n. So n-- this has to be this. And in general, it would have to be this in order to ensure that if I integrate over this first set of variables of the joint probability distribution which would give me the unconditional, cancels the unconditional in the denominator to give me 1.

Other questions? OK.

So I'm going to erase this last board to be underneath that top board in looking at the joint Gaussian distribution. So that was the Gaussian, and we want to look at the joint Gaussian.

So we want to generalize the formula that we have over there for one variable to multiple variables. So what I have there initially is a factor, which is exponential of minus 1/2, x minus lambda squared. I can write this x minus lambda squared as x minus lambda x minus lambda.

And then put the variance. Let's call it is 1 over sigma rather than a small sigma squared or something like this. Actually, let me just write it as 1 over sigma squared for the time being. And then the normalization was 1 over root 2 pi sigma squared.

But you say, well, I have multiple variables, so maybe this is what I would give for my B variable. And then I would sum over all N, running from 1 to N. So this is essentially the form that I would have for an independent Gaussian variables.

And then I would have to multiply here factors of 2 pi sigma squared, so I would have 2 pi to the N over 2. And I would have product of-- actually, let's write it as 2 pi to the N square root. I would have the product of sigma i squared.

But that's just too limiting a form. The most general form that these quadratic will allow me to have also cross terms where it is not only the diagonal terms x1 and x1 that's are multiplying each other, but x2 and x3, et cetera.

So I would have a sum over both m and n running from 1 to n. And then to coefficient here, rather than just being a number, would be the variables that would be like a matrix. Because for each pair m and n, I would have some number. And I will call them the inverse of some matrix C.

And if you, again, think of the problem as a matrix, if I have the diagonal matrix, then the product of elements along the diagonal is the same thing as the determinant. If I were to rotate the matrix to have off diagonal elements, the determinant will always be there. So this is really the determinant of C that will appear here. Yes?

AUDIENCE: So are you inverting the individual elements of C or are you inverting the matrix C and taking its elements?

PROFESSOR: Actually a very good point. I really wanted to write it as the inverse of the matrix and then peak the mn [INAUDIBLE]. So we imagine that we have the matrix. And these are the elements of some-- so I could have called this whatever I want.

So I could have called the coefficients of x and n. I have chosen to regard them as the inverse of some other matrix C. And the reason for that becomes shortly clear, because the covariances will be related to the inverse of this matrix. And hence, that's the appropriate way to look at it.

AUDIENCE: Can [INAUDIBLE] what C means up there?

PROFESSOR: OK. So let's forget about this lambdas. So I would have in general for two variables some coefficient for x1 squared, some coefficient for x2 squared, and some coefficient for x1, x2.

So I could call this a11. I could call this a22. I could call this 2a12. Or actually I could, if I wanted, just write it as a12 plus a21 x2x1 or do a1 to an a21 would be the same. So what I could then regard this is as x1 2. The matrix a11, a12, a21, a22, x1, x2. So this is exactly the same as that. All right?

So these objects here are the elements of this matrix C inverse. So I could call this x1, x2 some matrix A x1 x2. That A is 2 by 2 matrix. The name I have given to that 2 by 2 matrix in C inverse. Yes?

AUDIENCE: The matrix is required to be symmetric though, isn't it?

PROFESSOR: The matrix is required to be symmetric for any quadrant form. Yes. So when I wrote it initially, I wrote as 2 a12. And then I said, well, I can also write it this fashion provided the two of them are the same. Yes?

AUDIENCE: How did you know the determinant of C belonged there?

PROFESSOR: Pardon?

AUDIENCE: How did you know that the determinant of C [INAUDIBLE]?

PROFESSOR: OK. How do I know the determinant of C? Let's say I give you this form. And then I don't know what the normalization is.

What I can do is I can do a change of variables from x1 x2 to something like y1 y2 such that when I look at y1 and y2, the matrix becomes diagonal. So I can rotate the matrix. So any matrix I can imagine that I will find some U such that A U U dagger is this diagonal matrix lambda.

Now under these procedures, one thing that does not change is the determinant. It's always the product of the eigenvalues. The way that I set up the problem, I said that if I hadn't made the problem to have cross terms, I knew the answers to be the product of that eigenvalues.

So if you like, I can start from there and then do a rotation and have the more general form. The answer would stay as the determinant. Yes?

AUDIENCE: The matrix should be positive as well or no?

PROFESSOR: The matrix should be positive definite in order for the probability to be well-defined and exist, yes. OK. So if you like, by stating that this is a probability, I have imposed a number of conditions such as symmetry, as well as positivity. Yes. OK.

But this is just linear algebra. I will assume that you know linear algebra. OK.

So this property normalized Gaussian joint probability. We are interested in the characteristic function. So what we are interested is the joint Gaussian characteristic. And so again we saw the procedure was that I have to do the Fourier transform.

So I have to take this probability that I have over there and do an integration product, say, alpha running from 1 to N dx alpha of e to the minus ik alpha x alpha. This product exists for all values. Then I have to multiply with this probability that I have up there, which would appear here. OK.

Now, again maybe an easy way to imagine is what I was saying to previously. Let's imagine that I have rotated into a basis where everything is diagonal. Then in the rotated basis, all you need to do is to essentially do product of characteristic functions such as what we have over here.

So the corresponding product to this first term would be exponential of minus i sum over N running from 1 to N k alpha lambda alpha. kn lambda n. I guess I'm using n as the variable here. And as long as things would be diagonal, the next ordered term would be a sum over alpha kn squared the corresponding eigenvalue inverted. So remember that in the diagonal form, each one of these sigmas would appear as the diagonal.

If I do my rotation, essentially this term would not be affected. The next term would give me minus 1/2 sum over m and n rather than just having k1 squared k2 squared, et cetera. Just like here, I would have km kn.

What happened previously was that each eigenvalue would get inverted. If you think about rotating a matrix, all of its eigenvalues are inverted, you are really rotating the inverse matrix. So this here would be the inverse of whatever matrix I have here. So this would be C mn. So I did will leave you to do the corresponding linear algebra here, but the answer is correct.

So the answer is that the generator of cumulants for a joint Gaussian distribution has a form which has a bunch of the linear terms-- kn lambda n. And a bunch of second order terms, so we will have minus 1/2 sum over m and n km kn times some coefficient.

And the series terminates here. So for the joint Gaussian, you have first cumulant. So the expectation value of nm cumulant is the same thing as lambda m. You have covariances or second cumulants xm, xn, C is Cmn. And in particular, the diagonal elements would correspond to the variances. And all the higher orders are 0 because there's no further term in the expansion.

So for example, if I were to calculate this thing that I have on the board here for the case of a Gaussian, for the case of the Gaussian, I would not have this third term. So the answer that I would write down for the case of the third term would be something that didn't have this. And in the way that we have written things, the answer would have been x1 squared x2-- would be just a lambda 1 squared lambda 2 plus sigma 1 squared, or let's call it C11, times lambda 2 plus 2 lambda 1 C12. And that's it.

So there is something that follows from this that it is used a lot in field theory. And it's called Wick's theorem. So that's just a particular case of this, but let's state it anyway.

So for Gaussian distributed variables of 0 mean, following condition applies. I can take the first variable raised to power n1, the second variable to n2, the last variable to some other nN and look at a joint expectation value such as this. And this is 0 if sum over alpha and alpha is odd and is called sum over all pairwise contraction if a sum over alpha and alpha is even.

So actually, I have right here an example of this. If I have a Gaussian variable-- jointly distributed Gaussian variables where the means are all 0-- so if I say that lambda 1 and lambda 2 are 0, then this is an odd power, x1 squared x2. Because of the symmetry it has to be 0, but you explicitly see that every term that I have will be multiplying some power of [INAUDIBLE].

Whereas if for other than this, I was looking at something like x1 squared x2 x3 where the net power is even, then I could sort of imagine putting them into these kinds of diagrams. Or alternatively, I can imagine pairing these things in all possible ways. So one pairing would be this with this, this with this, which would have given me x1 squared C x2 x3 C.

Another pairing would have been x1 with x2. And then, naturally, x1 with x3. So I would have gotten x1 with x2 covariance x1 with extreme x3 covariance. But I could have connected the x1 to x2 or the second x1 to x2.

So this comes in 2 variance. And so the answer here would be C11 C23 plus 2 C1 2 C13. Yes?

AUDIENCE: In your writing of x1 to the n1 [INAUDIBLE]. It should be the cumulant, right? Or is it the moment?

PROFESSOR: This is the moment.

AUDIENCE: OK.

PROFESSOR: The contractions are the covariances.

AUDIENCE: OK.

PROFESSOR: So the point is that the Gaussian distribution is completely characterized in terms of its covariances. Once you know the covariances, essentially you know everything. And in particular, you may be interested in some particular combination of x's. And then you use to express that in terms of all possible pairwise contractions, which are the covariances.

And essentially, in all of field theory, you expand around some kind of a Gaussian background or Gaussian 0 toward the result. And then in your perturbation theory you need various powers of your field or some combination of powers, and you express them through these kinds of relationships.

Any questions? OK. This is fine. Let's get rid of this. OK.

Now there is one result that all of statistical mechanics hangs on. So I expect that as I get old and I get infirm or whatever and my memory vanishes, the last thing that I will remember before I die would be the central limit theorem.

And why is this important is because you end up in statistical physics adding lots of things. So really, the question that you have or you should be asking is thermodynamics is a very precise thing. It says that heat goes from the higher temperature to lower temperature. It doesn't say it does that 50% of the time or 95% of the time. It's a definite statement.

If I am telling you that ultimately I'm going to express everything in terms of probabilities, how does that jive? The reason that it jives is because of this theorem. It's because in order to go from the probabilistic description, you will be dealing with so many different-- so many large number of variables-- that probabilistic statements actually become precise deterministic statements.

And that's captured by this theorem, which says that let's look at the sum of N random variables. And I will indicate the sum by x and my random variables as small x's. And let's say that this is, for the individual set of things that I'm adding up together, some kind of a joint probability distribution out of which I take these random variables.

So each instance of this sum is selected from this joint PDF, so x itself is a random variable because of possible choices of different xi from this probability distribution. So what I'm interested is what is the probability for the sum? So what is the p that determines this sum?

I will go by the root of these characteristic functions. I will say, OK, what's the expectation value of-- well, let's-- what's the Fourier transform of this probability distribution? If we transform, by definition it is the expectation of e to the minus ik this big X, which is the sum over all off the small x's.

Do I have that definition somewhere? I erased it. Basically, what is this? If this k was, in fact, different k's-- if I had a k1 multiplying x1, k2 multiplying x2, that would be the definition of the joint characteristics function for this joint probability distribution.

So what this is is you take the joint characteristic function, which depends on k1 k2, all the way to kn. And you set all of them to be the same. So take the joint characteristic function depends on N Fourier variables. Put all of them the same k and you have that for the sum.

So I can certainly do that by adding a log here. Nothing has changed. I know that the log is the generator of the cumulants. So this is a sum over, let's say, n running from 1 to infinity minus ik to the power of n over n factorial, the joint cumulant of the sum.

So what is the expansion that I would have for log of the joint characteristic function? Well, typically I would say have at the lowest order k1 times the mean of the first variable, k2 times the mean of the second variable. But all of them are the same. So the first order, I would get minus i the same k sum over n of the first cumulant of the N-th variable.

Typically, this second order term, I would have all kinds of products. I would have k1 k3 k2 k4, as well as k1 squared. But now all of them become the same, and so what I will have is a minus ik squared. But then I have all possible pairings mn of xm xn cumulants.

AUDIENCE: Question.

PROFESSOR: Yes?

AUDIENCE: [INAUDIBLE] expression you probably should use different indices when you're summing over elements of Taylor serious and when you're summing over your [INAUDIBLE] random variables. Just-- it gets confusing when both indexes are n.

PROFESSOR: This here, you want me to right here, say, i?

AUDIENCE: Yeah.

PROFESSOR: OK. And here I can write i and j. So I think there's still a 2 factorial. And then there's higher orders.

Essentially then, matching the coefficients of minus ik from the left minus ik from the right will enable me to calculate relationships between cumulants of the sum and cumulants of the individual variables. This first one of them is not particularly surprising. You would say that the mean of the sum is sum of the means of the individual variables.

The second statement is that the variance of the sum really involves a pair, i and j, running from 1 to N. So if these variable were independent, you would be just adding the variances. Since they are potentially dependent, you have to also keep track of covariances.

And this kind of summation extends to higher and higher cumulants, essentially including more and more powers of cumulants that you would put on that side.

And what we do with that, I guess we'll start next time around.