All right. Welcome back. We have a sparse group here today. So, we'll definitely have
a quiz so you guys are all in luck. What are we talking about today? So, large scale optimization.
There are some handouts. I didn't get a chance to post them before class, but again, there
are some handouts that go over a lot of the same stuff that we'll see here that you can
check out after class, too. So, things we're going to see. Why is large scale optimization
challenging? We'll then talk about this idea called limited memory quasi-Newton, which
allows you to get something a little bit better. Sorry. It's sort of a hybrid technique that
gives you a little bit of curvature information. It's not perfect, but it generally does a
little bit better than just doing pure gradient descent. Other ideas sort of in that vein
are conjugate gradient. And then we'll go back to cases where you sort of have a little
bit -- like, you can actually do something with the Hessian. So, sparsity in exact Newton.
So, I don't think we're going to get to all of this stuff today. So, we'll do whatever
we don't get to today on Thursday. And, yeah. So, I guess in terms of projects, I should
do a little bit about projects. So, projects... I forget when they're due now, but they're
due whenever I said they were due. Two weeks? Okay. Due in two weeks. So, you don't have
any more interim status reports. So, your last one was last time. Again, in terms of
things that are important, if you're doing the optimization on the surface of the hypersphere,
I actually do expect you to try some gradient based optimization. So, if you're not going
to do some gradient based optimization, you probably want to have a conversation with
me. And in particular, if you're doing gradient based optimization, what do you always do
with what you call your gradient? Check it. So, if you don't check it, you're probably
going to lose some points on your write-up. If you implement your own optimization software,
I'm not going to review it, but I do want you to give me some justification that it's
plausibly correct. So, what might that look like? Take some problem where you know the
answer, make sure you get it. Yeah?
>> Many of the algorithms for the software are coming from textbooks. Is it okay if we
basically iterate through those proofs?
>> So, that is a theoretical justification that what you have done in principle is correct.
That does not mean your software is correct. So, I'm looking for a software based demonstration
of this, not a, hey, in theory, these ideas work. Yes, in theory, lots of ideas work,
but you may have made an error when you translated those ideas onto the computer. A subtle error,
but it might -- again, how do you know you haven't made a subtle error in translating
the algorithms onto the computer? So, I would like to see some justification of that as
well.
If you're using libraries, again, that is not an excuse to trust the library. Libraries
have errors just like your software does, too. They're less likely, but they still do
happen. So, I would expect the same thing there. Yes?
>> What's the value you use for project 2?
>> So, I'll get to project 2 in a second. So, again, on project 1, essentially, I want
to see a write-up that I don't have to read your code to understand that what you've done
is correct. So, you should definitely justify that in your things, and then hopefully you've
done something interesting in this gen rule space. So, other questions about project 1,
the hypersphere? Yes?
>> Yeah, just a quick question. So, on the grade scope, there's a portion or a place
to upload the report, as well as upload the code. If we're doing project 1, do we...
>> You do not need to upload code.
>> Just a report?
>> Yeah, just a report.
>> We're not going to get marked down for not doing anything?
>> You will not get marked down for not doing anything. I will only... I mean, I'm only
going to download the codes for project 2. I mean, sorry. I guess if you upload a code
for project 1, I'm not going to run it. And I probably won't even look at it. So, I mean...
Although, I wouldn't test that, because I may look at it. But, anyway... I guess I should
say test that at your own risk. So, if you want to upload something fun there, you know...
All right. There you go. I'd suggest your time could be better spent in improving your
writing or running your stuff through some grammar checks or spelling checks or things
like that. So... All right. So project... Other questions about project 1? So project
2, I definitely need a Julia file, because what I'm going to do is I'm going to include
your Julia file. So I'll probably specify a file name convention that I'll want you
to use. Like, you know, just your group's last name or something like that. Actually,
it will probably just be project2.jl. Let me just make that clear. So if you need obscure
packages, you definitely want to check those with me ahead of time. So if you're using
things that we've used in class, you can sort of assume that I'll have them. But if you're
using exotic or obscure packages, just let me know ahead of time. So... But in particular,
you probably shouldn't be using things like jump or other types of solvers. So those ones
would be... I don't want to say forbidden, because if you have a good reason, I'm open
to it. But you should have a very good reason for depending on any solver package.
So I'm going to put a note here, just accept solvers. What tolerance? 10 to the minus 6.
10 to the minus 4. 10 to the minus 5. I don't know. I haven't decided which one I'm going
to run yet. But certainly... I would try and make sure my software is well-behaved for
a variety of tolerances. In the past, people have complained, because I ran with a different
tolerance than what they thought I was going to run with. And then they said, well, if
I tweak this constant here, then it works. I'm like... Yeah, but I told you I wasn't
going to run with a particular tolerance ahead of time. And so if you're overly reliant on
your tolerance for how you've sort of tweaked your algorithm, that is the kind of thing
you tend to lose points for on this one. So best advice is try a bunch of tolerances,
and make sure your algorithms behave nicely and/or fail gracefully for lots of them. Yes?
Yes, you can do smarter linear algebra. I should be clear. Linear systems. Let me put
it that way. But again, if you're using some obscure package or things that we haven't
used in class, you might just want to check those with me. I haven't seen anyone in their
projects reporting on that yet, or anything that sort of alarmed me. But if I do see it,
I will let you know. I mean, I also might check stuff like 10 to the minus 2, 10 to
the minus 3, to see if you're sort of solving them appropriately for the tolerance as well.
So you look skeptical about 10 to the minus 6 as a tolerance. Of course the performance
is going to vary between the tolerances. So presumably if I'm going to specify a higher
tolerance, I'll specify a higher maximum number of iterations, too. But again, if your solver
runs out of iterations, you should tell me that, right? So for instance, if I call your
solver with a tolerance of 10 to the minus 6 and 5 iterations, you should tell me what
in your output. Maximum number of iterations reached. So that then I know that your solver
knows that it needs more iterations, rather than... I mean, if it declares success, I'm
going to assume that it's correctly solved the problem and evaluate it. But if it declares
that I need more iterations, then that wouldn't be fair of me to demand that it solves it.
Or sorry. To evaluate that. I do both manual inspections and a test harness.
But again, the test harness logic is exactly what you might expect. Run it with something
like this. Check. Does it report I need more iterations? Or throw an error? Or something
like that. And then if so, go back and increase the number of iterations.
So again, you've got to have a file called project2.jl. I know someone was talking about
doing this in Python. So if you're still thinking about that, please do come up and chat with
me just to figure out if you're still doing that or not.
Other questions about project2?
Anything you need in order to run it. Like, you've got the interface that I'm gonna call
your software on.
You don't have to handle anything for loading data. You can assume I will give you IPLP
problems.
Well, you probably need the IPLP struct definition, too.
I would put that into the report, not in what you submit. Because if you submit something
and then it causes a problem on my end, you lose points for it.
So I'd make sure that you can sort of load this. If someone posts a note on edstem, I'll
try and give you a test case, test harness that you can use.
You do not have to do integer programming.
All right. Should we go on to large-scale optimization?
So, what is large-scale optimization? I don't know. I think...
Certainly I think we'd all agree that if you've got a million variables and you're trying
to solve an optimization with a million variables, that's large-scale.
So a million is definitely large-scale. Is 10,000 variables large-scale?
Well, I would argue that 10,000 is sort of an intermediate point.
And I'll be a little bit more precise about this.
But what I really mean by large... My mental concept of large-scale optimization is something
where if you tried to use a quasi-Newton method, just a vanilla quasi-Newton method, it wouldn't
work because you're out of memory or it's too expensive.
So can you do a quasi-Newton method with 10,000 variables? 10,000, yes. 25,000? Probably.
50,000? Yeah, but it's getting expensive. 100,000? It's getting real expensive at 100,000,
although you can still do it.
And sort of somewhere... But you can sort of see that there's a gradient here, where
it feels like there's some sort of transition point around 10,000, where things get more
interesting and maybe you want to do something else.
So example... So this was one of my PhD students. Old PhD student. Now at Texas A&M. So he's
now at Texas A&M. So he was looking at a problem with 3 trillion constraints and 161 million
variables. And so this took days and weeks to solve. So with 3 trillion
constraints, you can't even really form all the constraints. And so you end up with some
interesting stuff there. And again, this was just one problem out of many we would like
to solve. But certainly, we're going to need some different strategies for something like
that. Yes?
So in this case, it was a particular method for clustering data. So every data point had
basically n choose 3 associated constraints and n choose 2 associated variables.
So do we need to do this to cluster data? No. We were sort of looking at some more interesting
slash exotic slash really principled algorithms in this space that are then much, much, much
more expensive to solve than a lot of other ones were. If you want a slightly more refined
answer, we were trying to get lower bounds on performance of some clustering algorithms.
But again, I think we all know of optimization problems with trillions of constraints. Sorry,
with trillions of variables. So has anyone heard of something called chatGPT? So chatGPT,
when they're constructing chatGPT, what they're doing is they're really optimizing a function
with the number of weights or parameters as the number of constraints. And although they
haven't disclosed it, it's reported to be in the multiple trillions.
Can I ask a question? Yeah.
[INAUDIBLE] Sorry. So you've got a machine learning model
to approximate the constraints? Yeah. Based on the given data.
So in general, I think just throwing a machine learning problem at it is bad science. On
the other hand, it's very good practice in the sense that it's a useful tool. But you
don't really learn anything about it. And if it works-- if it fails, you don't-- you've
got no understanding of why. So I think it's one of those things where you throw it at
it, see if it works. And if it works, you say, all right, great. Then maybe you think
about it a little bit harder and think, is there a reason that this works?
But as far as do I think it's a good idea or not, it is a tool in a tool chest. So tools
are always good ideas. Does this mean that this idea is particularly good? No, I don't
think there's anything special about ML algorithms. So if you've got a tough optimization problem
with lots of constraints, you need some way of handling them. And if there's a particular
reason that they work well for your application, great. But a priori, I don't see any sort
of special reason why they would work sort of fundamentally better than other techniques.
[INAUDIBLE]
So I guess we'll have to chat a little bit later. Because again, chat GPT is like-- the
states here are constructed to build an algorithm. And so it gets a little bit tricky to talk
about some of these ideas in a way that's coherent. Certainly, you can optimize over
the space of algorithms, right? You parameterize your algorithm by some features, and then
you optimize over it. So you need a parametric class of algorithms, and then you can do things
like this.
So why is it hard? So there's one reason it could be hard, which is my function takes
a really long time to evaluate. So again, if you're actually evaluating a function with
trillions of variables, it might sort of fundamentally take hours, days, to potentially just evaluate
a single function call and get a gradient. We are not going to look at methods that take
this particular scenario. So for these things, you want surrogate optimization, which is
a different--
There's this other idea called sequential quadratic programming, which also shows up
a lot, where you essentially build a quadratic surrogate to your function, and then spend
all of your time optimizing that. And then every so often, you'll update your quadratic
surrogate with calls to your function.
But we are going to assume...
So we're going to assume that you can evaluate your function or gradient, say, 1,000 or 10,000
times within whatever time budget you want to run your optimization problem. So if you're
willing to wait a week, then maybe you could go 10,000 or 100,000 times. If you're willing
to wait a day, then maybe 1,000 to 10,000 times. So these things aren't so expensive
that it takes a day just to evaluate your function a single time.
Does that make sense with everyone? Because this is sort of a difference, where in small
scale optimization, you generally assume that your functions are not super hard to evaluate,
although they may be. But if they are, then you're not going to evaluate them that many
times. But in large scale stuff, you do get this sort of break in how you might think
about that.
So if you want to formally... I would want them to be little o of n squared. That is,
they take less time to evaluate than the squared number of variables. So eg... So if I've got
n variables, then your function might take order, say, n log n to evaluate, or n to the
3/2, or something along those lines. It's just not order n squared.
So I'm going to do a little bit of math here. So I'm going to take n log n to the 3/2, and
then I'm going to... So the implications of this is we can run line search. You have to
be able to store all of your variables. So I guess I didn't say that, but I am assuming
you can store all of your variables. Hopefully everyone's okay with that.
Again, in the chat GPT case, where you honest to gosh have trillions of variables, or in
the 3 trillion constraint case, this does sort of become an issue. But for keeping our
discussion compact, we are going to assume that we can store the variables.
So the implications of these two is we can run line search, and if we can run line search,
then we can run gradient descent. So we can do optimization in this scenario. There's
nothing fundamentally that's broken about this. Typically, when I teach this stuff,
I talk about an algorithm called conjugate gradient before this, but I decided to switch
it up this year and talk about limited memory quasi-Newton first.
So we can also run conjugate gradient, which I think we're going to see next lecture. But
let's do it. So could we do something like Newton with 100,000 dimensional problems?
So f takes as input 100,000 variables and outputs a single scalar. And this is going
to be your quiz. I'd like you to think about this for a second and give one of the following
two answers. Yes, no, maybe. And also why.
All right. So I am going to shuffle these up a little bit and then go through them and
report out what the class thinks about this. So yes, but very expensive. Yes.
Yes. Maybe. Another. I've seen two very expensive. No. I like the definitiveness.
Sorry. I guess this was 10 to the 15 is too big. Yes. Another yes. Another 10 to the 15
is too complicated. Another yes. Maybe.
I'm going to call this one a maybe. No. Sorry. No.
Another yes, but very expensive. No. All right. Another no. Another maybe. I think we're going
to end up tied, a three-way tie amongst the class here in a second. No or maybe. OK. I'm
going to go with maybe here. I think that's-- all right. And at the end, no's come in to
too big to handle. All right. So we have almost a perfect tie amongst everyone in this class.
So among people who said it was possible, they said, look, there's nothing that goes
wrong when you try and do this. And 10 to the 15 is big. But so what do we need to do?
Newton said Newton. 10 to the 5 to the third. So you need to do O of n cubed. So Newton
needs O of n cubed work per iteration. So just to store the hashing, you need about
80 gigabytes of memory. All right. So 80 gigabytes is a lot of memory. Who has access to a computer
with more than 80 gigabytes of memory? I know a few of you do. So it's not like an impossible
amount of memory. On the other hand, every step you're going to have to solve a linear
system of equations with this, which 10 to the 15 work. Again, that's not impossible.
So I would actually say the answer to this one is probably maybe. It depends on how quickly
you need to do it. The point is it's going to be really expensive. So I'm actually sort
of pleased with this distribution of answers in the sense that I think the no's and maybe's
probably have it over the yeses. But on the other hand, if you have access to sufficient
compute resources, you can do this. Now, if n is equal to 10 to the 6, can I get a quick
show of hands who thinks it's yes, no, and maybe? So can I see the yeses?
Okay. Six of you said yes for 10 to the 5th. You're all going to say no for 10 to the 6th?
You're going to say yes? What's that? We've got two yeses.
That still won't store your Hessian. That's 8 terabytes of RAM. But on the other hand,
you're right. You can rent a computer on Amazon with 32 terabytes of RAM. So we can run it
on that one, which I think costs $200 an hour to rent or something like that.
Per iteration. He's got his own research funding, by the way. No, I'm teasing. No. Who thinks
no for this one? Okay. Like 10 and maybe? A few.
So again, I think these are all consistent answers. I haven't said anything about the
type of problem. So if your Hessian has any type of structure in it, I think all of you
would start to say, like, oh, you're all going to switch over to the yes column if I tell
you your Hessian's diagonal, right? Yes, you should all switch over to the yes column if
your Hessian happens to be diagonal, in which case then doing Newton is not really any more
work or expense than gradient descent. Now, you have to know that your Hessian's diagonal
up front in order to avoid computing all the irrelevant stuff. But that would be like a
super easy Hessian to work with. Yes?
Didn't you say f was the [INAUDIBLE] so why would-- like, the Hessian would be that square,
right?
The Hessian would be that square. But I never said-- like, your Hessian might have structure,
right? So like, it might not. But if you're solving something, then maybe you're solving
a problem where you know your Hessian has tridiagonal structure or something like that.
In particular, I solve a lot of problems where your Hessian is graph-structured. So it's
a sparse matrix. So if I tell you you've got to solve a million by a million sparse matrix
at every turn, well, that still becomes expensive, but doesn't become impossibly problematic
expensive. So that's why, like, I think sort of shifting from the no column over to the
maybe column. But again, all of this just becomes harder as the problems get bigger
and bigger and bigger. So. Yes?
Excuse me, question, but why is space complexity is cubed, not squared?
Space complexity is squared. Time complexity is cubed.
Oh, sorry. This is-- I miswrote this. Space. Oh, no, no, no. That was end of the work.
Okay.
So
we'll talk about structure in a little bit.
Do quasi-Newton methods help?
So quasi-Newton methods allowed us to get super linear convergence rates, which is sort
of what we want.
Really fast convergence.
They don't need the Hessian explicitly, right?
They were working with some type of Hessian surrogate or Hessian approximate.
So I would say no.
Why would I probably say no?
If I tell you my answer is no, what do you think is the best reason for why quasi-Newton
methods aren't going to help this?
What's that?
So there's no history in quasi-Newton methods.
So I think of memory as the bigger constraint than time, in that if you are pressed to,
you can always let your computer run longer, but you can't get more memory.
And so quasi-Newton methods still take order n squared memory.
So they're going to take 8 terabytes of RAM for that million by million problem.
And in fact, I would argue it's even worse in that even if your Hessian does have some
structure, quasi-Newton methods won't capture that at all.
They're going to give you a fully dense approximate matrix.
All right.
But is all hope lost?
Let's study BFGS for a second.
So remember, BFGS maintains an approximation of your inverse Hessian.
So the way I like to think of this is I minus rho.
So just because I get tired of writing these things, I'm going to call this matrix L, and
I'm going to call this matrix R. In some other notes, we're going to use VK
for those matrices.
So I'm just trying to keep things simple here as I explain a concept.
So what do we need to do?
So what we need to do with TK+1 is we don't actually need TK+1.
What we need to do with TK+1 is we just need to multiply it against the negative gradient.
So the idea of numerical linear algebra and lots of other things is if a problem has structure,
we should try and exploit that structure.
So we do have a little bit of structure here.
Suppose I gave you a routine to multiply TK by a vector.
I claim then we could compute TK+1 times a vector in light of that routine for TK times
a vector.
How would you do it?
Well, you'd say PK is equal to TK+1 times minus G is equal to L times TK times R plus
rho SS transpose -- let me write it this way.
So here and here, this is just a scalar, and R times -- let me call this a vector Z.
So
hopefully I didn't screw up any constants while I was doing this one.
So what do I have to do here?
So this is an inner product.
This is an inner product.
And then I just need to form a linear combination of G and this inner product times Y.
So does everyone agree I can compute the vector Z?
Like I've got access to the vectors Y and S and the quantity rho.
I see a lot of people give me blank stares.
What are your questions about computing the quantity Z?
If I gave you access to Y, S, and rho on the computer and gave you access to the vector
minus G, I claim we can compute Z.
You just literally plug these quantities into Julia, and it'll happily do it.
I'm seeing a few people nod along.
If you've got questions about this, ask, because it's about to get more complicated.
So it's worth appreciating what's going on here first.
So then we inductively-- I've said that we've got some way of doing TK times Z.
Let's call this one-- what's a good variable?
Man, I'm running out of variables here.
So I've used Y, S, T.
Let me go with D.
I'm going to call this one D. I'm going to call this one D. I'm going to call this one
Z.
So then this entire thing just becomes L times D, which we do just like Z, plus beta, which
is this quantity over here, times rho times S.
So the idea is-- what is going on?
We all on the same page?
Essentially, if I just store the change-- so I think you were getting at this point
with history.
Some of you have looked at limited memory quasi-Newton methods already in project one.
But here we're just going through them for the entire class.
So the idea is, if I just store the change, I can just sort of update what I've seen before.
And so the idea is, I'm just going to do this.
So let's see what happens.
So let Vk is equal to-- oops, these are all k's.
Oops.
Sorry.
So if T(k+1) is equal to that expression, then T(k)--
T(k) is equal to that expression just with k switched to k minus 1.
This is true at every step of the process.
And so if I just substitute this in, or I can do it one more step, T(k-1) is equal to
2 transpose T(k-2) V(k-2) plus rho(k-2) S(k-2) S(k-2) transpose.
And so I get T(k) is equal to-- if I just substitute all of these together.
V(k-1) transpose.
So I just take this guy and plug him in here, and then this guy and plug him in here, except
I'm just going to unroll it the other way.
So V(k-1) times V(k-2) transpose dot dot dot V(0) transpose T(0) V(0) V(k-1) plus rho(k-1)
S(k-1) S(k-1) transpose plus rho(k-2) times S-- sorry, times V(k)-- sorry, this is V(k-1).
This is where you need to be a little careful.
So essentially this is what I get from unrolling all of these, rho dot dot dot rho(1) times
V(k-1) transpose dot dot dot V(k)-- or sorry, V(1).
So you've got to be careful to get your indices right, and this is where having access to
a computer really helps, because you can just do this all on paper and pencil and then plug
it into something like Julia or Python or MATLAB and just double check that you're not
off by an index somewhere.
So I can't remember if that last one's right, but it's-- like, I certainly assume you guys
get the idea.
And so there's an algorithm to do this, which-- I mean, I guess I can write out in the last
few minutes of class.
But essentially what we do is we use this implicit representation, and the only thing
that's not specified here is T0.
So it turns out when you're using limited memory quasi-Newton methods-- so I guess I
should say this is usually called a limited memory quasi-Newton method.
And so what's the idea?
Are we going to store-- so when I say limited memory, what we're going to do is we're only
going to store a finite number of these updates.
So you might store 50 of them.
You might store 100 of them.
You could store 1,000 of them.
The point is you're not going to store capital-- like, little n of them, because if you stored
little n of them, then you'd be storing more than you would have stored the Hessian.
And then when you want to work with them, you can compute Hessian-- or sorry, inverse
Hessian times vector products, using an algorithm that I can sketch out, so-- so alg.
So it is really important how you scale that initial diagonal approximation.
So usually, you use some scaled multiple of the identity.
And then for--
[INAUDIBLE]
Scaling is important.
So here is a common scaling you will see.
So what this is trying to do is it's trying to estimate the magnitude of the Hessian based
on your most recent iteration.
And so it's attempting to roughly scale everything.
And so you could probably also use some ideas from momentum here.
I imagine someone's worked out a few of those things.
They're trying to get at this same type of information, just with different types of
techniques.
And then once-- OK.
So you just keep the k most recent things.
And usually, people will say something like in a circular buffer.
So you just have k slots allocated.
And as you keep updating them, you just overwrite the last one in your buffer.
So there's an algorithm to actually compute this t of k times the vector in the nodes.
It's not particularly interesting in the sense that all we do is we just take advantage of
the structure and compute things sort of like a step at a time.
So conceptually, it runs down through all of the k's.
So from k plus 1 all the way down to k 0 to compute some things.
And then it runs upward to compute the rest.
And so it's the kind of algorithm where I imagine anyone in this class could solve it
if you were given like a week of time to do it or something maybe even more like a day.
So it's not a-- it's nothing really super sophisticated or deep.
So that's the sort of high level pitch about limited memory quasi-Newton methods.
So I don't actually think anyone's ever proved the convergence rate of this is any faster
than linear.
But empirically, they do go quite a bit faster than linear.
So you do really benefit from having this Hessian information on a lot of problems.
But again, I don't think there's ever been a formal proof of that.
I think it was-- at least as of a few years ago, it was still unsolved.
So I should double check if it's still unsolved or not.
What are people's questions about these methods?
I don't have any good advice on how to pick the amount of updates you store, k.
In general, you don't want it too big.
And you also don't want it too small.
So like 50 to 100 tends to be where people often set them.
The reason you don't want it too big is, yes, you do more accurate work on approximating
your Hessian if you have it really big.
But then you spend a lot more work computing these products.
And it's not clear that you get a benefit from that versus doing something that's a
little bit more gradient descent-like.
So you just have a trade-off between how much work you do at each iteration and work with
your optimization algorithm versus work towards your objective.
If you wanted to do something fancy, I suspect once you start getting close to a solution,
maybe you want to increase it a little bit to make the directions just a little bit more
Newton-like there.
So you might want something that grows slightly as the iterations proceed.
And I should also say that this is for unconstrained optimization.
If you want to use these ideas on constrained optimization, again, then you go down the
route of some of the augmented Lagrangian methods, all of which convert this stuff into
an unconstrained problem that you solve at every iteration.
All right.
Questions about this before I jump into a different topic?
Time to figure out what I want to do next, because I don't have time to do too many...
So let me just give you...
So structured Hessians.
So imagine...
Shoot, there's a scalar here.
I can't remember the scalar here.
But there's some scalar.
So this is basically diagonal.
Where might you see this?
Well, if you're doing some type of linear fitting where you want non-negativity and
you're using a log-barrier term, then essentially you're going to get a very nice structured
Hessian to that one.
If your only non-linearity comes from your log-barrier term.
In which case, again, you can totally use Newton on very large scale problems of this
form, because your Hessian has tons of structure.
It's just a diagonal.
There are no off-diagonal terms in that at all.
Let me do a row here.
So, in a lot of problems I look at, you have something like...
So...
So you end up with something like this.
You've got F of X, which is X transpose LX, where L is a Laplacian matrix, if you know
what that is.
But it's a sparse matrix here with some structure I know.
So it's basically an input.
And I also have some fractional power of X over here.
So think P is like 4, something like this.
In which case, H of X is equal to 2L plus P times P minus 1 times...
Sorry.
Let me write it like this.
So again, there's some type of sparsity structure to your Hessian, combined with some other
type of nonlinearity.
So it's not just constant.
But you have some very well-defined structure.
So another common structure you'll see would be a banded diagonal.
And these come...
So, if I look at all the interactions, and for whatever reason, each variable only interacts
with, say, its second nearest neighbors, then you'd get these linear coupling terms.
That would give you a banded diagonal.
So this shows up a lot in signal processing, where you have some type of implicit filter
structure that has some sequential set of variables.
It also shows up in partial differential equations, if you're doing anything on a one-dimensional
grid, where you just sort of have nearest-neighbor interactions.
Other places where this shows up...
I don't know.
It shows up all over the place.
But that's another sort of common structure you'll see.
What else?
Has anyone else seen structured Hessians?
I guess your Hessian could be low rank.
That would be another type of structure.
You might see...
But those are slightly less common, as usually you want your Hessian to be full rank for
a lot of different reasons.
But again, the point here is not that any of these structures are things you'll see.
It's just...
If you have any of these structures in your Hessian, and you can show that they arise,
you can still do a lot of ideas in large-scale optimization, without resorting to things
like limited memory, quasi-Newton, or these other types of ideas.
All right.
I think that's it for today.
So we'll resume next time talking about conjugate gradient, and how to take advantage of some
of the types...
Some of these types of sparsity.
So I will see you folks on Thursday.
[BLANK_AUDIO]