Computational Methods in Optimization

David Gleich

Purdue University

Spring 2023

Course number CS-52000

Monday, Wednesday, Friday, 1:30-2:20pm

Location Lawson 1106

Homework 4

Please answer the following questions in complete sentences in a clearly prepared manuscript and submit the solution by the due date on Gradescope.

Remember that this is a graduate class. There may be elements of the problem statements that require you to fill in appropriate assumptions. You are also responsible for determining what evidence to include. An answer alone is rarely sufficient, but neither is an overly verbose description required. Use your judgement to focus your discussion on the most interesting pieces. The answer to "should I include 'something' in my solution?" will almost always be: Yes, if you think it helps support your answer.

Problem 0: Homework checklist

• Please identify anyone, whether or not they are in the class, with whom you discussed your homework. This problem is worth 1 point, but on a multiplicative scale.

• Make sure you have included your source-code and prepared your solution according to the most recent Piazza note on homework submissions.

Problem 1: Log-barrier terms

The basis of a class of methods known as interior point methods is that we can handle non-negativity constraints such as $\vx \ge 0$ by solving a sequence of unconstrained problems where we add the function $b(\vx; \mu) = -\mu \sum_i \log(x_i)$ to the objective. Thus, we convert $$\MINone{}{f(\vx)}{\vx \ge 0}$$ into $$\MIN{}{f(\vx) + b(\vx; \mu).}$$

1. Explain why this idea could work. (Hint: there's a very useful picture you should probably show here!)

2. Write a matrix expression for the gradient and Hessian of $f(\vx) + b(\vx; \mu)$ in terms of the gradient vector $g(\vx)$ and the Hessian matrix $\mH(\vx)$ of $f$.

3. (Open ended) Explain if there is anything special about $\log$ that makes it especially useful here. For instance, are there other functions that could work and give you the same effect? Would $\log$ be superior?

Problem 2: Inequality constraints

Draw a picture of the feasible region for the constraints: $$\bmat{1 - x_1 - x_2 \\ 1 - x_1 + x_2 \\ 1 + x_1 - x_2 \\ 1 + x_1 + x_2} \ge 0.$$

Problem 3: Necessary and sufficient conditions

Let $f(\vx) = \frac{1}{2} \vx^T \mQ \vx - \vx^T \vc$.

1. Write down the necessary conditions for the problem: $$\MINone{}{f(\vx)}{\vx \ge 0}.$$

2. Write down the sufficient conditions for the same problem.

3. Consider the two-dimensional case with $$\mQ = \bmat{2 & 3 \\ 3 & 2} \qquad \vc = \bmat{-1.5 \\ 1.5}.$$ Determine the solution to this problem by any means you can, and justify your work.

4. Produce a Julia or hand illustration of the solution showing the function contours, and gradient. What are the active constraints at the solution? What is the value of $\lambda$ in $\mA^T \lambda = \vg$?

5. What changes when we set $\mQ = \bmat{-2 & 3 \\ 3 & -2}$?

Problem 4: Constraints can make a non-smooth problem smooth.

Show that

$$\MIN{}{\sum_i \max (\va_i^T \vx - b_i, -0.5)}$$

can be reformulated as a constrained optimization problem with a continuously differentiable objective function and both linear equality and inequality constraints.