\documentclass{article}

% Packages required to support encoding
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

% Packages required by code


% Packages always used
\usepackage{hyperref}
\usepackage{xspace}
\usepackage[usenames,dvipsnames]{color}
\hypersetup{colorlinks=true,urlcolor=blue}

\input{homework.tex}


\begin{document} 
\hypertarget{homework_3_1}{}\section*{{Homework 3}}\label{homework_3_1}

Please answer the following questions in complete sentences in a typed manuscript and submit the solution to me in class on February 7th, 2012.

\hypertarget{problem_1_checking_hessians_and_matrix_calculus_2}{}\subsection*{{Problem 1: Checking Hessians and matrix calculus}}\label{problem_1_checking_hessians_and_matrix_calculus_2}

In class, we mentioned that it was vital to double-check gradient calculations, and we then used the {\colorbox[rgb]{1.00,0.93,1.00}{\tt gradientcheck\char46m}} function to do this. Of course, if we want to utilize Newton'{}s method to solve a problem, then we need to check the Hessian matrix as well.

\begin{enumerate}%
\item Implement {\colorbox[rgb]{1.00,0.93,1.00}{\tt hessiancheck\char46m}} to verify that the Hessian is correct. You may reuse the function {\colorbox[rgb]{1.00,0.93,1.00}{\tt gradientcheck\char46m}}


\item Determine \emph{and verify} the Hessian of $f(\vx) = e^{-\vx^T \mA \vx}$ where $\mA$ is not necessarily symmetric or positive definite. Show the result of {\colorbox[rgb]{1.00,0.93,1.00}{\tt hessiancheck}} for a set of test matrices $\mA$ and test points $\vx$. Be adversarial in your choice. Analyze any erratic behavior your find.


\item Determine \emph{and verify} the Hessian of $f(\vx) = \ve^T \cos(\vx)$. Show the result of {\colorbox[rgb]{1.00,0.93,1.00}{\tt hessiancheck}} for a set of test matrices test points $\vx$. Be adversarial in your choice. Analyze any erratic behavior your find.


\item Determine \emph{and verify} the Hessian of $f(\bmat{\vx \\ \vy}) = 
\trace( \bmat{ \vx  & \vy } \bmat{ \vx^T  \\ \vy^T } )$. Show the result of {\colorbox[rgb]{1.00,0.93,1.00}{\tt hessiancheck}} for a set of test matrices test points $\vx$. Be adversarial in your choice. Analyze any erratic behavior your find.



\end{enumerate}
\hypertarget{problem_2_accuracy_of_finite_differences_3}{}\subsection*{{Problem 2: Accuracy of finite differences}}\label{problem_2_accuracy_of_finite_differences_3}

In class, your professor mentioned that using a step size of the square root of machine precision ($\sqrt{2^{-52}} \approx 10^{-8}$) was a reasonable compromise when computing finite differences. The tension is between the error in the function evaluation and the error in the finite difference approximation.

\begin{enumerate}%
\item For the function $f(x) = \frac{1}{2} \vx^T \vx$, show (i.e. use matlab to compute) how the error varies in a forward finite difference approximation of the \emph{gradient} when $\vx \in \RR^{10}$ as the step-length approximation changes from $h \approx 2^{-52}$ to $h \approx 2^{-8}$. What do you observe?


\item Does this change if you switch to a centered finite difference formula?



\end{enumerate}
\hypertarget{problem_3_steepest_descent_4}{}\subsection*{{Problem 3: Steepest descent}}\label{problem_3_steepest_descent_4}

(Nocedal and Wright, Exercise 3.6) Let'{}s conclude with a quick problem to show that steepest descent can converge very rapidly! Consider the steepest descent method with exact line search for the function $f(\vx) = (1/2) \vx^T \mQ \vx 
- \vx^T \vb$. Suppose that we know $\vx_0 - \vx^*$ is parallel to an eigenvector of $\mQ$. Show that the method will converge in a single iteration.


\end{document}
