$\newcommand{\eps}{\varepsilon} \newcommand{\kron}{\otimes} \DeclareMathOperator{\diag}{diag} \DeclareMathOperator{\trace}{trace} \DeclareMathOperator{\rank}{rank} \DeclareMathOperator*{\minimize}{minimize} \DeclareMathOperator*{\maximize}{maximize} \DeclareMathOperator{\subjectto}{subject to} \newcommand{\mat}[1]{\boldsymbol{#1}} \renewcommand{\vec}[1]{\boldsymbol{\mathrm{#1}}} \newcommand{\vecalt}[1]{\boldsymbol{#1}} \newcommand{\conj}[1]{\overline{#1}} \newcommand{\normof}[1]{\|#1\|} \newcommand{\onormof}[2]{\|#1\|_{#2}} \newcommand{\MIN}[2]{\begin{array}{ll} \minimize_{#1} & {#2} \end{array}} \newcommand{\MINone}[3]{\begin{array}{ll} \minimize_{#1} & {#2} \\ \subjectto & {#3} \end{array}} \newcommand{\MINthree}[5]{\begin{array}{ll} \minimize_{#1} & {#2} \\ \subjectto & {#3} \\ & {#4} \\ & {#5} \end{array}} \newcommand{\MAX}[2]{\begin{array}{ll} \maximize_{#1} & {#2} \end{array}} \newcommand{\MAXone}[3]{\begin{array}{ll} \maximize_{#1} & {#2} \\ \subjectto & {#3} \end{array}} \newcommand{\itr}[2]{#1^{(#2)}} \newcommand{\itn}[1]{^{(#1)}} \newcommand{\prob}{\mathbb{P}} \newcommand{\probof}[1]{\prob\left\{ #1 \right\}} \newcommand{\pmat}[1]{\begin{pmatrix} #1 \end{pmatrix}} \newcommand{\bmat}[1]{\begin{bmatrix} #1 \end{bmatrix}} \newcommand{\spmat}[1]{\left(\begin{smallmatrix} #1 \end{smallmatrix}\right)} \newcommand{\sbmat}[1]{\left[\begin{smallmatrix} #1 \end{smallmatrix}\right]} \newcommand{\RR}{\mathbb{R}} \newcommand{\CC}{\mathbb{C}} \newcommand{\eye}{\mat{I}} \newcommand{\mA}{\mat{A}} \newcommand{\mB}{\mat{B}} \newcommand{\mC}{\mat{C}} \newcommand{\mD}{\mat{D}} \newcommand{\mE}{\mat{E}} \newcommand{\mF}{\mat{F}} \newcommand{\mG}{\mat{G}} \newcommand{\mH}{\mat{H}} \newcommand{\mI}{\mat{I}} \newcommand{\mJ}{\mat{J}} \newcommand{\mK}{\mat{K}} \newcommand{\mL}{\mat{L}} \newcommand{\mM}{\mat{M}} \newcommand{\mN}{\mat{N}} \newcommand{\mO}{\mat{O}} \newcommand{\mP}{\mat{P}} \newcommand{\mQ}{\mat{Q}} \newcommand{\mR}{\mat{R}} \newcommand{\mS}{\mat{S}} \newcommand{\mT}{\mat{T}} \newcommand{\mU}{\mat{U}} \newcommand{\mV}{\mat{V}} \newcommand{\mW}{\mat{W}} \newcommand{\mX}{\mat{X}} \newcommand{\mY}{\mat{Y}} \newcommand{\mZ}{\mat{Z}} \newcommand{\mLambda}{\mat{\Lambda}} \newcommand{\mSigma}{\ensuremath{\mat{\Sigma}}} \newcommand{\mPbar}{\bar{\mP}} \newcommand{\ones}{\vec{e}} \newcommand{\va}{\vec{a}} \newcommand{\vb}{\vec{b}} \newcommand{\vc}{\vec{c}} \newcommand{\vd}{\vec{d}} \newcommand{\ve}{\vec{e}} \newcommand{\vf}{\vec{f}} \newcommand{\vg}{\vec{g}} \newcommand{\vh}{\vec{h}} \newcommand{\vi}{\vec{i}} \newcommand{\vj}{\vec{j}} \newcommand{\vk}{\vec{k}} \newcommand{\vl}{\vec{l}} \newcommand{\vm}{\vec{l}} \newcommand{\vn}{\vec{n}} \newcommand{\vo}{\vec{o}} \newcommand{\vp}{\vec{p}} \newcommand{\vq}{\vec{q}} \newcommand{\vr}{\vec{r}} \newcommand{\vs}{\vec{s}} \newcommand{\vt}{\vec{t}} \newcommand{\vu}{\vec{u}} \newcommand{\vv}{\vec{v}} \newcommand{\vw}{\vec{w}} \newcommand{\vx}{\vec{x}} \newcommand{\vy}{\vec{y}} \newcommand{\vz}{\vec{z}} \newcommand{\vpi}{\vecalt{\pi}} \newcommand{\vlambda}{\vecalt{\lambda}}$

Basic Notation

Let us begin by introducing basic notation for matrices and vectors.

Matrices

We'll use $\RR$ to denote the set of real-numbers and $\CC$ to denote the set of complex numbers.

We write the space of all $m \times n$ real-valued matrices as $\RR^{m \times n}$. Each Sometimes, I'll write: instead. With only a few exceptions, matrices are written as bold, capital letters. Sometimes, we'll use a capital greek letter. Matrix elements are written as sub-scripted, unbold letters.
When clear from context, instead, e.g. $A_{11}$ instead of $A_{1,1}$.

In class I'll usually write matrices with just upper-case letters. If you are unsure if something is a matrix or an element, raise your hand and ask, or quietly ask a neighbor.

Another notation for $\mA \in \RR^{m \times n}$ is
Sometimes this is nicer to write on the board.

Vectors

We write the set of length-$n$ real-valued vectors as $\RR^{n}$. Each Vectors are denoted by lowercase, bold letters. As with matrices, elements are sub-scripted, unbold letters. Sometimes, we'll write vector elements as Usually, this choice is motivated by a particular application. Throughout the class, vectors are column vectors.

In class I'll usually write vectors with just lower-case letters and will try to follow the convention of underlining them.

Scalars

Lower-case greek letters are scalars.

Quick test

Identify the following:

Operations

Transpose Let $\mA : m \times n$, then

Example $\mA = \sbmat{ 2 & 3 \\ 1 & 4 \\ 3 & -1 } \quad \mA^T = \sbmat{ 2 & 1 & 3 \\ 3 & 4 & -1 }$

Hermitian (Also called conjugate transpose) Let $\mA \in \CC^{m \times n}$, then

Example $\mA = \sbmat{ 2 & 3 \\ i & 4 \\ & 3 & -i } \quad \mA^* = \sbmat{ 2 & -i & 3 \\ 3 & 4 & i }$

Addition Let $\mA : m \times n$ and $\mB : m \times n$, then

Example $\mA = \sbmat{ 2 & 3 \\ 1 & 4 \\ 3 & -1 }, \mB = \sbmat{ 1 & -1 \\ 2 & 3 \\ -1 & 1 }$ $\mA + \mB = \sbmat{3 & -2 \\ 3 & 2 \\ 2 & 0 }$.

Scalar Multiplication Let $\mA : m \times n$ and $\alpha \in \RR$, then

Example $\mA = \sbmat{ 2 & 3 \\ 1 & 4 \\ 3 & -1 }, 5 \mA = \sbmat{ 10 & 15 \\ 5 & 20 \\ 15 & -5 }$

Matrix Multiplication Let $\mA : m \times n$ and $\mB : n \times k$, then

Matrix-vector Multiplication Let $\mA : m \times n$ and $\vx \in \RR^{n}$, then This operation is just a special case of matrix multiplication that follows from treating $\vx$ and $\vc$ as $n \times 1$ and $m \times 1$ matrices, respectively.

Vector addition, Scalar vector multiplication These are just special cases of matrix addition and scalar matrix multiplication where vectors are viewed as $n \times 1$ matrices.

Partitioning

It is often useful to represent a matrix as a collection of vectors. In this case, we write where each $\va_j \in \RR^{m}$. This form corresponds to a partition into columns.

Alternatively, we may wish to partition a matrix into rows. Here, each $\vr_i \in \RR^{n}$.

Using the column partitioning: And with the row partitioning:

Another useful partitioned representation of a matrix is into blocks: or
Here, the sizes "just have to work out" in the vernacular. Formally, all $\mA_{i,\cdot}$ must have the same number of rows and all $\mA_{\cdot,j}$ must have the same number of columns.