Matrix and Vector Calculus

Diagonal

Prop For any matrix $A\in {\mathbb{R}}^{n\times n}$, pre-multiply by a diagonal matrix scales its the rows, while post-multiply by a diagonal matrix scales its the columns.

Inverse

Def Pseudo-Inverse The Pseudo-Inverse of some matrix $X$ is defined as $X^{†}=(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}$. Specifically, if $X$ is invertible, we have $X^{†}=X^{-1}$

Trace

Def Trace The trace of a square matrix $A\in {\mathbb{R}}^{n\times n}$ is the sum of diagonal elements: $\mathrm{tr}\left(A\right)={\sum }_{i=1}^n{A}_{ii}$

Prop Trace satisfies the following properties:

• $\mathrm{tr}(A)=\mathrm{tr}(\text{tr}A)$.
• $\mathrm{tr}\left(\alpha A+\beta B\right)=\alpha \mathrm{tr}\left(A\right)+\beta \mathrm{tr}\left(B\right)$.
• if $AB$ is square then $\text{trace}AB=\mathrm{tr}\left(BA\right)$.

Determinant

Prop Determinant satisfies the following properties:

• $\det I=1$
• $\det \alpha A={\alpha }^n\det A$
• form $B$ by swapping 2 rows of $A$, then $\det B=-\det A$
• $\det AB=\det A\det B$

Norm

Def Euclidean Norm For $x\in {\mathbb{R}}^n$, we define the Euclidean(L2) norm as

$$\Vert x{\Vert }_2=\sqrt{x_1^2+\cdots +x_n^2}=\sqrt{\text{tr}xx}$$

Factorizations

Thrm LU Factorization Every non-singular matrix $A\in {\mathbb{R}}^{n\times n}$ can be factored as $A=PLU$ where $P\in {\mathbb{R}}^{n\times n}$ is a permutation matrix, $L\in {\mathbb{R}}^{n\times n}$ is unit lower triangular, and $U\in {\mathbb{R}}^{n\times n}$ upper triangular and non-singular.

Thrm Cholesky Factorization Every symmetric positive definite matrix $A\in {\mathbb{R}}^{n\times n}$ can be factored as $A=L\text{tr}L$ where $L\in {\mathbb{R}}^{n\times n}$ is lower triangular and non-singular with positive diagonal elements.

Thrm Singular Value Decomposition Any matrix $A$ can be decomposed as $A=U\Sigma \text{tr}V$

Differentiation

Def Jacobian Suppose $f:{\mathbb{R}}^{m\times n}\to \mathbb{R}$ is a function. Then the gradient of $f(A)$ with respect to $A\in {\mathbb{R}}^{m\times n}$ is the matrix of partial derivatives called Jacobian:

$${\nabla }_{A}f(A)=\left[\begin{array}{cccc}\frac{\partial f}{\partial A_{11}}& \frac{\partial f}{\partial A_{12}}& \cdots & \frac{\partial f}{\partial A_{1n}}\\ \frac{\partial f}{\partial A_{21}}& \frac{\partial f}{\partial A_{22}}& \cdots & \frac{\partial f}{\partial A_{2n}}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial A_{m1}}& \frac{\partial f}{\partial A_{m2}}& \cdots & \frac{\partial f}{\partial A_{mn}}\end{array}\right]$$

Def Hessian Suppose that $f:{\mathbb{R}}^n\to \mathbb{R}$ is a function. Then the hessian of $f(x)$ with respect to $x\in {\mathbb{R}}^n$ is the $n\times n$ matrix of second partial derivatives:

$${\nabla }_x^{2}f(x)=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$

Prop Hessian is symmetric.

Thrm Differentiation Laws

• Constant: $\partial C=0$
• Addition: $\partial (\text{X}\pm \text{Y})=\partial \text{X}\pm \text{Y}$
• Matrix multiplication: $\partial (\text{X}\text{Y})=(\partial \text{X})\text{Y}+\text{X}(\partial \text{Y})$
• Element-wise multiplication: $\partial (\text{X}\odot \text{Y})=(\partial \text{X})\odot \text{Y}+\text{X}\odot (\partial \text{Y})$
• Transpose: $\partial ({\text{X}}^{\top })=(\partial \text{X}{)}^{\top }$
• Inverse: $\partial {\text{X}}^{-1}=-{\text{X}}^{-1}(\partial \text{X}){\text{X}}^{-1}$
• Trace: $\partial (\mathrm{tr}(\text{X}))=\mathrm{tr}(\partial \text{X})$
• Derivative: $\text{dd}f=\mathrm{tr}\left(\text{tr}\left(\text{pddf}fX\right)\text{dd}\text{X}\right)$
• Determinant: $\partial |\text{X}|=\mathrm{tr}({\text{X}}^{\#}\partial \text{X})$
  • Specially, if $\text{X}$ is invertible, $\partial |\text{X}|=|\text{X}|\mathrm{tr}({\text{X}}^{-1}\partial \text{X})$.

e.g. Suppose $a\in {\mathbb{R}}^m,b\in {\mathbb{R}}^n,X\in {\mathbb{R}}^{m\times n}$. Consider $f(X)=\text{tr}aXb$. Then we have

$$\text{dd}f=\text{tr}a(\text{dd}X)b=\text{trace}\text{tr}a(\text{dd}X)b=\text{trace}b\text{tr}a(\text{dd}X)⟹\text{pddf}fX=\text{tr}(b\text{tr}a)=a\text{tr}b$$

Block Matrix

Thrm Block matrix Determinant

$$\begin{array}{rl}& \det \left[\begin{array}{cc}A& 0\\ C& D\end{array}\right]=\det (A)\det (D)=\det \left[\begin{array}{cc}A& B\\ 0& D\end{array}\right]\\ \text{ A is invertible:}& \det \left[\begin{array}{cc}A& B\\ C& D\end{array}\right]=\det (A)\det \left(D-C{A}^{-1}B\right)\\ \text{C and D commute:}& \det \left[\begin{array}{cc}A& B\\ C& D\end{array}\right]=\det (AD-BC)\\ & \det \left[\begin{array}{cc}A& B\\ B& A\end{array}\right]=\det (A-B)\det (A+B)\end{array}$$

Thrm Block Matrix Inverse

$${\left[\begin{array}{cc}\mathbf{A}& \mathbf{B}\\ \mathbf{C}& \mathbf{D}\end{array}\right]}^{-1}=\left[\begin{array}{cc}{\mathbf{A}}^{-1}+{\mathbf{A}}^{-1}\mathbf{B}(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B}{)}^{-1}\mathbf{C}{\mathbf{A}}^{-1}& -{\mathbf{A}}^{-1}\mathbf{B}(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B}{)}^{-1}\\ -{\left(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B}\right)}^{-1}\mathbf{C}{\mathbf{A}}^{-1}& {\left(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-1}\mathbf{B}\right)}^{-1}\end{array}\right]$$