Matrix Operations
Table of Contents
1. Matrix Addition
If we think of two matrices as linear transformations, matrix addition will only be defined if their codomain and domain are the same; in other words, \(A+B\) is only defined if \(A\) and \(B\) are the same size.
If \(A = \begin{bmatrix} a_{ij} \end{bmatrix}\) and \(B = \begin{bmatrix} b_{ij}\end{bmatrix}\),
\begin{align} \boxed{A + B = \begin{bmatrix} a_{ij} + b_{ij} \end{bmatrix}} \end{align}2. Matrix Multiplication
If \(A\) and \(B\) are matrices, then we can consider their associated linear transformations \(T\) and \(S\). We define matrix multiplication to be analogous to the composition of two transformations, i.e. \(T \circ S\).
This requires the codomain of \(S\) to be the domain of \(T\). In other words, if \(A\) is an \(m\times n\) matrix, B must be \(n\times p\).
We know that multiplying a matrix by a column vector is done by forming a new matrix where each entry in each row is the dot product between the corresponding row in the matrix and the column vector. For matrix-matrix multiplication, we extend that by considering \(B\) to be a matrix of columns \(b_1, b_2, \dots, b_p\). Then,
\begin{align} \boxed{AB = \begin{bmatrix} Ab_1 & Ab_2 & \cdots & Ab_p \end{bmatrix}} \end{align}2.1. Multiplication Properties
Matrix multiplication is associative:
\begin{align} A(BC) = (AB)C \end{align}However, matrix multiplication is not commutative:
\begin{align} AB \neq BA \end{align}We can understand this through the analogue of the composition of two transformation functions. It can be seen that given two functions \(f\) and \(g\), \(f \circ g \neq g \circ f\).
2.2. Identity
The \(n \times n\) identity matrix \(I_n\) is the matrix that implements the identity transformation:
\begin{align} I_n = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & & 0 \\ \vdots \\ 0 & & & 1 \end{bmatrix} \notag \end{align}The identity matrix has 1s on the diagonal and 0s everywhere else. It satisfies the following equality for any \(n \times n\) matrix \(A\):
\begin{align} A \cdot I_n = I_n \cdot A = A \end{align}2.3. Perspectives of Matrix Multiplication
Say we have an \(m \times n\) matrix \(A\) and a \(n \times p\) matrix \(B\), with product \(C=AB\) of size \(m \times p\). Then we have four equivalent views on this matrix multiplication:
2.3.1. Dot Product
Each entry \({\color{purple}c_{ij}}\) is the dot product of \({\color{blue}\text{row } i \text{ of } A}\) with \({\color{red}\text{column } j \text{ of } B}\):
\begin{align} {\color{purple}c_{ij}} = {\color{blue}\begin{bmatrix} a_{i1} & a_{i2} & \cdots & a_{in} \end{bmatrix}} {\color{red}\begin{bmatrix} b_{1j} \\ b_{2j} \\ \vdots \\ b_{nj} \end{bmatrix}} = \sum_{k=1}^{n} {\color{blue}a_{ik}}\,{\color{red}b_{kj}} \notag \end{align}2.3.2. Linear Combination of Columns
Each \({\color{purple}\text{column of } C}\) is a linear combination of the \({\color{blue}\text{columns of } A}\), with coefficients given by the corresponding \({\color{red}\text{column of } B}\). Writing \({\color{blue}\mathbf{a}_k}\) for column \(k\) of \({\color{blue}A}\) and \({\color{purple}\mathbf{c}_j}\) for column \(j\) of \({\color{purple}C}\):
\begin{align} {\color{purple}\mathbf{c}_j} = {\color{red}b_{1j}}\,{\color{blue}\mathbf{a}_1} + {\color{red}b_{2j}}\,{\color{blue}\mathbf{a}_2} + \cdots + {\color{red}b_{nj}}\,{\color{blue}\mathbf{a}_n} = \sum_{k=1}^{n} {\color{red}b_{kj}}\,{\color{blue}\mathbf{a}_k} \notag \end{align}For example, with \(n=3\) and \(p=2\):
\begin{align} {\color{blue}\begin{bmatrix} \mid & \mid & \mid \\ \mathbf{a}_1 & \mathbf{a}_2 & \mathbf{a}_3 \\ \mid & \mid & \mid \end{bmatrix}} {\color{red}\begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \\ b_{31} & b_{32} \end{bmatrix}} = \begin{bmatrix} \mid & \mid \\ {\color{red}b_{11}}{\color{blue}\mathbf{a}_1} + {\color{red}b_{21}}{\color{blue}\mathbf{a}_2} + {\color{red}b_{31}}{\color{blue}\mathbf{a}_3} & {\color{red}b_{12}}{\color{blue}\mathbf{a}_1} + {\color{red}b_{22}}{\color{blue}\mathbf{a}_2} + {\color{red}b_{32}}{\color{blue}\mathbf{a}_3} \\ \mid & \mid \end{bmatrix} \notag \end{align}2.3.3. Linear Combination of Rows
Each \({\color{purple}\text{row of } C}\) is a linear combination of the \({\color{red}\text{rows of } B}\), with coefficients given by the corresponding \({\color{blue}\text{row of } A}\). Writing \({\color{red}\mathbf{b}_{k*}}\) for row \(k\) of \({\color{red}B}\) and \({\color{purple}\mathbf{c}_{i*}}\) for row \(i\) of \({\color{purple}C}\):
\begin{align} {\color{purple}\mathbf{c}_{i*}} = {\color{blue}a_{i1}}\,{\color{red}\mathbf{b}_{1*}} + {\color{blue}a_{i2}}\,{\color{red}\mathbf{b}_{2*}} + \cdots + {\color{blue}a_{in}}\,{\color{red}\mathbf{b}_{n*}} = \sum_{k=1}^{n} {\color{blue}a_{ik}}\,{\color{red}\mathbf{b}_{k*}} \notag \end{align}For example, with \(m=2\) and \(n=3\):
\begin{align} {\color{blue}\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \end{bmatrix}} {\color{red}\begin{bmatrix} \text{---}\; \mathbf{b}_{1*} \;\text{---} \\ \text{---}\; \mathbf{b}_{2*} \;\text{---} \\ \text{---}\; \mathbf{b}_{3*} \;\text{---} \end{bmatrix}} = \begin{bmatrix} \text{---}\; {\color{blue}a_{11}}{\color{red}\mathbf{b}_{1*}} + {\color{blue}a_{12}}{\color{red}\mathbf{b}_{2*}} + {\color{blue}a_{13}}{\color{red}\mathbf{b}_{3*}} \;\text{---} \\ \text{---}\; {\color{blue}a_{21}}{\color{red}\mathbf{b}_{1*}} + {\color{blue}a_{22}}{\color{red}\mathbf{b}_{2*}} + {\color{blue}a_{23}}{\color{red}\mathbf{b}_{3*}} \;\text{---} \end{bmatrix} \notag \end{align}2.3.4. Sum of Outer Products
\({\color{purple}C}\) is the sum of \(n\) rank-1 matrices, where the k-th term is \({\color{blue}\text{column } k \text{ of } A}\) times \({\color{red}\text{row } k \text{ of } B}\):
\begin{align} {\color{purple}C} = \sum_{k=1}^{n} {\color{blue}\mathbf{a}_k} \, {\color{red}\mathbf{b}_{k*}} = {\color{blue}\mathbf{a}_1} {\color{red}\mathbf{b}_{1*}} + {\color{blue}\mathbf{a}_2} {\color{red}\mathbf{b}_{2*}} + \cdots + {\color{blue}\mathbf{a}_n} {\color{red}\mathbf{b}_{n*}} \notag \end{align}Each term is an \(m \times p\) rank-1 matrix:
\begin{align} {\color{blue}\begin{bmatrix} a_{1k} \\ a_{2k} \\ \vdots \\ a_{mk} \end{bmatrix}} {\color{red}\begin{bmatrix} b_{k1} & b_{k2} & \cdots & b_{kp} \end{bmatrix}} = \begin{bmatrix} {\color{blue}a_{1k}}{\color{red}b_{k1}} & {\color{blue}a_{1k}}{\color{red}b_{k2}} & \cdots & {\color{blue}a_{1k}}{\color{red}b_{kp}} \\ {\color{blue}a_{2k}}{\color{red}b_{k1}} & {\color{blue}a_{2k}}{\color{red}b_{k2}} & \cdots & {\color{blue}a_{2k}}{\color{red}b_{kp}} \\ \vdots & \vdots & \ddots & \vdots \\ {\color{blue}a_{mk}}{\color{red}b_{k1}} & {\color{blue}a_{mk}}{\color{red}b_{k2}} & \cdots & {\color{blue}a_{mk}}{\color{red}b_{kp}} \end{bmatrix} \notag \end{align}3. Matrix Transposition
The transpose \(A^T\) of a matrix \(A\) is \(A\) with the rows and columns switched. Thus, if \(A\) was a \(m \times n\) matrix, then \(A^T\) would be a \(n \times m\) matrix:
\begin{align} \boxed{A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \quad A^T = \begin{bmatrix} a_{11} & \cdots & a_{m1} \\ a_{12} & \cdots & a_{m2} \\ \vdots \\ a_{1n} & \cdots & a_{mn} \end{bmatrix}} \end{align}3.1. Transposition Properties
Two useful properties of transpose:
\begin{align} (A+B)^T &= A^T + B^T \\ (AB)^T &= B^TA^T \end{align}