Vector Spaces

If that line does go through \(\mathbf{0}\), then we can multiply points on the line by any number \(c\) and still stay within the line. We can also add points on the line without ever leaving the line. This line through \(\mathbf{0}\) in \(\mathbb{R}^n\) is an example of a subspace: a vector space inside another vector space.

In a vector space \(V\), say we have the set of vectors \({\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_p}\). Then the set of all linear combinations of those vectors is a subspace of \(V\).

The column space of a matrix \(A\) is the subspace spanned by the columns of \(A\), denoted \(\text{Col }A\). In other words, it consists of all linear combinations of the columns. To solve the equation \(A\mathbf{x} = \mathbf{b}\) is to express \(\mathbf{b}\) as a combination of the columns — the system is consistent if and only if \(\mathbf{b}\) is in the column space of \(A\).

The row space is the subspace spanned by the rows of \(A\), denoted \(\text{Row }A\). Since we usually work with column vectors, we can also say that the row space of \(A\) is the column space of the transpose matrix \(A^T\).

The nullspace of \(A\) is the space consisting of all the solutions \(\mathbf{x}\) to the homogeneous system \(A\mathbf{x}=0\), denoted \(\text{Nul }A\). To get the nullspace, we can solve for \(A\mathbf{x}=0\) for parametric vector form, and then the vectors in that form span \(\text{Nul }A\).

We can see that the nullspace is a subspace with the following demonstration. Say \(\mathbf{x}\) and \(\mathbf{y}\) are in the nullspace of \(A\). Then \(\mathbf{x} + \mathbf{y}\) is also in the nullspace, since:

\begin{align} A(\mathbf{x}+\mathbf{y}) &= A\mathbf{x}+A\mathbf{y} \notag \\ &= 0 + 0 \notag \\ &= 0 \end{align}

Example: Nullspace as a subspace

Given the set of all vectors \((a, b, c, d)\) such that it satisfies the following two equations:

\begin{cases} a - 2b = 4c \\ 2a = c + 3d \end{cases}

We want to know if this set is a subspace. To do so, notice that we can rewrite the equations as:

\begin{align} a - 2b - 4c &= 0 \notag \\ 2a - c - 3d &= 0 \notag \end{align}

This is simply a system of linear equations with \(a, b, c, d\) as the variables, which means we can write this like so:

\begin{align} \begin{bmatrix} 1 & -2 & -4 & 0 \\ 2 & 0 & -1 & -3 \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \notag \end{align}

Now, it is clear that \((a, b, c, d)\) is the nullspace, and so therefore it is a subspace.

To find the basis for \(\text{Nul }A\), we can use the REF/RREF to locate free variables, solve \(A\mathbf{x} = 0\), and the parametric vector form of the solution gives us the basis.

The basis for \(\text{Col }A\) is the set of the pivot columns of \(A\). To see why this is, consider the RREF of the matrix. Using RREF, we see that each pivot column is not a linear combination of the other pivot columns, as for any pivot column, the entry that has a \(1\) in that pivot column has a \(0\) in the corresponding entry of the other pivot coumns, and so the pivot columns themselves are linearly independent.

Now, take a non-pivot column in RREF:

\begin{align} \begin{bmatrix} a_1 \\ \vdots \\ a_m \\ 0 \\ \vdots \\ 0 \end{bmatrix} \notag \end{align}

This non-pivot column can be written as a linear combination of the other pivot columns, and as such the pivot columns of \(A\) form the basis for the column space.

The basis for \(\text{Row }A\) are the pivot rows in the row echelon form of \(A\).

In general, a one-to-one linear transformation from a vector space \(V\) onto \(W\) is called an isomorphism from \(V\) onto \(W\). In particular, any real vector space with a basis of \(n\) vectors is isomorphic with \(\mathbb{R}^n\).

If \(A\) is a real \(m\times n\) matrix, we have:

\begin{align} \boxed{\text{rank }A + \text{nullity }A = n} \end{align}

We can use this to derive the following important fact, for an \(m \times n\) matrix \(A\):

\begin{align} \text{rank} (A^TA) = \text{rank }A \end{align}

To prove this, we know that:

\begin{align} \text{rank }A &= n - \text{dim}(\text{Nul}(A)) \notag \\ \text{rank}(A^TA) &= n - \text{dim}(\text{Nul}(A^TA)) \notag \end{align}

Thus, it is sufficient to show that \(\text{Nul }A = \text{Nul}(A^TA)\). First, it can be seen that if \(A\mathbf{x} = 0\), then \(A^TA\mathbf{x} = 0\). Then, we want to show that if \(A\mathbf{x} \neq 0\), then \(A^TA\mathbf{x} \neq 0\). To do this, set \(\mathbf{v} = A^TA\mathbf{x}\), and multiply both sides on the left by \(\mathbf{x}^T\):

\begin{align} \mathbf{x}^T\mathbf{v} &= \mathbf{x}^TA^TA\mathbf{x} \notag \\ &= (A\mathbf{x})^T(A\mathbf{x}) \notag \\ &= \left|A\mathbf{x}\right|^2 \notag \end{align}

Since the right hand side cannot be zero if \(A\mathbf{x} \neq 0\), this means that \(\mathbf{v}\) is nonzero, so \(A^TA\mathbf{x} \neq 0\). Thus, we have shown (5) to be true.