Search for probability and statistics terms on Statlect
StatLect
Index > Matrix algebra

Projection matrix

by , PhD

In linear algebra, a projection matrix is a matrix associated to a linear operator that maps vectors into their projections onto a subspace.

Table of Contents

Preliminary notions

Let us start by reviewing some notions that are essential for understanding projections.

Let $S$ be a linear space. Let $S_{1}$ and $S_{2}$ be subspaces of $S$.

Remember that the sum $S_{1}+S_{2}$ is the set[eq1]

When $S_{1}$ and $S_{2}$ have only the zero vector in common (i.e., [eq2]), then the sum is called a direct sum and it is denoted by $S_{1}oplus S_{2}$.

Moreover, when the direct sum is equal to the whole space, that is,[eq3]we say that the two spaces are complementary.

As we have proved, when $S_{1}$ and $S_{2}$ are complementary, any vector $s$ belonging to $S$ can be uniquely written as[eq4]where $s_{1}in S_{1}$ and $s_{2}in S_{2}$.

Projections

After revising all these notions, we are ready to define projections.

Definition Let $S$ be a linear space. Let $S_{1}$ and $S_{2}$ be complementary subspaces (i.e., [eq5]). Let $sin S$ with its unique decomposition[eq6]in which $s_{1}in S_{1}$ and $s_{2}in S_{2}$. Then, the vector $s_{1}$ is called the projection of $s$ onto $S_{1}$ along $S_{2}$, and the vector $s_{2}$ is called the projection of $s$ onto $S_{2}$ along $S_{1}$.

We note that the locutions "along $S_{1}$" and "along $S_{2}$" are needed because the complement of a given subspace is not necessarily unique. For example, there may be another subspace $S_{3}$ that is complementary to $S_{1}$. As a consequence, when we project a vector onto $S_{1}$, we need to specify whether we are considering $S_{2}$ or $S_{3}$ as a complement of $S_{1}$.

Example Let $S$ be the space of all real $3 imes 1$ vectors. Let $S_{1}$ be the space spanned by [eq7]which contains all the scalar multiples of $b_{1}$. Let $S_{2}$ be the space spanned by the two vectors [eq8]which contains all the linear combinations of $c_{1}$ and $c_{2}$. We have that [eq9] because no non-zero vector of $S_{1}$ can be written as a linear combination of $c_{1}$ and $c_{2}$. Therefore, [eq10]. Now, consider the vector [eq11]We have that[eq12]Thus, the unique decomposition of $s$ is[eq13]where $s_{1}=2b_{1}$ and $s_{2}=2c_{1}+c_{2}$. The projection of $s$ onto $S_{1}$ along $S_{2}$ is [eq14]and the projection of $s$ onto $S_{2}$ along $S_{1}$ is [eq15]

Oblique projections

Projections as defined above are also sometimes called oblique projections in order to distinguish them from orthogonal projections, which are a particular kind of projection in which the two complementary subspaces $S_{1} $ and $S_{2}$ are orthogonal complements.

Projection operator

We now define projection operators.

Definition Let $S$ be a linear space and $S_{1}$ and $S_{2}$ two subspaces such that [eq16]. The function [eq17] that associates to each $sin S$ its projection onto $S_{1}$ along $S_{2}$ is called the projection operator onto $S_{1}$ along $S_{2}$.

The first important property of the projection operator is that it is a linear operator, that is, it preserves addition and multiplication by scalars.

Proposition The projection operator onto $S_{1}$ along $S_{2}$ is a linear operator.

Proof

Arbitrarily choose two vectors $s,tin S$. They have the unique decompositions[eq18]where [eq19] and [eq20]. Denote by $P_{S_{1},S_{2}}$ the projection operator onto $S_{1}$ along $S_{2}$. Then,[eq21]Take any two scalars $lpha $ and $eta $ and consider the linear combination[eq22]Then,[eq23]Therefore,[eq24]Since $s$, $t$, $lpha $ and $eta $ were arbitrary, the latter equality implies that the projection operator is linear.

A couple of observations are in order:

Matrix of the projection operator

Let [eq25] be a basis for $S$. Any vector $sin S$ can be represented by its coordinate vector with respect to $B$, denoted by [eq26]. If $s$ can be written as a linear combination of the basis as[eq27]then [eq28]

Moreover, any linear operator $f:S ightarrow S$ can be represented by a square matrix, called matrix of the operator with respect to $B$ and denoted by [eq29], such that[eq30]

In the case of a projection operator $P_{S_{1},S_{2}}$, this implies that there is a square matrix [eq31] that, once post-multiplied by the coordinates [eq32] of a vector $s$, gives the coordinates of the projection of $s$ onto $S_{1}$ along $S_{2}$. Such a matrix is called a projection matrix (or a projector).

Definition The matrix of a projection operator with respect to a given basis is called a projection matrix.

How to derive the projection matrix

Now that we know what a projection matrix is, we can learn how to derive it.

In the lecture on complementary subspaces we have shown that, if [eq33] is a basis for $S_{1}$, [eq34] is a basis for $S_{2}$, and [eq35]then $Bcup C$ is a basis for $S$.

For the sake of legibility, denote the projection $P_{S_{1},S_{2}}$ simply by $P$ in what follows.

Note that $P$ projects:

By applying the general rule for deriving the matrix of a linear operator, we obtain that[eq36]where I is the $K imes K$ identity matrix and the other blocks are zero matrices (in particular, the diagonal one is $L imes L$).

In step $ rame{A}$ we have used the fact that the coordinate vector of $b_{k}$ with respect to the basis $Bcup C$ (to which $b_{k}$ itself belongs, occupying the k-th position) is a vector that has a single entry equal to 1 (the k-th) and all the other entries equal to 0.

Thus, the projection $P$ has an extremely simple structure: when we use it to project a vector on $S_{1}$, we leave the coordinates corresponding to the basis of $S_{1}$ unchanged and we set all the other coordinates to zero.

However, in most cases we are not so lucky as to have coordinates already expressed with respect to $Bcup C$. In such cases, we need to perform a change of basis (please revise how it works here).

Suppose that the basis used to express coordinates is E. Then, the change-of-basis from $Bcup C$ to E is[eq37]

The matrix of the projection operator with respect to the basis E is[eq38]

Example As in the previous example, we consider the space $S$ of all real $3 imes 1$ vectors. Coordinates are naturally expressed with respect to the canonical basis [eq39]where[eq40]The basis of $S_{1}$ is [eq41] where[eq42]and the basis of $S_{2}$ is [eq43] where[eq44]We have already argued that [eq45]. As before, denote $P_{S_{1},S_{2}}$ (the projection operator onto $S_{1}$ along $S_{2}$) simply by $P$. First of all, we have that [eq46]because the projection operator preserves the first coordinate and annihilates the other two (when coordinates are expressed with respect to [eq47]). The change-of-basis matrix that fits our purposes is[eq48]Its inverse is[eq49]The projection matrix under the canonical basis is[eq50]Let us compute the projection onto $S_{1}$ of the vector[eq11]We have done it already in the previous exercise, but this time we can use the projection matrix:[eq52]which is the same result we have derived previously.

Complementary projector

Once we have derived the projection matrix [eq53] that allows us to project vectors onto $S_{1}$, it is very easy to derive the matrix [eq54] that allows us to project vectors onto the complementary subspace $S_{2}$.

If a vector $s$ is decomposed as [eq55]then we can write the projection onto $S_{2}$ as[eq56]and its coordinates as[eq57]

Thus, the matrix of the projection operator onto $S_{2}$, sometimes called complementary projector, is[eq58]

In the derivation above we have also seen that[eq59]

Thus, we have that[eq60]

A matrix is idempotent iff it is a projection matrix

A square matrix A is said to be idempotent if and only if it is equal to its square:[eq61]

It turns out that idempotent matrices and projection matrices are the same thing!

Proposition A matrix is idempotent if and only if it is a projection matrix.

Proof

Let us prove the "if part". We start from the hypothesis that A is a projection matrix. As such, it is the matrix of a projection operator $P$ with respect to some basis E, that is, [eq62]. As proved above, we have that[eq63]Therefore,[eq64]which proves that A is idempotent. Let us now prove the "only if" part, starting from the hypothesis that A is idempotent. Suppose that A is $K imes K$. Let $S$ be the space of all Kx1 vectors. Define the two subspaces[eq65]and[eq66]In other words, $S_{1}$ and $S_{2}$ are the range and kernel of the operator defined by the matrix A. We have that[eq67]since any vector $sin S$ can be written as[eq68]where: $Asin S_{1}$ and [eq69] because[eq70]Suppose that a vector $t$ belongs to both $S_{1}$ and $S_{2}$. Since $tin S_{1}$, there exists $sin S$ such that[eq71]We can pre-multiply both sides by A and obtain[eq72]Since $tin S_{2}$, we have that $At=0$. As a consequence, $As=0$ by equation (3), and $t=0$ by equation (2). Therefore,[eq73]Hence,[eq74]From equation (1) we know that A projects $s$ into its component $Asin S_{1}$. Therefore, it is the matrix of the projection operator that projects vectors of $S$ into $S_{1}$ along $S_{2}$. Hence, A is a projection matrix.

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

Consider the projection problem analyzed in the previous two examples, where we have already derived the projection matrix of the projection operator onto $S_{1}$. Derive the complementary projection matrix (onto $S_{2}$) and use it to find the projection onto $S_{2}$ of the vector[eq75]

Solution

Define [eq76]. We have that[eq77]The projection of $s$ onto $S_{2}$ is[eq78]

How to cite

Please cite as:

Taboga, Marco (2021). "Projection matrix", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/projection-matrix.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.