Search for probability and statistics terms on Statlect
StatLect
Index > Matrix algebra

Norm of a vector

by , PhD

The norm is a function, defined on a vector space, that associates to each vector a measure of its length. In abstract vector spaces, it generalizes the notion of length of a vector in Euclidean spaces.

There is a tight connection between norms and inner products, as every inner product can be used to induce a norm on its space.

Before reading this lecture you should be familiar with inner products and with the basics of the algebra of complex vectors and matrices.

Table of Contents

Definition of norm

We are going to give an abstract, axiomatic definition of norm. Later we will show some examples of norm to clarify its meaning.

Definition Let $S$ be a vector space. A norm on $S$ is a function [eq1] that associates to each $sin S$ a positive real number, denoted by [eq2], which has the following properties.

  1. Definiteness:[eq3]

  2. Absolute homogeneity:[eq4]where F is the field over which the vector space is defined (i.e., the set of scalars used for scalar multiplication); [eq5] denotes the absolute value if $F=U{211d} $ and the modulus if $F=U{2102} $.

  3. Triangle inequality:[eq6]

These properties are pretty intuitive.

As the norm is a measure of the length of a vector, it is reasonable to require that it should always be a positive number.

The definiteness property imposes that all vectors except the zero vector should have a strictly positive length.

Absolute homogeneity means that if you scale up (or down) a vector by a factor $lpha $, then its length is re-scaled accordingly.

Finally, in the triangle inequality, $r+s$ is to be interpreted as the side of a triangle, while $r$ and $s$ are the other two sides. A well-known fact from geometry is that the length of one side of a triangle is less than the sum of the lengths of the other two sides. The triangle inequality axiom extends this property to the notion of length in an abstract vector space.

Every inner product induces a norm

Before providing some examples of normed vector spaces, we need to introduce an important connection between inner products and norms.

Definition Let $S$ be a vector space and [eq7] an inner product on $S$. Then, the function [eq8] defined by[eq9]is called an induced norm on $S$.

We will prove that an induced norm indeed satisfies all the properties of a norm. But before doing that, we need to prove some preliminary results.

We highly recommend reading the proofs of the next three propositions as not only they are very short and simple to understand, but also they are a good way to hone our skills in dealing with norms and inner products.

Pythagoras' theorem

In order to understand the following generalization of the well-known Pythagoras' theorem, we need to remember that two vectors are said to be orthogonal if and only if their inner product is equal to zero.

Proposition Let $S$ be a vector space equipped with an inner product [eq10] and its induced norm [eq11]. If two vectors $r,sin S$ are orthogonal (i.e., [eq12]), then[eq13]

Proof

The proof is as follows:[eq14]where: in step $ rame{A}$ we have used the additivity of the inner product; in step $ rame{B}$ we have used the orthogonality of $r$ and $s$; in step $ rame{C}$ we have used the definition of induced norm.

Pythagoras' theorem says that the squared length of the hypotenuse ([eq15]) of a right triangle is equal to the sum of the squared lengths of the other two sides of the triangle ([eq16]). The triangle is right because we assume orthogonality (a 90 degrees angle) between the two sides $r $ and $s$.

Orthogonal decomposition

Given two vectors, we can always write the first as a scalar multiple of the second plus a third vector orthogonal to the second.

Proposition Let $S$ be a vector space equipped with an inner product [eq17] and its induced norm [eq11]. Given two vectors $r,sin S$, if $s eq 0$, then there exist a scalar [eq19] and a vector [eq20]such that[eq21]

Proof

It suffices to verify the first equality:[eq22]and the second one:[eq23]where: in step $ rame{A}$ we have used the additivity and homogeneity in the first argument of the inner product; in step $ rame{B}$ we have used the definition of induced norm.

Cauchy-Schwarz inequality

The inequality presented in the next proposition is known as Cauchy-Schwarz inequality.

Proposition Let $S$ be a vector space equipped with an inner product [eq24] and its induced norm [eq11]. For any two vectors $r,sin S$, the following inequality holds:[eq26]

Proof

If $s=0$, then [eq27], [eq28] and, as a consequence, the inequality holds. We can thus focus on the case in which $s eq 0$. In this case, we can use the orthogonal decomposition shown in the previous section:[eq29]where $u$ is orthogonal to $s$. Thanks to the orthogonality of $u$, we can apply Pythagoras' theorem:[eq30]Since [eq31], the equality just derived implies that[eq32]which becomes, by taking square roots on both sides,[eq33]If we multiply both sides by [eq34], we obtain the desired inequality.

A couple of things to remember about complex numbers

We now have almost all the tools that we need to prove that a norm induced by an inner product indeed satisfies all the properties of a norm.

We just need to remember a couple of facts about complex numbers.

First, if we add to a complex number[eq35]its complex conjugate[eq36]we get[eq37]where [eq38] is the so-called real part of the complex number.

Second, the modulus of $a+ib$ satisfies the inequality [eq39]because [eq40]

Proof that the induced norm is indeed a norm

We are now good to go!

Proposition Let $S$ be a vector space and [eq41] an inner product on $S$. Then, the norm induced by the inner product[eq9]is a norm on $S$.

Proof

Let us first prove positivity and definiteness. For any $sin S$, by the positivity of inner products, we have that[eq43]and by the definiteness of inner products, we have that[eq44]Now, let us prove absolute homogeneity. Suppose $S$ is defined over the complex field $U{2102} $. Then, for every [eq45], we have [eq46]where: in step $ rame{A}$ we have used the homogeneity in the first argument of the inner product; in step $ rame{B}$ we have used the conjugate homogeneity in the second argument of the inner product ($overline{lpha }$ is the complex conjugate of $lpha $); in step $ rame{C}$ [eq47] denotes the modulus of $lpha $. Absolute homogeneity holds also when the vector space is over the real field R because [eq48] in that case, so that [eq49] is the absolute value of $lpha $. Finally, let us prove the triangle inequality. For any $r,sin S$, we have that[eq50]where: in step $ rame{A}$ we have used the additivity of the inner product in its two arguments; in step $ rame{B}$ we have used the definition of induced norm; in step $ rame{C}$ we have used the conjugate symmetry of the inner product; in step $ rame{D}$ and $ rame{E}$ we have used the elementary properties of complex numbers discussed in the previous section; in step $ rame{F}$ we have used the Cauchy-Schwarz inequality. By taking the square root of both sides of the inequality derived above, we obtain[eq51]

Example: the norm of a real array

Wow! This was a long sequence of theoretical facts without ever presenting a single example. The wait was worthwhile because we can now make examples involving norms induced by inner products, which are the most practically relevant in linear algebra.

In this example we are going to discuss the norm of a Kx1 column vector having real entries.

Let $S$ be the space of all Kx1 real vectors (on the real field R).

In the lecture on inner products, we have explained that the most common way of defining an inner product between two real Kx1 vectors $r,sin S$ is[eq52]where $s^{intercal }$ is the transpose of $s$, [eq53] are the K entries of $r$ and [eq54] are the K entries of $s$.

The norm induced by this inner product is[eq55]

Thus, the norm of a real vector is equal to the square root of the sum of the squares of its entries. This is the ordinary way to compute the length of a vector in Euclidean space.

Example Define[eq56]Then[eq57]

Example: the norm of a complex array

Another important example is the norm of a Kx1 column vector having complex entries.

Let $S$ be the space of all Kx1 complex vectors (on the complex field $U{2102} $).

In the lecture on inner products, the inner product between two Kx1 vectors $r,sin S$ was defined to be[eq58]where $s^{st }$ is the conjugate transpose of $s$, [eq59] are the K entries of $r$ and [eq60] are the complex conjugates of the K entries of $s$.

The norm induced by this inner product is[eq61]where we have used the fact that the modulus [eq62] is equal to the square root of $s_{k}$ times its complex conjugate $overline{s_{k}}$.

Thus, the formula for the norm of a complex vector is not very different from the formula for real vectors: we just need to replace the absolute values of the entries of the vector with their moduli.

Example Define[eq63]Then[eq64]

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

Let $S$ be the space of all Kx1 real vectors.

For any $sin S$, define [eq65]where [eq66] are the K entries of $s$. Show that [eq67] is a norm.

Solution

The function [eq34] is positive since it is the sum of positive terms [eq62]. It is definite because [eq70] if and only if [eq71] for $k=1,ldots ,K$, which happens if and only if $s=0$. Absolute homogeneity is satisfied because[eq72]Note that for any couple of real numbers $a,bin U{211d} $, we have[eq73]Therefore, for any two vectors $r,sin S$[eq74]so that the triangle inequality holds. Thus, all the properties of a norm are satisfied.

Exercise 2

Let $S$ be the space of all Kx1 real vectors.

For any $sin S$, define [eq75]where [eq66] are the K entries of $s$. Show that [eq67] is a norm.

Solution

The function [eq34] is positive since it is the maximum over a set of positive terms [eq62]. It is definite because [eq80] if and only if [eq81] for $k=1,ldots ,K$, which happens if and only if $s=0$. Absolute homogeneity is satisfied because[eq82]Note that given two real numbers $a,bin U{211d} $, we have[eq73]For any couple of vectors $r,sin S$, we have[eq84]so that the triangle inequality holds. Thus, all the properties of a norm are satisfied.

How to cite

Please cite as:

Taboga, Marco (2021). "Norm of a vector", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/vector-norm.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.