mathematics-physics-wiki/docs/en/mathematics/linear-algebra/inner-product-spaces.md

Inner product spaces

Definition

An introduction of length in a vector space may be formulated in terms of an inner product space.

Definition 1: an inner product V is an operation on V that assigns, to each pair of vectors \mathbf{x},\mathbf{y} \in V a real number \langle \mathbf{x},\mathbf{y}\rangle satisfying the following conditions

1. \langle \mathbf{x},\mathbf{x}\rangle > 0, \text{ for } \mathbf{x} \in V\backslash\{\mathbf{0}\} \text{ and } \langle \mathbf{x},\mathbf{x}\rangle = 0, \; \text{for } \mathbf{x} = \mathbf{0},
2. \langle \mathbf{x},\mathbf{y}\rangle = \overline{\langle \mathbf{y},\mathbf{x}\rangle}, \; \forall \mathbf{x}, \mathbf{y} \in V,
3. \langle a \mathbf{x} + b \mathbf{y}, \mathbf{z}\rangle = a \langle \mathbf{x},\mathbf{z}\rangle + b \langle \mathbf{y},\mathbf{z}\rangle, \; \forall \mathbf{x}, \mathbf{y}, \mathbf{z} \in V \text{ and } a,b \in \mathbb{K}.

A vector space V with an inner product is called an inner product space.

Euclidean inner product spaces

The standard inner product on the Euclidean vector spaces V = \mathbb{R}^n with n \in \mathbb{N} is given by the scalar product defined by

\langle \mathbf{x},\mathbf{y}\rangle = \mathbf{x}^T \mathbf{y},

for all \mathbf{x},\mathbf{y} \in V.

??? note "Proof:"

Will be added later.

This can be extended to matrices V = \mathbb{R}^{m \times n} with m,n \in \mathbb{N} for which an inner product may be given by

\langle A, B\rangle = \sum_{i=1}^m \sum_{j=1}^n a_{ij} b_{ij},

for all A, B \in V.

??? note "Proof:"

Will be added later.

Function inner product spaces

Let V be a function space with a domain X. An inner product on V may be defined by

\langle f, g\rangle = \int_X \bar f(x) g(x) dx

for all f,g \in V.

??? note "Proof:"

Will be added later.

Polynomial inner product spaces

Let V be a polynomial space of degree n \in \mathbb{N} with the set of numbers \{x_i\}_{i=1}^n \subset \mathbb{K}^n. An inner product on V may be defined by

\langle p, q \rangle = \sum_{i=1}^n \bar p(x_i) q(x_i),

for all p,q \in V.

??? note "Proof:"

Will be added later.

Properties of inner product spaces

Definition 2: let V be an inner product space, the Euclidean length \|\mathbf{v}\| of a vector \mathbf{v} is defined as

|\mathbf{v}| = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle},

for all \mathbf{v} \in V.

Which is consistent with Euclidean geometry. According to definition 1 the distance between two vectors \mathbf{v}, \mathbf{w} \in V is \|\mathbf{v} - \mathbf{w}\|.

Definition 3: let V be an inner product space, the vectors \mathbf{u} and \mathbf{v} are orthogonal if

\langle \mathbf{u}, \mathbf{v} \rangle = 0,

for all \mathbf{u}, \mathbf{v} \in V.

A pair of orthogonal vectors will satisfy the theorem of Pythagoras.

Theorem 1: let V be an inner product space and \mathbf{u} and \mathbf{v} are orthogonal then

|\mathbf{u} + \mathbf{v}|^2 = |\mathbf{u}|^2 + |\mathbf{v}|^2,

for all \mathbf{u}, \mathbf{v} \in V.

??? note "Proof:"

let $V$ be an inner product space and let $\mathbf{u}, \mathbf{v} \in V$ be orthogonal, then

$$
\begin{align*}
    \|\mathbf{u} + \mathbf{v}\|^2 &= \langle \mathbf{u} + \mathbf{v}, \mathbf{u} + \mathbf{v}\rangle, \\
                                &= \langle \mathbf{u}, \mathbf{u} \rangle + 2 \langle \mathbf{u}, \mathbf{v} \rangle + \langle \mathbf{v}, \mathbf{v} \rangle, \\
                                &= \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2. 
\end{align*}
$$

Interpreted in \mathbb{R}^2 this is just the familiar Pythagorean theorem.

Definition 4: let V be an inner product space then the scalar projection a of \mathbf{u} onto \mathbf{v} is defined as

a = \frac{1}{|\mathbf{v}|} \langle \mathbf{u}, \mathbf{v} \rangle,

for all \mathbf{u} \in V and \mathbf{v} \in V \backslash \{\mathbf{0}\}.

The vector projection p of \mathbf{u} onto \mathbf{v} is defined as

\mathbf{p} = a \bigg(\frac{1}{|\mathbf{v}|} \mathbf{v}\bigg) = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\langle \mathbf{v}, \mathbf{v} \rangle} \mathbf{v},

for all \mathbf{u} \in V and \mathbf{v} \in V \backslash \{\mathbf{0}\}.

It may be observed that \mathbf{u} - \mathbf{p} and \mathbf{p} are orthogonal since \langle \mathbf{p}, \mathbf{p} \rangle = a^2 and \langle \mathbf{u}, \mathbf{p} \rangle = a^2 which implies

\langle \mathbf{u} - \mathbf{p}, \mathbf{p} \rangle = \langle \mathbf{u}, \mathbf{p} \rangle - \langle \mathbf{p}, \mathbf{p} \rangle = a^2 - a^2 = 0.

Additionally, it may be observed that \mathbf{u} = \mathbf{p} if and only if \mathbf{u} is a scalar multiple of \mathbf{v}; \mathbf{u} = b \mathbf{v} for some b \in \mathbb{K}. Since

\mathbf{p} = \frac{\langle b \mathbf{v}, \mathbf{v} \rangle}{\langle \mathbf{v}, \mathbf{v} \rangle} \mathbf{v} = b \mathbf{v} = \mathbf{u}.

Theorem 2: let V be an inner product space then

| \langle \mathbf{u}, \mathbf{v} \rangle | \leq | \mathbf{u} | | \mathbf{v} |,

is true for all \mathbf{u}, \mathbf{v} \in V. With equality only holding if and only if \mathbf{u} and \mathbf{v} are linearly dependent.

??? note "Proof:"

let $V$ be an inner product space and let $\mathbf{u}, \mathbf{v} \in V$. If $\mathbf{v} = \mathbf{0}$, then

$$
    | \langle \mathbf{u}, \mathbf{v} \rangle | = 0 = \| \mathbf{u} \| \| \mathbf{v} \|,
$$

If $\mathbf{v} \neq \mathbf{0}$, then let $\mathbf{p}$ be the vector projection of $\mathbf{u}$ onto $\mathbf{v}$. Since $\mathbf{p}$ is orthogonal to $\mathbf{u} - \mathbf{p}$ it follows that

$$
    \| \mathbf{p} \|^2 + \| \mathbf{u} - \mathbf{p} \|^2 = \| \mathbf{u} \|^2,
$$

thus 

$$
    \frac{1}{\|\mathbf{v}\|^2} \langle \mathbf{u}, \mathbf{v} \rangle^2 = \| \mathbf{p}\|^2 = \| \mathbf{u} \|^2 - \| \mathbf{u} - \mathbf{p} \|^2,
$$

and hence

$$
    \langle \mathbf{u}, \mathbf{v} \rangle^2 = \|\mathbf{u}\|^2 \|\mathbf{v}\|^2 - \|\mathbf{u} - \mathbf{p}\|^2 \|\mathbf{v}\|^2 \leq \|\mathbf{u}\|^2 \|\mathbf{v}\|^2,
$$

therefore

$$
    | \langle \mathbf{u}, \mathbf{v} \rangle | \leq \| \mathbf{u} \| \| \mathbf{v} \|.
$$

Equality holds if and only if $\mathbf{u} = \mathbf{p}$. From the above observations, this condition may be restated to linear dependence of $\mathbf{u}$ and $\mathbf{v}$. 

A consequence of the Cauchy-Schwarz inequality is that if \mathbf{u} and \mathbf{v} aer nonzero vectors in an inner product space then

-1 \leq \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{|\mathbf{u}| |\mathbf{v}|} \leq 1,

and hence there is a unique angle \theta \in [0, \pi] such that

\cos \theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{|\mathbf{u}| |\mathbf{v}|}.

Normed spaces

Definition 5: a vector space V is said to be a normed linear space if to each vector \mathbf{v} \in V there is associated a real number \| \mathbf{v} \| satisfying the following conditions

1. \|\mathbf{v}\| > 0, \text{ for } \mathbf{v} \in V\backslash\{\mathbf{0}\} \text{ and } \| \mathbf{v} \| = 0, \text{ for } \mathbf{v} = \mathbf{0},
2. \|a \mathbf{v}\| = |a| \|\mathbf{v}\|, \; \forall \mathbf{v} \in V \text{ and } a \in \mathbb{K},
3. \| \mathbf{v} + \mathbf{w}\| \geq \|\mathbf{v}\| + \| \mathbf{w}\|, \; \forall \mathbf{v}, \mathbf{w} \in V,

is called the norm of \mathbf{v}.

With the third condition, the triangle inequality.

Theorem 3: let V be an inner product space then

| \mathbf{v} | = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle},

for all \mathbf{v} \in V defines a norm on V.

??? note "Proof:"

Will be added later.

We therefore have that the Euclidean length (definition 2) is a norm, justifying the notation.