From 0003318cb720eabaffbdf4f9c9898492948ee2d4 Mon Sep 17 00:00:00 2001 From: Luc Date: Sat, 20 Apr 2024 14:00:57 +0200 Subject: [PATCH] Finished section orthogonality. --- .../linear-algebra/orthogonality.md | 103 +++++++++++++++++- 1 file changed, 101 insertions(+), 2 deletions(-) diff --git a/docs/en/mathematics/linear-algebra/orthogonality.md b/docs/en/mathematics/linear-algebra/orthogonality.md index 2c5786b..3d23217 100644 --- a/docs/en/mathematics/linear-algebra/orthogonality.md +++ b/docs/en/mathematics/linear-algebra/orthogonality.md @@ -46,7 +46,7 @@ for all $\mathbf{v} \in S$, and hence $\mathbf{u}_1 + \mathbf{u}_2 \in S^\perp$. ### Fundamental subspaces -Let $V$ be an Euclidean inner product space $V = \mathbb{R}^n$ with its inner product defined by the [scalar product](inner-product-spaces/#euclidean-inner-product-spaces). With this definition of the inner product on $V$ the following theorem may be posed. +Let $V$ be an Euclidean inner product space $V = \mathbb{R}^n$ with its inner product defined by the [scalar product](../inner-product-spaces/#euclidean-inner-product-spaces). With this definition of the inner product on $V$ the following theorem may be posed. > *Theorem 1*: let $A$ be an $m \times n$ matrix, then > @@ -118,7 +118,7 @@ Known as the fundamental theorem of linear algebra. Which can be used to prove t S^\perp = R(X^T)^\perp = N(X), $$ - from the [rank nullity theorem](vector-spaces/#rank-and-nullity) it follows that + from the [rank nullity theorem](../vector-spaces/#rank-and-nullity) it follows that $$ \dim S^\perp = \dim N(X) = n - r. @@ -153,12 +153,38 @@ Known as the fundamental theorem of linear algebra. Which can be used to prove t since $\{\mathbf{v}_i\}_{i=1}^r$ and $\{\mathbf{v}_i\}_{i=r+1}^n$ are linearly independent, we must also have that $\{\mathbf{v}_i\}_{i=1}^n$ are linearly independent and therefore form a basis of $V$. +We may further extend this with the notion of a direct sum. + +> *Definition 3*: if $U$ and $V$ are subspaces of a vector space $W$ and each $\mathbf{w} \in W$ can be written uniquely as +> +> $$ +> \mathbf{w} = \mathbf{u} + \mathbf{v}, +> $$ +> +> with $\mathbf{u} \in U$ and $\mathbf{v} \in V$ then $W$ is a **direct sum** of U and $V$ denoted by $W = U \oplus V$. + +In the following theorem it will be posed that the direct sum of a subspace and its orthogonal complement make up the whole vector space, which extends the notion of theorem 2. + +> *Theorem 3*: if $S$ is a subspace of the inner product space $V = \mathbb{R}^n$, then +> +> $$ +> V = S \oplus S^\perp. +> $$ + +??? note "*Proof*:" + + Will be added later. + +The following results emerge from these posed theorems. + > *Proposition 1*: let $S$ be a subspace of $V$, then $(S^\perp)^\perp = S$. ??? note "*Proof*:" Will be added later. +Recall that the system $A \mathbf{x} = \mathbf{b}$ is consistent if and only if $\mathbf{b} \in R(A)$ since $R(A) = N(A^T)^\perp$ we have the following result. + > *Proposition 2*: let $A \in \mathbb{R}^{m \times n}$ and $\mathbf{b} \in \mathbb{R}^m$, then either there is a vector $\mathbf{x} \in \mathbb{R}^n$ such that > > $$ @@ -366,3 +392,76 @@ $$ ## Least squares solutions of overdetermined systems +A standard technique in mathematical and statistical modeling is to find a least squares fit to a set of data points. This implies that the sum of squares fo errors between the model and the data points are minimized. A least squares problem can generally be formulated as an overdetermined linear system of equations. + +For a system of equations $A \mathbf{x} = \mathbf{b}$ with $A \in \mathbb{R}^{m \times n}$ with $m, n \in \mathbb{N}[m>n]$ and $\mathbf{b} \in \mathbb{R}^m$ then for each $\mathbf{x} \in \mathbb{R}^n$ a *residual* $\mathbf{r}: \mathbb{R}^n \to \mathbb{R}^m$ can be formed + +$$ + \mathbf{r}(\mathbf{x}) = \mathbf{b} - A \mathbf{x}. +$$ + +The distance between $\mathbf{b}$ and $A \mathbf{x}$ is given by + +$$ + \| \mathbf{b} - A \mathbf{x} \| = \|\mathbf{r}(\mathbf{x})\|, +$$ + +We wish to find a vector $\mathbf{x} \in \mathbb{R}^n$ for which $\|\mathbf{r}(\mathbf{x})\|$ will be a minimum. A solution $\mathbf{\hat x}$ that minimizes $\|\mathbf{r}(\mathbf{x})\|$ is a *least squares solution* of the system $A \mathbf{x} = \mathbf{b}$. Do note that minimizing $\|\mathbf{r}(\mathbf{x})\|$ is equivalent to minimizing $\|\mathbf{r}(\mathbf{x})\|^2$. + +> *Theorem 6*: let $S$ be a subspace of $\mathbb{R}^m$. For each $b \in \mathbb{R}^m$, there exists a unique $\mathbf{p} \in S$ that suffices +> +> $$ +> \|\mathbf{b} - \mathbf{s}\| > \|\mathbf{b} - \mathbf{p}\|, +> $$ +> +> for all $\mathbf{s} \in S\backslash\{\mathbf{p}\}$ and $\mathbf{b} - \mathbf{p} \in S^\perp$. + +??? note "*Proof*:" + + Will be added later. + +If $\mathbf{p} = A \mathbf{\hat x}$ in $R(A)$ that is closest to $\mathbf{b}$ then it follows that + +$$ + \mathbf{b} - \mathbf{p} = \mathbf{b} - A \mathbf{x} = \mathbf{r}(\mathbf{\hat x}), +$$ + +must be an element of $R(A)^\perp$. Thus, $\mathbf{\hat x}$ is a solution to the least squares problem if and only if + +$$ + \mathbf{r}(\mathbf{\hat x}) \in R(A)^\perp = N(A^T). +$$ + +Thus to solve for $\mathbf{\hat x}$ we have the *normal equations* given by + +$$ + A^T A \mathbf{x} = A^T \mathbf{b}. +$$ + +Uniqueness of $\mathbf{\hat x}$ can be obtained if $A^T A$ is nonsingular which will be posed in the following theorem. + +> *Theorem 7*: let $A \in \mathbb{R}^{m \times n}$ be an $m \times n$ matrix with rank $n$, then $A^T A$ is nonsingular. + +??? note "*Proof*:" + + Let $A \in \mathbb{R}^{m \times n}$ be an $m \times n$ matrix with rank $n$. Let $\mathbf{v}$ be a solution of + + $$ + A^T A \mathbf{x} = \mathbf{0}, + $$ + + then $A \mathbf{v} \in N(A^T)$, but we also have that $A \mathbf{v} \in R(A) = N(A^T)^\perp$. Since $N(A^T) \cap N(A^T)^\perp = \{\mathbf{0}\}$ it follows that + + $$ + A\mathbf{v} = \mathbf{0}, + $$ + + so $\mathbf{v} = \mathbf{0}$ by the nonsingularity of $A$. + +It follows that + +$$ + \mathbf{\hat x} = (A^T A)^{-1} A^T \mathbf{b}, +$$ + +is the unique solution of the normal equations for $A$ nonsingular and consequently, the unique least squares solution of the system $A \mathbf{x} = \mathbf{b}$. \ No newline at end of file