Finished section orthogonality.

docs/en/mathematics/linear-algebra/orthogonality.md

@@ -46,7 +46,7 @@ for all $\mathbf{v} \in S$, and hence $\mathbf{u}_1 + \mathbf{u}_2 \in S^\perp$.
 
 ### Fundamental subspaces
 
-Let $V$ be an Euclidean inner product space $V = \mathbb{R}^n$ with its inner product defined by the [scalar product](inner-product-spaces/#euclidean-inner-product-spaces). With this definition of the inner product on $V$ the following theorem may be posed.
+Let $V$ be an Euclidean inner product space $V = \mathbb{R}^n$ with its inner product defined by the [scalar product](../inner-product-spaces/#euclidean-inner-product-spaces). With this definition of the inner product on $V$ the following theorem may be posed.
 
 > *Theorem 1*: let $A$ be an $m \times n$ matrix, then
 >
@@ -118,7 +118,7 @@ Known as the fundamental theorem of linear algebra. Which can be used to prove t
         S^\perp = R(X^T)^\perp = N(X),
     $$
 
-    from the [rank nullity theorem](vector-spaces/#rank-and-nullity) it follows that
+    from the [rank nullity theorem](../vector-spaces/#rank-and-nullity) it follows that
 
     $$
         \dim S^\perp = \dim N(X) = n - r.
@@ -153,12 +153,38 @@ Known as the fundamental theorem of linear algebra. Which can be used to prove t
 
     since $\{\mathbf{v}_i\}_{i=1}^r$ and $\{\mathbf{v}_i\}_{i=r+1}^n$ are linearly independent, we must also have that $\{\mathbf{v}_i\}_{i=1}^n$ are linearly independent and therefore form a basis of $V$.
 
+We may further extend this with the notion of a direct sum.
+
+> *Definition 3*: if $U$ and $V$ are subspaces of a vector space $W$ and each $\mathbf{w} \in W$ can be written uniquely as
+>
+> $$
+>   \mathbf{w} = \mathbf{u} + \mathbf{v},
+> $$
+>
+> with $\mathbf{u} \in U$ and $\mathbf{v} \in V$ then $W$ is a **direct sum** of U and $V$ denoted by $W = U \oplus V$. 
+
+In the following theorem it will be posed that the direct sum of a subspace and its orthogonal complement make up the whole vector space, which extends the notion of theorem 2. 
+
+> *Theorem 3*: if $S$ is a subspace of the inner product space $V = \mathbb{R}^n$, then 
+>
+> $$
+>   V = S \oplus S^\perp. 
+> $$
+
+??? note "*Proof*:"
+
+    Will be added later.
+
+The following results emerge from these posed theorems.
+
 > *Proposition 1*: let $S$ be a subspace of $V$, then $(S^\perp)^\perp = S$. 
 
 ??? note "*Proof*:"
 
     Will be added later.
 
+Recall that the system $A \mathbf{x} = \mathbf{b}$ is consistent if and only if $\mathbf{b} \in R(A)$ since $R(A) = N(A^T)^\perp$ we have the following result.
+
 > *Proposition 2*: let $A \in \mathbb{R}^{m \times n}$ and $\mathbf{b} \in \mathbb{R}^m$, then either there is a vector $\mathbf{x} \in \mathbb{R}^n$ such that 
 >
 > $$
@@ -366,3 +392,76 @@ $$
 
 ## Least squares solutions of overdetermined systems
 
+A standard technique in mathematical and statistical modeling is to find a least squares fit to a set of data points. This implies that the sum of squares fo errors between the model and the data points are minimized. A least squares problem can generally be formulated as an overdetermined linear system of equations. 
+
+For a system of equations $A \mathbf{x} = \mathbf{b}$ with $A \in \mathbb{R}^{m \times n}$ with $m, n \in \mathbb{N}[m>n]$ and $\mathbf{b} \in \mathbb{R}^m$ then for each $\mathbf{x} \in \mathbb{R}^n$ a *residual* $\mathbf{r}: \mathbb{R}^n \to \mathbb{R}^m$ can be formed
+
+$$
+    \mathbf{r}(\mathbf{x}) = \mathbf{b} - A \mathbf{x}.
+$$
+
+The distance between $\mathbf{b}$ and $A \mathbf{x}$ is given by
+
+$$
+    \| \mathbf{b} - A \mathbf{x} \| = \|\mathbf{r}(\mathbf{x})\|,
+$$
+
+We wish to find a vector $\mathbf{x} \in \mathbb{R}^n$ for which $\|\mathbf{r}(\mathbf{x})\|$ will be a minimum. A solution $\mathbf{\hat x}$ that minimizes $\|\mathbf{r}(\mathbf{x})\|$ is a *least squares solution* of the system $A \mathbf{x} = \mathbf{b}$. Do note that minimizing $\|\mathbf{r}(\mathbf{x})\|$ is equivalent to minimizing $\|\mathbf{r}(\mathbf{x})\|^2$.
+
+> *Theorem 6*: let $S$ be a subspace of $\mathbb{R}^m$. For each $b \in \mathbb{R}^m$, there exists a unique $\mathbf{p} \in S$ that suffices
+>
+> $$
+>   \|\mathbf{b} - \mathbf{s}\| > \|\mathbf{b} - \mathbf{p}\|,
+> $$
+>
+> for all $\mathbf{s} \in S\backslash\{\mathbf{p}\}$ and $\mathbf{b} - \mathbf{p} \in S^\perp$. 
+
+??? note "*Proof*:"
+
+    Will be added later.
+
+If $\mathbf{p} = A \mathbf{\hat x}$ in $R(A)$ that is closest to $\mathbf{b}$ then it follows that
+
+$$
+    \mathbf{b} - \mathbf{p} = \mathbf{b} - A \mathbf{x} = \mathbf{r}(\mathbf{\hat x}),
+$$
+
+must be an element of $R(A)^\perp$. Thus, $\mathbf{\hat x}$ is a solution to the least squares problem if and only if 
+
+$$
+    \mathbf{r}(\mathbf{\hat x}) \in R(A)^\perp = N(A^T).
+$$
+
+Thus to solve for $\mathbf{\hat x}$ we have the *normal equations* given by
+
+$$
+    A^T A \mathbf{x} = A^T \mathbf{b}. 
+$$
+
+Uniqueness of $\mathbf{\hat x}$ can be obtained if $A^T A$ is nonsingular which will be posed in the following theorem.
+
+> *Theorem 7*: let $A \in \mathbb{R}^{m \times n}$ be an $m \times n$ matrix with rank $n$, then $A^T A$ is nonsingular. 
+
+??? note "*Proof*:"
+
+    Let $A \in \mathbb{R}^{m \times n}$ be an $m \times n$ matrix with rank $n$. Let $\mathbf{v}$ be a solution of 
+
+    $$
+        A^T A \mathbf{x} = \mathbf{0},
+    $$
+
+    then $A \mathbf{v} \in N(A^T)$, but we also have that $A \mathbf{v} \in R(A) = N(A^T)^\perp$. Since $N(A^T) \cap N(A^T)^\perp = \{\mathbf{0}\}$ it follows that 
+
+    $$
+        A\mathbf{v} = \mathbf{0},
+    $$
+
+    so $\mathbf{v} = \mathbf{0}$ by the nonsingularity of $A$. 
+
+It follows that 
+
+$$
+    \mathbf{\hat x} = (A^T A)^{-1} A^T \mathbf{b},
+$$
+
+is the unique solution of the normal equations for $A$ nonsingular and consequently, the unique least squares solution of the system $A \mathbf{x} = \mathbf{b}$.