An Introduction to Analysis/Appendix

The complex number associated to a pair of vectors x and y is called the inner product of $x$ and $y$, denoted by $(x|y)$ with the following properties:

• $(x|y)=\overline{(y|x)}$ (Hermitian property)
• $(x+z|y)=(x|y)+(z|y)$ and $(x|y+z)=(x|y)+(x|z)$ (distribution law)
• $(\alpha x|y)=\alpha (x|y)$ and $(x|\alpha y)=\overline{\alpha }(x|y)$
• $(x|x)\ge 0$ where the equality holds if and only if $x=0$.

Note if $(x|y)$ is real, then $(x|y)=(y|x)$ (symmetric relation); e.g., $(x|y)=0$ and to mean this we say $x$ is orthogonal to $y$.

The most important example is an inner-product of the form $\sum (z_j|w_j)=z_j{\overline{w}}_j$ if $z=\sum z_j{e}_j$, $w=\sum w_j{e}_j$. Indeed,

$(z|w)=\sum z_j{\overline{w}}_j=\overline{\sum w_j{\overline{z}}_j}=\overline{(w|z)}$

$(z+s|w)=\sum (z_j+s_j){\overline{w}}_j=(z|w)+(s|w)$.

In the next chapter we will see $\sum$ is replaced by $\int$ and that still gives inner-products. To give another example, let u be fixed, and $f(x)=(x|u)u$. If $\Vert u\Vert =1$, then f is a projection; i.e., $f\circ f=f$.

We know that $\Vert z\Vert =z\overline{z}$, and this naturally leads to how to define a norm in terms of an inner product and this results in a normed space called an inner product space. We let $\Vert x\Vert =(x|x{)}^{2^{-}1}$ and check if it is indeed a norm by showing the following.

Theorem (angle between) If $x$ and $y$ are in an inner product space, then

$\Vert x+y{\Vert }^2\le \Vert x{\Vert }^2+\Vert y{\Vert }^2+2\Vert x\Vert \Vert y\Vert \cos \theta$

where $\text{Re }(x|y)=|(x|y)|\cos \theta \le \Vert x\Vert \Vert y\Vert \cos \theta$ (Schwarz inequality), and

$\theta$	$=0$ (pythagorean law) if $(x|y)=0$.
	$=\pi /2$ modulo $\pi$ if and only if $|(x|y)|=\Vert x\Vert \Vert y\Vert$

Proof: Otherwise the theorem holds trivially, we suppose $x$ and $y$ are nonzero. Let $z=x\Vert x{\Vert }^{-1}$ and $w=y\Vert y{\Vert }^{-1}$. Then we have:

$0$	$\le (z-w|z-w)=(z|z)-(z|w)-(w|z)+(w|w)$
	$=\Vert z{\Vert }^2+\Vert w{\Vert }^2-2\text{Re }(z|w)=2-2\text{Re }(z|w)$

Thus, $|(z|w)|cos\theta =\text{Re }(z|w)\le 1$ and $\text{Re }(x|y)\le \Vert x\Vert \Vert y\Vert$. Finally,

$\Vert x+y{\Vert }^2$	$=(x+y|x+y)=(x|x)+(x|y)+(y|x)+(y|y)$
	$=\Vert x{\Vert }^2+\Vert y{\Vert }^2+2\text{Re }(x|y)$
	$\le \Vert x{\Vert }^2+\Vert y{\Vert }^2+2\Vert x\Vert \Vert y\Vert \cos \theta$

If $(x|y)=0$, then by the above, $\Vert x+y\Vert =\Vert x\Vert +\Vert y\Vert$ (pythagorean law).

Corollary (triangular inequality)

$\left|\sum _1^n{x}_j\right|=\sum _1^n\left|x_j\right|$ if $x_j$ in an inner-product space.

Proof: Since we have:

$\Vert x+y{\Vert }^2=\Vert x{\Vert }^2+\Vert y{\Vert }^2+2\Vert x\Vert \Vert y\Vert \cos \theta \le (\Vert x\Vert +\Vert y\Vert {)}^2$,

taking the square root of both sides shows $\Vert x+y\Vert \le \Vert x\Vert +\Vert y\Vert$. From the induction the inequality follows. $◻$

For every ${ℝ}^n$, we can make it a Hilbert space by letting for $x,y\in {ℝ}^n$,

$(x|y)=\sum _1^n{x}_j{y}_j$ if $x=\sum _1^n{x}_j{e}_j$ and $y=\sum _1^n{y}_j{e}_j$.

Indeed,

$(x+z|y)=\sum _1^n(x_j+z_j)y_j=\sum _1^n{x}_j{y}_j+\sum _1^n{x}_j{z}_j=(x|y)+(x|z)$.

The rest of the properties can be shown by similar computation. Note in $ℝ$ norms are the same as absolute values. By analogy, we denote norms in ${ℝ}^n$ by $|\cdot |$ instead of $\Vert \cdot \Vert$. The difficulty that is left is to check completeness, which we do by showing the following.

Since ${ℝ}^n$ is characterized by its completeness, we have what follows.

Theorem every Hilbert space of finite dimension n is isomorphic to ${ℝ}^n$.

FIXME: Bessel's inequality follows from Parseval's identity.

3. Theorem (Gram-Schmidt process) Let G be an inner-product space of finite dimension. Then G has an orthonormal (i.e., orthogonal and normal) basis.
Proof: We may suppose $G\in ̸0$. Let some $x_1\in G$ be nonzero, and $y_1=x_1$. Also let recursively

$u_n=\Vert y_n{\Vert }^{-1}{y}_n$ and $y_n=x_n-\sum _1^{n-1}(x_n|u_k)u_k$ for $x_n\in G\mathrm{\setminus }\text{span }\{x_1,x_2,...\}$.

We use proof by induction. First, $u_1$ is defined since $y_1=x_1$ is nonzero and $\Vert u_1\Vert =1$. Now suppose $(u_j|u_k)=1$ if $j=k$ and $=0$ if $j\ne k$ for $j,k\le n-1$ and some $n$. We have $(u_n|u_n)=1$. Also, if $j\le n-1$,

$\Vert y_n{\Vert }^{-1}(u_n|u_n)$	$=(y_n|u_j)=(x_n-\sum _1^{n-1}(x_n|u_k)u_k|u_j)$
	$=(x_n|u_j)-\sum _1^{n-1}(x_n|u_k)(u_k|u_j)$
	$=(x_n|u_j)-(x_n|u_j)(u_j|u_j)$
	$=0$

Hence, $u_1,u_2,...u_n$ are orthonomal, and the induction proves the theorem. $◻$

3. Corollary (QR decomposition) Any m x n matrix can be factored into a product of QR if Q is an orthogonal matrix and R is invertible and upper triangular.