MATH411

MATH411 is a continuation of MATH410, but into higher dimensions.

• Generalizations to Rn
• Metric Spaces

Derivatives in Several Variables §

Limits §

Let $A\subseteq {\mathbb{R}}^n$. Then, $x^{\ast }$is a limit point of $A$ if there is a sequence $\exists \{x_k\}\subseteq A/\{x^{\ast }\}$ such that $\{x_k\}\to x^{\ast }$.

In other words, there is a sequence not containing $x^{\ast }$ converging to it.

If we have function $f:A\to \mathbb{R}$, and $x^{\ast }$ is a limit point of $A$, then the limit of the function is defined as

$$\underset{x\to x^{\ast }}{\lim }f(x)=l$$

if $\forall \{x_k\}\subseteq A/\{x^{\ast }\}$,

$$\underset{k\to \infty }{\lim }f(x_k)=l$$

Example: Existence of Limits (1)

$$f(x,y)=\frac{xy}{x^2+y^2}$$

$f:{\mathbb{R}}^2\to \{(0,0)\}$.

${\lim }_{(x,y)\to (0,0)}f(x,y)$ does not exist, as we can chose $\{x_k\}=(1/k,0)$, and $\{x_k\}=(1/k,1/k)$, which have different limits as $k\to \infty$.

Example: Existence of Limits (2)

$$g(x,y)=\frac{x^{2}y}{x^2+y^2}$$

In this case, ${\lim }_{(x,y)\to (0,0)}g(x,y)$ exists and equals 0.

Let $(x_k,y_k)\to (0,0)$ not containing $(0,0)$. Then,

$$|g(x_k,y_k)|\le |x_k|\frac{|x_k{y}_k|}{x_k^2+y_k^2}\le |x_k|\frac{1}{2}\to 0$$

Theorem: Compositions of Limits

Let $A\subseteq {\mathbb{R}}^n$, $x^{\ast }\in A$ be a limit point. Let $f:A\to \mathbb{R}$, $g:A\to \mathbb{R}$ be functions, $l_1,l_2\in \mathbb{R}$ such that

$$\underset{x\to x^{\ast }}{\lim }f(x)=l_1\underset{x\to x^{\ast }}{\lim }g(x)=l_2$$

Then:

1. $$\underset{x\to x^{\ast }}{\lim }f(x)+g(x)=l_1+l_2$$
2. $$\underset{x\to x^{\ast }}{\lim }f(x)g(x)=l_1{l}_2$$
3. If $g(x)\ne 0$ for all $x\in A$, and $l_2\ne 0$,
   $$\underset{x\to x^{\ast }}{\lim }\frac{f(x)}{g(x)}=\frac{l_1}{l_2}$$

The quotient rule for limits is the most interesting of the 3, and there is a broad study of limits of quotients

$$\underset{x\to x^{\ast }}{\lim }\frac{f(x)}{g(x)}$$

Where ${\lim }_{x\to x^{\ast }}f(x)={\lim }_{x\to x^{\ast }}g(x)=0$.

These limits can occur frequently, and we commonly ask if such limits exist (think of derivatives!).

Example: Limit Example

$$\underset{(x,y)\to (0,0)}{\lim }\frac{x^3}{x^2+y^2}$$

We ask if this limit exists. To determine this, we will establish a bound on the function.

$$\left|\frac{x^3}{x^2+y^2}\right|\le \left|\frac{x^3}{x^2}\right|=|x|$$

For any $(x,y)$ where $x$ and $y$ are both not equal 0, our function is bounded by $|x|$! Thus, as $(x,y)\to (0,0)$, $|x|\to 0$, so by the Comparison Lemma, $|\frac{x^3}{x^2+y^2}|\to 0$.

Thus, the limit exists and is equal to 0!

Theorem: Limit Equivalences

Let $A\subseteq {\mathbb{R}}^n$ and let $x^{\ast }$ be a limit point of $A$. For a function $f:A\to \mathbb{R}$, and $l\in \mathbb{R}$, the following assertions are equivalent:

1. $$\underset{x\to x^{\ast }}{\lim }f(x)=l$$
   In other words, for any $\{x_k\}\in A/\{x^{\ast }\}$, if ${\lim }_{k\to \infty }{x}_k=x^{\ast }$, then
   $$\underset{k\to \infty }{\lim }f(x_k)=l$$
2. $\forall ϵ>0$, there exists some $\delta >0$ such that
   $$d(x,x^{\ast })<\delta \to |f(x)-l|<ϵx\in A/\{x^{\ast }\}$$

We can also use the following property to show that such limits exist.

A function $f:{\mathbb{R}}^n/\{0\}\to \mathbb{R}$ is homogeneous of degree $k$ if

$$f(tx)=t^{k}f(x)\forall t>0,\forall x\in {\mathbb{R}}^n/\{0\}$$

Basically, we should be able to replace $x,y$ with $t$, and take out the $t$ into a $t^k$ term.

Example: Homogenoeus Functions

$$g(x,y)=\frac{x^{2}y}{x^2+y^2}$$

Is homogeneous of degree 1, because

$$g(tx,ty)=\frac{t^{3}xy}{t^2(x^2+y^2)}=tg(x,y)$$

$$f(x,y)=\frac{xy}{x^2+y^2}$$

Is homogeneous of degree 0, because

$$f(tx,ty)=t^{0}f(x,y)$$

Curiously, $g$ is homogeneous of degree 1 and has a limit to $(0,0)$, whereas $f$ is homogeneous of degree 0 and doesn’t. Does there suggest some generalization?

Proposition: Limits of Homogeneous Functions

If $f:{\mathbb{R}}^n/\{0\}\to \mathbb{R}$ is continuous, and homogeneous of degree $k>0$, then

$$\underset{x\to 0}{\lim }f(x)=0$$

Proof

We look at $f(x)$, and we try to make the case that as $x\to 0$, $f(x)\to 0$.

$$f(x)=f\left(||x||\frac{x}{||x||}\right)$$

We write this as a product of something that approaches 0 and something that is bounded.

By assumption, $f(x)$ is homogeneous of degree $k$, so letting $t=||x||$,

$$f\left(||x||\frac{x}{||x||}\right)=||x|{|}^{k}f\left(\frac{x}{||x||}\right)$$

As $x\to 0$, $||x|{|}^k\to 0$! Also, as $\frac{x}{||x||}\in S^{n-1}$ (the unit sphere of dimension $n-1$), which is a sequentially compact, then continuous functions on sequentially compact sets are bounded.

$$||x|{|}^{k}f\left(\frac{x}{||x||}\right)\to 0$$

Example: Continuity with Homogeneity

$$f(x,y)=\frac{x^{3}y}{x^2+y^2}\frac{\partial f}{\partial x}(0,0)=0$$

We show that $f$ is $C^1$ by homogeneity.

We know that $f$ is $C^{\infty }$ in ${\mathbb{R}}^n/\{(0,0)\}$ is homogeneous of degree 2. So, $\frac{\partial f}{\partial x}$ is homogeneous of degree 1.

It is a fact that if $f\in C^1({\mathbb{R}}^n/\{0\})$ and is homogeneous of degree $k$, then $\frac{\partial f}{\partial x_i}$ is homogeneous of degree $k-1$.

Let $x\ne 0$, $t>0$. Then, say $f$ is homogeneous degree $k$.

$$\begin{array}{r}f(t,x)=t^{k}f(x)\\ \frac{\partial }{\partial x_i}f(tx)=\frac{\partial }{\partial x_i}[t^{k}f(x)]=t^k\frac{\partial f}{\partial x_i}(x)\\ t\frac{\partial f}{\partial x_i}(tx)=t^k\frac{\partial }{\partial x_i}{t}^{k}f(x)⟹\frac{\partial f}{\partial x_i}(tx)=t^{k-1}f(x)\end{array}$$

By a theorem, homogeneous functions of positive degree go to 0 as $x\to 0$. So, $f\in C^1$.

This does not necessarily mean that homogeneous functions of degree $k=0$ don’t have a limit at 0! Just that some don’t.

Any constant function (ex. $f(x)=1$) have defined limits as $x\to 0$!

Partial Derivatives §

Let $f:O\to \mathbb{R}$, $x=(x_1,\dots x_n)\in O$.

For $1\le i\le n$, we define the partial derivative of $f$ with respect to $x_i$ at $x$ as

$$\frac{\partial f}{\partial x_i}(x)=\underset{t\to 0}{\lim }\frac{f(x+t{e}_i)-f(x)}{t}$$

if the latter limit exists. Note that $e_i$ is the $i^{th}$ basis vector,

Keep all variables constant except $x_i$, and take $\frac{d}{d{x}_i}$!

Example: Partial Derivatives and Continuity

Let $f:{\mathbb{R}}^2\to \mathbb{R}$,

$$f(x,y)=\{\begin{array}{ll}\frac{xy}{x^2+y^2}& (x,y)\ne (0,0)\\ 0& (x,y)=(0,0)\end{array}$$

We noticed that $f$ is not continuous at $(0,0)$, and $\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}$ exist (by the quotient rule) at all $(x,y)\ne (0,0)$.

Then, does $\frac{\partial f}{\partial x}$ exist at $(0,0)$?

$$\underset{t\to 0}{\lim }\frac{f(t,0)-f(0,0)}{t}=\underset{t\to 0}{\lim }\frac{0-0}{t}$$

Yes! It exists and is equal to 0.

Similarly, we can find $\frac{\partial f}{\partial y}=0$.

This is very interesting! Even though $f(x,y)$ is not continuous at $(0,0)$, our partial derivatives still exist! This goes against our understanding of differentiability and continuity in the single variable case.

Let $O\subseteq {\mathbb{R}}^n$ open, $f:O\mathbb{R}$. Then, we say $f$ has first-order partial derivatives if for all $1\le i\le n$, the function has a partial derivative with respect to its $i^{th}$ component, at every point in its domain.

Differentiability Need Not Imply Continuity

In the single-variable case, a function with a derivative was continuous. However, this is no longer true in multiple variables!

A function with first-order derivatives need not be continuous. Consider the following example.

Example: Differentiability Need Not Imply Continuity

Define

$$f(x,y)=\{\begin{array}{ll}\frac{xy}{x^2+y^2}& (x,y)\ne (0,0)\\ 0& (x,y)=(0,0)\end{array}$$

We show that the partial derivatives of the function exist at $(0,0)$. For all $t$,

$$f(0+t{e}_i)=f(t,0)=0$$

So,

$$\frac{\partial f}{\partial x_i}(0,0)=\underset{t\to 0}{\lim }\frac{f(0+t{e}_i)-f(0)}{t}=\underset{t\to 0}{\lim }\frac{f(t,0)-f(0,0)}{t}=0$$

However, this function is not continuous! For sequence $\{(\frac{1}{k},\frac{1}{k})\}\to 0$, $f(\frac{1}{k},\frac{1}{k})=\frac{1}{2}$ for all $k$, but $f(0)=0$!

It is only if all partials are continuous, that our theorems from the single-variable case hold.

We say that $f$ is continuously differentiable, $C^1$ if it has first-order partial derivatives such that each partial derivative $\frac{\partial f}{\partial x_i}$ is continuous for $1\le i\le n$.

---

Let’s now consider second-order partial derivatives, denoted like

$$\frac{\partial f}{\partial x_j\partial x_i}$$

Where we apply the partial derivative of $x_i$ first, then $x_j$ after.

Order matters! There are some functions where swapping the order of derivatives changes the result.

• We say $f$ has second-order partial derivatives of it has first-order partials, such that for $1\le i\le n$, each $\frac{\partial f}{\partial x_i}$ also has first-order partial derivatives (of every variable).
• We say $f$ has continuous second-order partial derivatives if it has second-order partial derivatives, and each $\frac{{\partial }^{2}f}{\partial x_i\partial x_j}$ are continuous.

Theorem: Partial Derivative Order

Let $O\subseteq {\mathbb{R}}^n$ open, and let $f:O\to \mathbb{R}$ have continuous second-order partial derivatives. Then, for any two $1\le i,j\le n$, and any $x\in O$,

$$\frac{d}{d{x}_i}\left(\frac{\partial f}{\partial x_j}\right)=\frac{d}{d{x}_j}\left(\frac{\partial f}{\partial x_i}\right)$$

Proof (TODO)

Let $x=x_i,y=x_j$. Then,

$$A=f(x_0+r,y_0+r)-f(x_0+r,y_0)-f(x_0,y_0+r)+f(x_0,y_0)$$

Define $\varphi (x)=f(x,y_0+r)-f(x,y_0)$. Then,

$$A=\varphi (x_0+r)-\varphi (x_0)$$

By the mean value theorem, this is equal to

$$\begin{array}{r}={\varphi }^{\prime }(x_0+{\theta }_1)\ast r=\frac{\partial }{\partial x}[f(x_0+{\theta }_1,y_0+r)-f(x_0+{\theta }_1,y_0)]R\end{array}$$

We create two equivalent functions that converge to the two partial derivatives, respectively, forcing equality.

Directional Derivatives and MVT §

Recall that in the single variable case, we had the Mean Value Theorem.

Theorem: Mean Value Theorem

Let $f:[a,b]\to \mathbb{R}$ be continuous, and differentiable on $(a,b)$. Then, $\exists c\in (a,b)$ such that

$$f(b)-f(a)=f^{\prime }(c)(b-a)$$

This is a really useful theorem! In this section, we generalize it to multiple variables.

This generalization requires we use the single-variable MVT!

Lemma: Mean Value Lemma

Let $O\subseteq {\mathbb{R}}^n$ open, and let $1\le i\le n$. Let $f:O\to \mathbb{R}$ have a partial derivative with respect to $x_i$ for all $x\in O$.

Let $x\in O$, and $a$ be a real number such that the segment between $x$ and $x+a{e}_i$ lies in $O$. Then, $\exists \theta ,0<\theta <1$ such that

$$f(x+a{e}_i)-f(x)=\frac{\partial f}{\partial x_i}(x+\theta a{e}_i)a$$

Intuition: If we view our function along an axis, we get a function on one-variable. On this, we can apply single-variable MVT!

Proof

Let $I$ be the open interval of real numbers containing 0 and $a$. Note that by assumption, $\forall t\in I$, $x+t{e}_i$ is in our open set $O$.

Now, define $\varphi (t)=f(x+t{e}_i)$. Then, as $f$ has a partial derivative with respect to $x_i$, we find that $\varphi (t)$ is differentiable, whose derivative is given as

$${\varphi }^{\prime }(t)=\frac{\partial f}{\partial x_i}(x+t{e}_i)$$

Thus, we can apply the single-variable MVT to find a $0<\theta <1$ such that

$$\begin{array}{r}\varphi (a)-\varphi (0)={\varphi }^{\prime }(\theta a)(a-0)\\ f(x+a{e}_i)-f(x)=\frac{\partial f}{\partial x_i}(x+\theta a{e}_i)a\end{array}$$

We use this Lemma to prove the following.

Proposition: Mean Value Proposition

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a function. Assume all partials $\frac{\partial f}{\partial x_i}$ exist $\forall x\in {\mathbb{R}}^n$, $\forall i\in \{1,\dots n\}$.

Choose an $x\in {\mathbb{R}}^n$, and an offset $h\ne 0,h\in {\mathbb{R}}^n$. Then, there exists a $z_1,\dots z_n$ in the ball around $x$ of radius $||h||$ ($B_{||h||}(x)$) such that

$$f(x+h)-f(x)=\frac{\partial f}{\partial x_1}(z_1)h_1+\cdots +\frac{\partial f}{\partial x_n}(z_n)h_n$$

Proof

We prove this for $n=2$, though the proof can very easily be extended to more dimensions.

Let $x=(x_1,x_2)$, $x+h=(x_1+h,x_2+h)$. Look at the difference. Our goal is to expand this difference into a sum of differences along one variable (only one variable changes), so that we can apply the Mean Value Lemma on each term!

$$\begin{array}{rl}f(x_1+h,x_2+h)-f(x_1,x_2)& =f(x_1+h,x_2+h)-f(x_1,x_2+h_2)\\ & +f(x_1,x_2+h_2)-f(x_1,x_2)\end{array}$$

This gives us two differences, where only one variable is changing in each. In other words, we have two differences in one-dimension!

$$f(x_1+h,x_2+h)-f(x_1,x_2+h_2)f(x_1,x_2+h_2)-f(x_1,x_2)$$

Thus, by the Mean Value Lemma,

$$\begin{array}{r}f(x_1+h,x_2+h)-f(x_1,x_2+h_2)=\frac{\partial f}{\partial x_1}(x_1+\theta h_1,x_2+h_2)h_1\\ f(x_1,x_2+h_2)-f(x_1,x_2)=\frac{\partial f}{\partial x_2}(x_1,x_2+{\theta }_2{h}_2)h_2\\ f(x_1+h,x_2+h)-f(x_1,x_2)=\frac{\partial f}{\partial x_1}(x_1+\theta h_1,x_2+h_2)h_1+\frac{\partial f}{\partial x_2}(x_1,x_2+{\theta }_2{h}_2)h_2\end{array}$$

Let $z_1=(x_1+{\theta }_1{h}_1,x_2+h_2)$, and $z_2=(x_1,x_2+{\theta }_2{h}_2)$. Note that each $z_i$ is within the ball of $B_{||h||}(x)$. We are done!

Recall that in our definitions of partial derivatives, we differentiate a function with respect to one of the axes

$$\underset{t\to 0}{\lim }\frac{f(x+t{e}_i)-f(x)}{t}$$

But what if we wanted to differentiate in a direction that isn’t aligned with the axes? This is where directional derivatives come in!

---

Let $O\subseteq {\mathbb{R}}^n$ open, and consider the function $f:O\to {\mathbb{R}}^n$. For a point $x\in O$, and direction $h$, we define the directional derivative as

$$\frac{\partial f}{\partial h}(x)=\underset{t\to 0}{\lim }\frac{f(x+tp)-f(x)}{t}$$

If the limit exists.

Now let’s define the gradient of the function, $\nabla f$, as the row vector

$$\nabla f(x)=\left(\frac{\partial f}{\partial x_1},\dots ,\frac{\partial f}{\partial x_n}\right)$$

In some cases, we can calculate the directional derivative using the gradient, which can be a lot easier than taking a limit!

Theorem: Directional Derivative Theorem

Let $O\subseteq {\mathbb{R}}^n$ open, and let $f:{\mathbb{R}}^n\to \mathbb{R}$ be $C^1$.

Then, $\forall x\in O$, and all directions $\forall h\ne 0$, the function has a directional derivative at $x$ in the direction $h$, which can be calculated as

$$\frac{\partial f}{\partial h}=\underset{t\to 0}{\lim }\frac{f(x+th)-f(x)}{t}=⟨\nabla f(x),h⟩=\sum _{i=1}^n\frac{\partial f}{\partial x_i}(x)h_i$$

In other words, the inner product of $h$ with the gradient of the function!

Proof

By the Mean Value Proposition,

$$\begin{array}{rl}\frac{f(x+th)-f(x)}{t}& =\frac{1}{t}\left(\frac{\partial f}{\partial x_1}(z_1)t{h}_1+\cdots +\frac{\partial f}{\partial x_n}(z_n)t{h}_n\right)\\ & =\frac{\partial f}{\partial x_1}(z_1)h_1+\cdots +\frac{\partial f}{\partial x_n}(z_n)h_n\end{array}$$

For $z_1,\dots z_n\in B_{||th||}(x)$. Then, as $t\to 0$, the ball of $B_{||th||}(x)$ will shrink towards $x$, forcing all $z_i$’s to converge to $x$! Thus, as $t\to 0,$ we have

$$\frac{\partial f}{\partial h}=\underset{tto0}{\lim }\frac{f(x+th)-f(x)}{t}=\frac{\partial f}{\partial x_1}(x)h_1+\cdots +\frac{\partial f}{\partial x_n}(x)h_n$$

Note that sometimes the directional derivative may be denoted as

$$\frac{\partial f}{\partial h}=\frac{d}{dt}{|}_{t=0}f(x+th)$$

Theorem: The Mean Value Theorem (Multi-Variable)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be continuously differentiable. Also let $x\in {\mathbb{R}}^n$, $h\in {\mathbb{R}}^n$ where $h\ne 0$.

Then, if the segment joining $x,x+h$ lies in $O$, then there exists $0<\theta <1$ such that

$$f(x+h)-f(x)=⟨\nabla f(x+\theta h),h⟩$$

This is the Mean Value Proposition, with the additional assertion that $z_1,\dots z_n$ are assumed to be at the same point.

Proof

Let $\varphi :\mathbb{R}\to \mathbb{R}$, $\varphi (t)=f(x+th)$. We know that for $t=1,0$, we have

$$\varphi (1)=f(x+h)\varphi (0)=f(x)$$

Then, $f(x+h)-f(x)=\varphi (1)-\varphi (0)={\varphi }^{\prime }(\theta )(1-0)$, $0<\theta <1$, by the single-variable MVT, and furthermore, as the derivative of $\varphi (t)$ is the directional derivative,

$$f(x+h)-f(x)={\varphi }^{\prime }(\theta )=⟨\nabla f(x+\theta h),h⟩$$

We can also use directional derivatives to make a few extra inferences.

Note that if $p$ is a vector of norm 1, we can interpret the directional derivative as the rate of change in a particular direction!

Theorem: Fastest Rate of Change

Let $f:{\mathbb{R}}^n\to \mathbb{R}$, $C^1$. Fix $x$, and assume $\nabla f(x)\ne 0$. Then, the maximum of the directional derivative at $x$ is given as

$$\underset{||P||=1}{\max }\frac{\partial f}{\partial P}(x)$$

Is attained for

$$P=\frac{\nabla f(x)}{||\nabla f(x)||}$$

In other words, the direction of the gradient.

Proof

If $||P||=1$, we have

$$\frac{\partial f}{\partial P}(x)=⟨\nabla f(x),P⟩$$

By Cauchy-Schwarz, this is

$$\le ||\nabla f(x)||\cdot ||P||=||\nabla f(x)||$$

We have an upper bound on our directional derivative! We can attain our upper bound if $P=\frac{\nabla f(x)}{||\nabla f(x)||}$.

$$\begin{array}{rl}⟨\nabla f(x),P⟩& =⟨\nabla f(x),\frac{\nabla f(x)}{||\nabla f(x)||}⟩\\ & =\frac{1}{||\nabla f(x)||}⟨\nabla f(x),\nabla f(x)⟩\\ & =||\nabla f(x)||\end{array}$$

We’ve found a maximizer.

Furthermore, we can use directional dervatives to prove a notion of continuity on multiple variables.

Theorem: Partial Derivatives and Continuity

Let $f:{\mathbb{R}}^n\to \mathbb{R}$, and assume $f$ is continuously differentiable. Then, $f$ is continuous.

Recall that if $f$ is $C^1$, then all partials exist and are continuous.

Proof

We look at $f(x+h)-f(x)$, and claim that as $h\to 0$, $f(x+h)\to f(x)$.

By MVT, for some $0<\theta <1$,

$$|f(x+h)-f(x)|=|⟨\nabla f(x+\theta h),h⟩|$$

By Cauchy Schwarz, we can bound this by

$$\le ||\nabla f(x+\theta h)||\cdot ||h||$$

But as $h$ is convergent, and the functions are continuous, we can find a bound for the first term!

$$||\nabla f(x+\theta h)||\le \max ||\nabla f(y)||\cdot ||x-y||\le C$$

So, this drops to 0.

By this proof, in fact, if all the partials exist $\forall x\in O$ and are bounded, then $f$ is still continuous!

---

We end with a small remark that will segway into the next section. Let $f:{\mathbb{R}}^n\to \mathbb{R},C^1$. Then,

$$\underset{h\to 0}{\lim }\frac{f(x+h)-f(x)-⟨\nabla f(x),h⟩}{||h||}=0$$

This can be proven by using Cauchy-Schwarz.

We use this to define differentiable functions! $f:{\mathbb{R}}^n\to \mathbb{R}$ is differentiable at $x$ if $\exists Q\in {\mathbb{R}}^n$ such that

$$\left\{\frac{f(x+h)-f(x)-⟨Q,h⟩}{||h||}\right\}\to 0$$

as $h\to 0$.

This is a stronger notion than partial diffentiation! So,

• $f\in C^1$ implies that $f$ is differentiable
• $f$ differentiable implies that all parties of $f$ exist

But, the converses are not true!

Local Approximation of Real-Valued Functions §

First Order Approximations §

Motivation

Say we have some function, and we want to analyze the behavior of it in an area around the point $x$. One way to do this is to choose another function $g$ that approximates $f$, yet is simpler! We can then work with $g$ to see what properties it has (and inherits from $f$).

Let $O\subseteq {\mathbb{R}}^n$, and $x\in O$. For a positive integer $k$, we say that functions $f,g:O\to \mathbb{R}$ are $k^{th}$ order approximations of one another at $x$ if

$$\underset{h\to 0}{\lim }\frac{f(x+h)-g(x+h)}{||h|{|}^k}=0$$

We ask, can we find a first-order approximation for a given function $f$?

Theorem: First Order Approximation Theorem

Let $O\subseteq {\mathbb{R}}^n$ open, $f:O\to \mathbb{R}$ be $C^1$. Then, for $x\in O$, we have first order approximation of $f$

$$\underset{h\to 0}{\lim }\frac{f(x+h)-[f(x)+⟨\nabla f(x),h⟩]}{||h||}=0$$

Proof

Recall previously that by MVT, we find $0<\theta <1$ such that

$$f(x+h)-f(x)=⟨\nabla f(x+\theta h),h⟩$$

We can subtract both sides by $⟨\nabla f(x),h⟩$ and apply Cauchy Schwarz to obtain

$$\begin{array}{rl}f(x+h)-f(x)-⟨\nabla f(x),h⟩& =⟨\nabla f(x+\theta h),h⟩-⟨\nabla f(x),h⟩\\ f(x+h)-f(x)-⟨\nabla f(x),h⟩& =⟨\nabla f(x+\theta h)-\nabla f(x),h⟩\\ |f(x+h)-f(x)-⟨\nabla f(x),h⟩|& \le ||\nabla f(x+\theta h)-\nabla f(x)||\cdot ||h||\\ \frac{|f(x+h)-f(x)-⟨\nabla f(x),h⟩|}{||h||}& \le ||\nabla f(x+\theta h)-\nabla f(x)||\end{array}$$

Because $f$ is continuously differentiable, we know that

$$\underset{h\to 0}{\lim }||\nabla f(x+\theta h)-\nabla f(x)||=0$$

So by the Comparison Lemma, we can force our original limit to be 0.

We can alternatively write this in a few ways.

• Let $E(x,h)$ denote some error depending on $x$ and $h$. Then, our approximation can be given as
  $$f(x+h)=f(x)+⟨\nabla f(x),h⟩+E(x,h)\underset{h\to 0}{\lim }\frac{E(x,h)}{||h||}=0$$
  As the error drops to 0 when dividing by $||h||$, we can also say that the error is of first order, $O(||h||)$.
• Letting $y=x+h$, $x$ fixed, we can also write our error as
  $$f(y)=f(x)+⟨\nabla f(x),(y-x)⟩+O(||x-y||)$$
  So if $x$ is fixed, and $y$ is sufficiently close to $x$, then we have a close approximation!

We can also interpret this formula geometrically. In fact, interestingly enough, our first order approximation is equivalent to a tangent plane approximation of our function!

Proof

Define $G$ to be the function $G=\{(x_1,x_2,f(x_1,x_2))\}$. This defines a surface in 3-dimensions. We will define the tangent plane at $(a,b)$.

At $(a,b)$, we find tangent directions by differentiating with respect to variables $x_1$ and $x_2$.

$$T_1=\left(1,0,\frac{\partial f}{\partial x_1}(a,b)\right)T_2=\left(0,1,\frac{\partial f}{\partial x_2}(a,b)\right)$$

We take the cross product, to find a vector orthogonal to both. This will give us a vector that is a normal to our surface.

$$N=\left(-\frac{\partial f}{\partial x_1}(a,b),\frac{\partial f}{\partial x_2}(a,b),1\right)$$

We can use this to define the tangent plane at $(a,b,f(a,b))$ as

$$\begin{array}{rl}& (x_1-a,x_2-b,f(x_1,x_2)-f(a,b))\cdot \left(-\frac{\partial f}{\partial x_1}(a,b),\frac{\partial f}{\partial x_2}(a,b),1\right)=0\\ & f(x_1,x_2)-f(a,b)-\frac{\partial f}{d{x}_1}(a,b)(x_1-a)-\frac{\partial f}{\partial x_2}(a,b)(x_2-b)=0\\ & f(x_1,x_2)-\left[f(a,b)+\frac{\partial f}{d{x}_1}(a,b)(x_1-a)+\frac{\partial f}{\partial x_2}(a,b)(x_2-b)\right]=0\end{array}$$

Thus is the same as our first-order approximation formula! Simply redefine $(x_1,x_2)$ to be offsets $(a,b)$, $(x_1,x_2)+h$.

Second Order Approximations and Second Derivatives §

Motivation

In the single-variable case, we had the second-derivative test for determining minimums and maximums. Here, we develop the corresponding test for multiple variables.

Definitions and Context §

Let $A$ be an $n\times n$ matrix. Note that for any vector $\vec{x}$, the matrix-vector product

$$Ax=y$$

Is equivalent to the values inner products of the $i^{th}$ row of $A$ and $x$! If $A_i$ denotes the $i^{th}$ row of $A$, then

$$Ax=(⟨A_1,x⟩,\dots ,⟨A_i,x⟩,\dots ,⟨A_n,x⟩)$$

This fact will be useful later!

---

Let $A$ be an $n\times n$ matrix. Then, the function $Q:{\mathbb{R}}^n\to \mathbb{R}$ given by

$$Q(h)=⟨Ah,h⟩$$

Is known as the quadratic function associated with the matrix $A$.

This function gives us a clean notation for generalizing directional derivatives into higher orders!

Let $f$ be $C^2$. We define the Hessian Matrix of $f$, denoted ${\nabla }^{2}f$, as the $n\times n$ matrix where for each pair of indices $i,j$,

$$({\nabla }^{2}f(x){)}_{ij}=\frac{\partial f}{\partial x_j\partial x_i}(x)$$

In other words,

$${\nabla }^{2}f(x)=\left[\begin{array}{ccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \dots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \dots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$

Note that if $f$ has continuous second-order partials, then the Hessian Matrix is symmetric because the $ij$ entry would equal the $ji$ entry!

We use the quadratic function notation to define higher order directional derivatives. If $f\in C^2(\mathbb{R})$, $x,h$ fixed, then

1. $$\frac{d}{dt}f(x+th)=⟨\nabla f(x+th),h⟩=\sum _{i=1}^n\frac{\partial f}{\partial x_i}(x+th)h_i$$
2. $$\frac{d^2}{d{t}^2}f(x+th)=⟨{\nabla }^{2}f(x+th,h)⟩=\sum _{i,j=1}^n\frac{{\partial }^{2}f}{\partial x_i\partial x_j}(x+th)h_i{h}_j$$

Notice the pattern!

Proof

For (1), this is a chain rule.

For (2), we have

$$\begin{array}{rl}\frac{d}{dt}\left[\frac{d}{dt}f(x+th)\right]& =\frac{d}{dt}\left[\sum _{i=1}^n\left(\frac{\partial f}{\partial x_i}\right)(x+th)h_i\right]\\ & =\sum _{i=1}^n\left[\sum _{j=1}^n\frac{\partial }{\partial x_j}\frac{\partial f}{\partial x_i}(x+th)h_j\right]h_i\end{array}$$

Remark

If $f\in C^3$, then

$$\frac{d^3}{d{t}^3}f(x+th)=\sum _{i,j,k=1}^n\frac{{\partial }^{3}f}{\partial x_i\partial x_j\partial x_k}(x+th)h_i{h}_j{h}_k$$

In the above formulas, (2) will be quite useful in establishing a second-derivative criterion for the multi-variable case. However, we will also need some way to estimate the sizes of the values that quadratic functions can take on! These tools are given as follows.

---

Let $A$ be an $n\times n$ matrix, $A=(a_{ij})$. The Hilbert-Schmidt norm of $A$ is given as

$$||A|{|}_{\text{HS}}={\left(\sum _{i,j=1}^n{a}_{ij}^2\right)}^{1/2}$$

We think of the matrix as a long vector, and take the vector norm.

With this norm for a matrix, we can generalize the Cauchy-Schwarz Inequality!

Theorem: Generalized Cauchy Schwarz Inequality

Let $A$ be $n\times n$, and $h\in {\mathbb{R}}^n$. Then,

$$||Ah||\le ||A||\cdot ||h||$$

Proof

$$\begin{array}{rl}Ah& =\left[\begin{array}{c}\text{Row 1}\\ ⋮\\ \text{Row n}\end{array}\right]h\\ & =\left[\begin{array}{c}⟨r_1,h⟩\\ ⋮\\ ⟨r_n,h⟩\end{array}\right]\\ ||Ah|{|}^2& =\left(⟨r_1,h{⟩}^2+\cdots +⟨r_n,h{⟩}^2\right)\\ & \le (||r_1|{|}^2+\cdots +||r_n|{|}^2)||h|{|}^2\\ & \le ||h|{|}^2\sum _{i,j=1}^n{a}_{ij}^2\\ & =||A|{|}^2||h|{|}^2\end{array}$$

We can also define the operator norm of $A$ as

$$||A|{|}_{\text{op}}=\underset{||h||=1}{\max }||Ah||$$

Based on this, and the Generalized Cauchy-Schwarz Inequality, we can find that for $||h||=1$,

$$||Ah||\le ||A|{|}_{HS}||A|{|}_{op}\le ||A|{|}_{HS}$$

---

Let $A$ be a $n\times n$ matrix. $A$ is positive definite if

$$⟨Au,u⟩>0u\ne 0$$

Similarly, $A$ is negative definite if

$$⟨Au,u⟩<0u\ne 0$$

Proposition: Properties of Positive Definite Matrices

Let $A$ be a positive definite matrix. Then, there exists a $c>0$ such that

$$Q(u)=⟨Au,u⟩\ge c||u|{|}^2$$

For all $u\in {\mathbb{R}}^n$.

Proof

Note that the LHS and the RHS are both homogeneous degree 2.

$$⟨A(tu),tu⟩=t^2⟨Au,u⟩||tu|{|}^2=t^2||u|{|}^2\forall t>0,\forall u\in {\mathbb{R}}^n$$

Thus, our equation is true for $u$ if and only if it is true for any other $tu$, $(t>0)$.

Thus, it suffices to show that our equation is true for unit vectors $\frac{u}{||u||}$, as by the earlier proposition, the argument applies for all $u$. Then,

$$⟨Au,u⟩\ge c\forall ||u||=1$$

This creates a continuous function along a $S^{n-1}$ hypersphere in ${\mathbb{R}}^n$, which is sequentially compact. Thus, it must have a minimum and maximum. Choose the minimum to find our $c$.

In fact, we can find $c$ by taking the minimum of the eigenvalues.

Second Order Approximation and Second Derivative Test §

Let $A\subseteq {\mathbb{R}}^n$, $f:A\to \mathbb{R}$. Also, let $x\in A$. Then, we have the following definitions:

• $x$ is a local minimizer if there exists a $\delta >0$ such that
  $$f(x)\le f(x+h)(x+h)\in A,\forall 0<||h||<\delta$$
• $x$ is a local maximizer if there exists a $\delta >0$ such that
  $$f(x)\ge f(x+h)\forall 0<||h||<\delta$$
• $x$ is a local extreme point if it is either a local minimizer or a local maximizer for $f$.

Note that $x$ is a strict minimizer / maximizer if the inequality is strictly less than or greater than.

In the single-variable case, we found that for a local extremum to occur, the derivative must be 0. We define the analogous case for multiple variables.

Theorem: Necessity for Local Extremum

Let $O\subseteq {\mathbb{R}}^n$ open, and let $f:O\to \mathbb{R}$ have first-order partial derivatives. If $x\in O$ is a local extreme point for $f$, then

$$\nabla f(x)=0$$

But unlike the single variable case, finding the $x$’s such that this holds is very difficult, as we get a system of equations! To help us with this, we need a more formal way to define the behaviors of functions! We define a test analogous to the single-variable Second-Derivative Test to help us with this.

By the Lagrange Remainder Theorem, recall that if $f:\mathbb{R}\to \mathbb{R}$, $f^{\prime \prime }(x)$ exists for every $x$, then for all $x,h\in \mathbb{R}$, there exists a $0<\theta <1$ such that

$$f(x+h)=f(x)+f^{\prime }(x)h+\frac{1}{2}{f}^{\prime \prime }(x+\theta h)h^2$$

We can generalize this to the multi-variable case!

Theorem: Multi-Variable Remainder Theorem

Let $f:{\mathbb{R}}^n\to \mathbb{R}$, $C^2$. Then, for $x,h\in {\mathbb{R}}^n$, there exists $0<\theta <1$ such that

$$f(x+h)=f(x)+⟨\nabla f(x),h⟩+\frac{1}{2}⟨{\nabla }^{2}f(x+\theta h)h,h⟩$$

Proof

Let $\varphi (t)=f(x+th)$. Then,

$$\varphi (1)=\varphi (0)+{\varphi }^{\prime }(0)+\frac{1}{2}{\varphi }^{\prime \prime }(\theta )$$

For some $0<\theta <1$.

Notice that this holds only because we assumed the second order derivatives are continuous. We find each term to be

$$\begin{array}{r}{\varphi }^{\prime }(0)={\frac{d}{dt}}_{t=0}f(x+th)=⟨\nabla f(x),h⟩\\ {\varphi }^{\prime \prime }(t)=\frac{d^2}{d{t}^2}f(x+th)=⟨{\nabla }^{2}f(x+th)h,h⟩\end{array}$$

This is in fact a second order approximation of $f$!

Theorem: Second Order Approximation Theorem

Let $f:{\mathbb{R}}^n\to \mathbb{R}$, $C^2$. Then,

$$\underset{h\to 0}{\lim }\frac{f(x+h)-[f(x)+⟨\nabla f(x),h⟩+\frac{1}{2}⟨{\nabla }^{2}f(x)h,h⟩]}{||h|{|}^2}=0$$

Proof

$$\begin{array}{rl}& \frac{f(x+h)-[f(x)+⟨\nabla f(x),h⟩+\frac{1}{2}⟨{\nabla }^{2}f(x)h,h⟩]}{||h|{|}^2}\\ & =\frac{|\frac{1}{2}⟨({\nabla }^{2}f(x+\theta h)-{\nabla }^{2}f(x))h,h⟩|}{||h|{|}^2}\\ & \le \frac{\frac{1}{2}||({\nabla }^{2}f(x+\theta h)-{\nabla }^{2}f(x))h||||h||}{||h|{|}^2}\\ & \le \frac{1}{2}||{\nabla }^{2}f(x+\theta h)-{\theta }^{2}f(x)||\to 0\end{array}$$

With the Second-Order Approximation Theorem, we can define a multi-variable analogy to the Second-Derivative test.

Theorem: Second Derivative Test

Let $f:{\mathbb{R}}^n\to \mathbb{R}$, $C^2$.

Let $x$ be a point such that $\nabla f(x)=0$.

• If the Hessian Matrix ${\nabla }^{2}f(x)$ is positive definite, then $x$ is a strict local minimizer.

• If the Hessian Matrix ${\nabla }^{2}f(x)$ negative definite, then $x$ is a strict local maximizer.

Proof (TODO)

We know

$$f(x+h)=f(x)+⟨\nabla f(x),h⟩+\frac{1}{2}⟨{\nabla }^{2}f(x)h,h⟩+R(h)$$

With ${\lim }_{h\to 0}\frac{R(h)}{||h|{|}^2}=0$.

By assumption, $\nabla f(x)=0$, and ${\nabla }^{2}f(x)$ is positive definite matrix. By definition, $\exists c$ such that $⟨{\nabla }^{2}f(x)h,h⟩\ge c||h|{|}^2$.

So,

$$f(x+h)\ge f(x)+0+\frac{c}{2}||h|{|}^2+R(h)$$

But we can’t guarantee that $R(h)$ is positive, and in fact, it can be negative! So now, we show that $R(h)$ could be negative, but its magnitude cannot be too large.

Since \lim_{h \to 0 \frac{R(h)}{||h||} = 0$,$\exists \delta > 0$ such that

$$|R(h)|<\frac{c}{2}||h|{|}^2||h||<\delta$$

Thus,

$$f(x+h)\ge f(x)+\frac{c}{2}||h|{|}^2>f(x)0<||h||<\delta$$

By definition, $x$ is a strict local minimizer.

Is the converse of this theorem also true? In other words, let $f:{\mathbb{R}}^n\to \mathbb{R},C^2$. Assume $x$ is a local minimizer. Then $\nabla f(x)=0$. But what about ${\nabla }^{2}f(x)$? Does it have to be positive definite?

No! As a counterexample, let $f(x,y)=x^4+y^4$. Then, we have a strict local minimizer at $0$, but the ${\nabla }^{2}f(x)$ is not positive definite.

But then, what’s a necessary condition for a minimizer? We discuss this below.

Let $A$ be a symmetric $n\times n$ matrix.

• $A$ is positive semi-definite if $⟨Ah,h⟩\ge 0$ for all $h\in {\mathbb{R}}^n$.
• $A$ is negative semi-definite if $⟨Ah,h⟩\ge 0$ for all $h\in {\mathbb{R}}^n$.

Note that the inner product can now be 0, where it couldn’t be before!

Theorem: Necessity Condition for Extremum

If $f:\mathbb{R}\to \mathbb{R},C^2$ has a minimizer at $x$, then $f^{\prime }(x)=0$, $f^{\prime \prime }(x)\ge 0$ (can be 0). In other words, ${\nabla }^{2}f(x)$ has to be positive semi-definite.

Proof

Look at $\varphi (t)=f(x+th)$. Then, $\varphi (t)\in C^2$, and has a minimizer at $t=0$. So, ${\varphi }^{\prime }(0)=⟨\nabla f,h⟩=0$, and ${\varphi }^{\prime \prime }(0)=⟨{\nabla }^{2}f(x)h,h⟩\ge 0$. In other words, the matrix is positive semi-definite.

In particular, if $f:{\mathbb{R}}^n\to \mathbb{R},C^2$, has a local minimum at $x$, then all

$$\frac{{\partial }^2}{\partial x_i^2}f(x)\ge 0$$

And similarly, if $f$ has a local maximum at $x$, then all

$$\frac{{\partial }^2}{\partial x_i^2}f(x)\le 0$$

Proposition (IMPORTANT FOR EXAM)

Let $U$ be open in ${\mathbb{R}}^n$, $f:U\to \mathbb{R},C^2$. Assume the Laplacian of $f$ at $x$ is positive for all $x\in U$.

$$\Delta f(x)=\frac{{\partial }^{2}f}{\partial x_1^2}(x)+\cdots +\frac{{\partial }^{2}f}{\partial x_n^2}(x)>0\forall x\in U$$

Then, $f$ has no maximizer in $U$.

Proof

Assume by contradiction that $f$ has a maximizer in $U$, given as $x$. Then, $\nabla f(x)=0$, and ${\nabla }^{2}f$ is positive semi-definite. So, $\frac{{\partial }^{2}f}{\partial x_i^2}\le 0$, meaning

$$\Delta f(x)=\frac{{\partial }^{2}f}{\partial x_1^2}(x)+\cdots +\frac{{\partial }^{2}f}{\partial x_n^2}(x)\le 0$$

Which is a contradiction.

Theorem

Let $U$ be a bounded open set, $\bar{U}=U\cup \text{Boundary of U}$ (giving us a sequentially compact set).

Let $f\in C^2(U)$, $f$ continuous on $\bar{U}$, satisfy

$$\Delta f(x)=0\forall x\in U$$

Then,

$$\underset{\bar{U}}{\max }f=\underset{\text{Boundary U}}{\max }f$$

In other words, the maximum always occurs at the boundary.

Proof

Because $U\subseteq \bar{U}$, ${\max }_{\text{Boundary U}}f\le {\max }_{\bar{U}}f$ trivially.

Look at $f_{ϵ}(x)=f(x)+ϵ||x|{|}^2$. Note that by this, $f_{ϵ}\ge f$. Then,

$$\Delta f_{ϵ}(x)=\nabla f(x)+2nϵ>0$$

Thus, $f_{ϵ}$ has no interior maximum, and so the maximum of $f_{ϵ}$ occurs at the boundary of $U$.

$$\underset{\bar{U}}{\max }f\le \underset{\bar{U}}{\max }{f}_{ϵ}\le \underset{\text{Boundary U}}{\max }f+ϵ||x|{|}^2\le \underset{\text{Boundary U}}{\max }f+ϵK$$

Where $K$ depends on $U$. Let $ϵ\to 0$ to get

$$\underset{\bar{U}}{\max }f\le \underset{\text{Boundary U}}{\max }f$$

We showed the inequalities in both directions, so we have equality. We are done.

Higher Order Approximations §

Let $x\in {\mathbb{R}}^n$, and let there be a multi-index $\alpha =({\alpha }_1,\dots ,{\alpha }_n)$ where ${\alpha }_i\in \{0,1\}$ (a vector of 1’s and 0’s).

The multi-index will be used to “select” things we want later on!

With the multi-index, we define operations

$$\begin{array}{r}|\alpha |={\alpha }_1+\cdots +{\alpha }_n\\ \alpha !={\alpha }_1!\dots {\alpha }_n!\\ x^{\alpha }=x_1^{{\alpha }_1}\dots x_n^{{\alpha }_n}\\ {\partial }^{\alpha }f=\left(\frac{{\partial }^{|\alpha |}}{\partial x_1^{{\alpha }_1}\dots \partial x_n^{{\alpha }_n}}f\right)(x)\end{array}$$

Proposition: Multinomial Formula

$$(x_1+\cdots +x_n{)}^k=\sum _{|\alpha |=k}\frac{k!}{\alpha !}{x}^{\alpha }$$

Proof

We prove this by induction. Start with $n=2$. Then, we have

$$(x_1+x_2{)}^k=\sum _{i=0}^k\frac{k!}{i!(k-i)!}{x}_1^i{x}_2^{k-i}$$

Now, suppose that our formula is true for $n-1$ ($n\ge 3$). Prove it for $n$.

$$\begin{array}{rl}((x_1+\cdots +x_{n-1})+x_n{)}^k& =\sum _{i=0}^k\frac{k!}{i!(k-i)!}(x_1+\cdots +x_{n-1}{)}^i{x}_n^{k-i}\\ & =\sum _{i=0}^k\frac{k!}{i!(k-i)!}\left[\sum _{|\beta |=i}\frac{i!}{\beta !}{\tilde{x}}^{\beta }\right]x_n^{k-i}\end{array}$$

Define $\alpha =(\beta ,k-i)$, $\beta$ having length $i$. Then, $\alpha$ has length $k$, and

$$\begin{array}{r}\frac{1}{(k-i)!(\beta )!}=\frac{1}{\alpha !}\\ {\tilde{x}}^{\beta }{x}_n^{k-i}=x_1^{{\beta }_1}\dots x_{n-1}^{{\beta }_{n-1}}{x}_n^{k-i}=x^{\alpha }\end{array}$$

So,

$$\sum _{i=0}^k\frac{k!}{i!(k-i)!}\left[\sum _{|\beta |=i}\frac{i!}{\beta !}{\tilde{x}}^{\beta }\right]x_n^{k-i}=\sum _{|\alpha |=k}\frac{k!}{\alpha !}{x}^{\alpha }$$

Let $f\in C^k({\mathbb{R}}^n)$. Look at

$$\varphi (t)=f(x+th),\varphi :\mathbb{R}\to \mathbb{R},C^1$$

We know that

$$\varphi (1)=\varphi (0)={\varphi }^{\prime }(0)+\frac{1}{2}{\varphi }^{\prime \prime }(0)+\cdots +\frac{1}{(k-1)!}{\varphi }^{k-1}(0)$$

The one dimensional Taylor expansion!

Express ${\varphi }^k(t)$ in terms of partials of $f$, $k\in \mathbb{N}$.

$$\begin{array}{rl}{\varphi }^{\prime }(t)& =⟨\nabla f(x+th),h⟩=(h\cdot \nabla )(x+th)=h_1\frac{\partial f}{\partial x_1}(x_1+th)+\cdots +h_n\frac{\partial f}{\partial x_n}(x+th)\\ & =[(h_1{\partial }_1+\cdots +h_n{\partial }_n)f](x+th)\\ {\varphi }^{\prime \prime }(t)& =⟨{\nabla }^{2}f(x+th)h,h⟩\\ & =[(h_1{\partial }_1+\cdots +h_n{\partial }_n{)}^{2}f](x+th)\\ {\varphi }^k(t)& =[(h_1{\partial }_1+\cdots +h_n{\partial }_n{)}^{k}f](x+th)=\sum _{|\alpha |=k}\frac{k!}{\alpha !}(h^{\alpha }{\partial }^{\alpha }f)(x+th)\end{array}$$

Let $t=0$. Then, we have

$$\frac{1}{i!}{\varphi }^i(0)=\sum _{|\alpha |=i}\frac{1}{\alpha !}{\partial }^{\alpha }f(x)h^{\alpha }$$

We have proven the following.

Theorem: Higher Order Approximations

Let $f:{\mathbb{R}}^n\to \mathbb{R},C^k$. Then,

$$\begin{array}{rl}f(x+th)& =\sum _{j=0}^{k-1}\sum _{|\alpha |=j}\frac{1}{\alpha !}{\partial }^{\alpha }f(x)h^{\alpha }+\sum _{|\alpha |=k}\frac{1}{\alpha !}{\partial }^{\alpha }f(x+\theta h)h^{\alpha }\\ & =\sum _{j=0}^{k-1}\sum _{|\alpha |\le k-1}\frac{1}{\alpha !}{\partial }^{\alpha }f(x)h^{\alpha }+\sum _{|\alpha |=k}\frac{1}{\alpha !}{\partial }^{\alpha }f(x+\theta h)\end{array}$$

For some $0<\theta <1$

Notice the similarity with $\varphi :\mathbb{R}\to \mathbb{R},C^k$.

$$\varphi (x+h)=\sum _{i=0}^{k-1}\frac{1}{i!}{\varphi }^i(x)h^i+\frac{1}{k!}{\varphi }^k(x+\theta h)h^k$$

For $0<\theta <1$, which is the 1-dimensional approximation formula!

Linear Map Approximations of Non-Linear Mappings §

Motivation

Before, we studied linear mappings, or in other words, functions that can be expressed as linear transformations. Now, we turn to examine mappings that may not necessarily be linear!

Linear Mappings §

We say a function $T:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is linear if for all $\alpha ,\beta \in \mathbb{R}$, $u,v\in {\mathbb{R}}^n$

$$T(\alpha u+\beta v)=\alpha T(u)+\beta T(v)$$

Theorem: Linear Mappings as Matrices

If $T:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is linear, then there exists a unique $m\times n$ matrix $A$ such that

$$T(u)=Au$$

Proof

Uniqueness §

Let $A,B$ be two non-unique matrices satisfying our property. Then,

$$Au-Bu=T(u)-T(u)=0⟹(A-B)u=0$$

Now, observe that for any row of $(A-B)$, we are taking the dot product of $u$ with that row. Let $u$ be the row, to see that the norm must be 0. This is only possible if the row is the 0 vector.

Existence §

Let $e_1,\dots e_n$ be the standard basis. Represent $u$ as

$$u=u_1{e}_1+\cdots +u_n{e}_n$$

Then,

$$\begin{array}{rl}T(u)& =T(u_1{e}_1+\cdots +u_{n}T(e_n)\\ & =u_{1}T(e_1)+\cdots +u_{n}T(e_n)\\ & =(T(e_1),\dots T(e_n))\left(\begin{array}{c}{u}_1\\ ⋮\\ v_n\end{array}\right)\end{array}$$

As given above, linear transformations can be given as their matrices, and in fact, many of their properties can be expressed in terms of matrices as well!

First, we consider compositions of transformations. Let us have linear transformations $T:{\mathbb{R}}^n\to {\mathbb{R}}^m$, $S:{\mathbb{R}}^m\to {\mathbb{R}}^k$. Let $A,B$ be matrices such that

$$T(u)=AuS(u)=Bu$$

Then, the matrix of the composition of these transformations is

$$T(S(u))=A(Bu)=(AB)uS(T(u))=B(Au)=(BA)u$$

Which is the product of the matrices!

Let’s now consider inverses of transformations.

Theorem: Invertible Transformations

$T:{\mathbb{R}}^n\to {\mathbb{R}}^n$ linear, is invertible (as a function) if and only if the corresponding matrix $A$ is invertible as a matrix if and only if $\det (A)\ne 0$.

We commonly determine that transformations are invertible by checking the matrices!

Theorem: Properties of Invertible Matrices

Let $A$ be an $n\times n$ matrix. Then, $A$ is invertible if and only if $\exists c>0$ such that

$$||Au||\ge c||u||u\in {\mathbb{R}}^n$$

By definition, $A$ is invertible if there exists a matrix $A^{-1}$ such $A{A}^{-1}=I$.

Proof

Proof ($\leftarrow$) §

If $||Au||\ge c||u||$ for all $u\in {\mathbb{R}}^n$, then the null space of $A$ is $\{0\}$, as if $Au=0$, then $||Au||\ge c||u||$ forces $u=0$.

From linear algebra, we know that if $A$ is on finite dimensions $n\times n$, and the null space is $\{0\}$, then the range of $A$ is ${\mathbb{R}}^n$ and $A$ is invertible.

Proof ($\to$) §

Conversely, if $\exists A^{-1}$ such that $A{A}^{-1}=A^{-1}A$, then we wish to find a $c>0$ such that $||Au||\ge c||u||$.

Because $A^{-1}A=I$, then $A^{-1}Au=u$.

$$u=||A^{-1}Au||\le ||A^{-1}||||Au||$$

This is the generalized Cauchy Schwarz inequality!

Thus,

$$||Au||\ge \frac{1}{||A^{-1}||}||u||$$

Let $c=\frac{1}{||A^{-1}||}$.

Recall that $||A||=(\sum a_{ij}^2{)}^{1/2}$.

Is $V$ is a vector space with bases $v_1,\dots v_n$, and also $w_1,\dots w_n$, and the bases are related by $(v_1,\dots v_n)=(w_1,\dots w_n)C$

If the matrix of $T$ with respect to $\{v_1,\dots v_n\}$ is $A$, $\{w_1,\dots w_n\}$ is $B$, then $A=CB{C}^{-1}$.

Proof

We know that $(v_1\dots v_n)=(w_1,\dots w_n)C$, then

$$\begin{array}{r}(T(v_1),\dots T(v_n))=(T(w_1),\dots T(w_n))C\\ (v_1,\dots v_n)A=(w_1,\dots w_n)CA\end{array}$$

So, $A=C^{-1}BC$.

The Derivative Matrix and Differential §

We consider the following classes of mappings. These are mappings that may be non-linear, and are approximatable by linear mappings.

Let $O\subseteq {\mathbb{R}}^n$, and consider mapping $F:O\to {\mathbb{R}}^m$ represented as component functions

$$F=(F_1,\dots ,F_m)$$

We have the following definitions for $F$

1. $F$ is said to have first-order partial derivatives at $x\in O$, provided that for all $1\le i\le m$, $F_i$ has first-order partial derivatives at $x$.
2. $F$ is said to have first-order partial derivatives, if it has first-order partial derivatives for all $x\in O$.
3. $F$ is said to be continuously differentiable provided that all of the $F_i$’s are continuously differentiable.

Theorem: Continuity on Mappings

Let $O\subseteq {\mathbb{R}}^n$, and $F:O\to {\mathbb{R}}^m$.

Let $F$ be continuously differentiable. Then, $F$ is continuous.

Now, define $F:O\to {\mathbb{R}}^m$ with first-order partials at $x\in O$. We define the derivative matrix of $F$ at $x$, denoted $DF(x)$, as the matrix whose $ij$th entry is given by

$$\begin{array}{r}DF(x{)}_{ij}=\frac{\partial F_i}{\partial x_j}(x)DF(x)=\left[\begin{array}{cccc}\frac{\partial F_1}{\partial x_1}(x)& \dots & \frac{\partial F_1}{\partial x_n}& \\ & ⋮& & ⋮\\ \frac{\partial F_m}{\partial x_1}(x)& \dots & \frac{\partial F_m}{\partial x_n}& \end{array}\right]=\left[\begin{array}{c}\nabla F_1(x)\\ ⋮\\ \nabla F_m(x)\end{array}\right]\end{array}$$

We define the gradient of a function as a row vector.

We use this derivative matrix to generalize our findings in earlier sections.

Theorem: Mean Value Theorem for Mappings

Let $F:{\mathbb{R}}^m\to {\mathbb{R}}^n,C^1$. Then, for $x,h\in {\mathbb{R}}^n$, we find $0<{\theta }_1<1,\dots ,0<{\theta }_m<1$ such that

$$F(x+h)-F(x)=\left[\begin{array}{c}\nabla F_1(x+{\theta }_{1}h)\\ ⋮\\ \nabla F_m(x+{\theta }_{m}h)\end{array}\right]h$$

This is the multi-variable MVT applied to each component!

Proof

Apply the MVT for each $F_i:{\mathbb{R}}^n\to \mathbb{R},C^1$.

Mean Value Theorem Misconception

Note that above, if we chose all ${\theta }_i$’s to be equal, then we would have

$$F(x+h)-F(x)=DF(x+\theta h)h$$

Which seems like a very clean generalization of the MVT! However, it is not guaranteed that we can find a single $\theta$ that works for each ${\theta }_i$.

Theorem: First-Order Approximation Theorem for Mappings

Let $F:{\mathbb{R}}^n\to {\mathbb{R}}^m,C^1$. Then,

$$\underset{h\to 0}{\lim }\frac{||F(x+h)-[F(x)+DF(x)h]||}{||h||}=0$$

Proof

The $i^{th}$ component of the above quantity, from the previous chapter, is

$$\frac{F_i(x+h)-F_i(x)-⟨\nabla F_i(x),h⟩}{||h||}\to 0$$

It can be shown that at $x$, $DF(x)$ is the only matrix in which this limit holds.

Theorem

Let $F:{\mathbb{R}}^n\to {\mathbb{R}}^m$. Fix $x$, and suppose there exists an $m\times n$ matrix $A$ such that

$$\underset{h\to 0}{\lim }\frac{||F(x+h)-[F(x)+Ah]||}{||h||}=0$$

Then, the mapping $F$ has first-order partial derivatives at $x$, and $A=DF(x)$.

Proof

Look at the $i^{th}$ component.

$$\begin{array}{r}\underset{h\to 0}{\lim }\frac{F_i(x+h)-F_i(x)-⟨\nabla F_i(x),h⟩}{||h||}\to 0\\ \underset{h\to 0}{\lim }\frac{F_i(x+h)-F_i(x)-⟨(a_{i1},\dots a_{in}),(h_1,\dots h_n)⟩}{||h||}\to 0\end{array}$$

In particular, for $h=t{e}_j$, $t\to 0$, we get

$$\begin{array}{r}\underset{t\to 0}{\lim }\frac{F_i(x+t{e}_j)-F_i(x)-t{a}_{ij}}{|t|}=0\\ \underset{t\to 0}{\lim }\frac{F_i(x+t{e}_j)-F_i(x)-t{a}_{ij}}{t}=0\\ \underset{t\to 0}{\lim }\frac{F_i(x+t{e}_j)-F_i(x)}{t}=a_{ij}\\ \frac{\partial F_i}{\partial x_j}(x)=a_{ij}\end{array}$$

We also say $F:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is differentiable at $x$ if there exists an $A$, $m\times n$ matrix such that

$$\underset{h\to 0}{\lim }\frac{||F_i(x+h)-[F_i(x)+Ah]||}{||h||}=0$$

So, $F\in C^1({\mathbb{R}}^n)$ implies that F is differentiable $\forall x\in {\mathbb{R}}^n$, which implies that $DF(x)$ exists $\forall x\in {\mathbb{R}}^n$.

These are strict implications! See the examples below.

Example: Counterexamples

The below function is an example of $f:{\mathbb{R}}^2\to \mathbb{R}$ for which $Df(x)$ exists $\forall x\in {\mathbb{R}}^2$, but there does not exist an $A$ such that

$$\begin{array}{r}\frac{f(x+h)-f(x)-[a_1{h}_1+a_2{h}_2]}{||h||}=0\\ f(x_1,x_2)=\{\begin{array}{ll}\frac{x_1{x}_2}{x_1^2+x_2^2}& (x_1,x_2)\ne (0,0)\\ 0& (x_1,x_2)=(0,0)\end{array}\end{array}$$

Example

Let $F:{\mathbb{R}}^n\to {\mathbb{R}}^m,C^1$. Assume $F(0)=0,DF(0)$ satisfies $||DF(0)h||\ge ||h||,\forall h\in {\mathbb{R}}^n$.

Prove that $\exists \delta >0$ such that $||F(h)||\ge \frac{1}{2}||h||,\forall ||h||<\delta$.

We know that

$$\underset{h\to 0}{\lim }\frac{F(x+h)-F(0)-DF(0)h}{||h||}=0$$

$$\begin{array}{r}||F(h)||=||F(h)-DF(0)h+DF(0)h||\ge ||DF(h)||-||F(h)-DF(0)h||\\ \frac{||F(h)||}{||h||}=\frac{||F(h)-DF(0)h+DF(0)h||}{||h||}\ge \frac{||DF(h)||}{||h||}-\frac{||F(h)-DF(0)h||}{||h||}\ge 1/2\end{array}$$

Since

$$\underset{h\to 0}{\lim }\frac{F(x+h)-F(0)-DF(0)h}{||h||}=0$$

We know that $\exists \delta >0$ such that

$$\frac{||F(h)-DF(0)h||}{||h||}\le \frac{1}{2}$$

The Chain Rule §

From the single-variable case, recall that for $g,f$, we can find the derivative of $(g\circ f{)}^{\prime }(x)$ as

$$(g\circ f{)}^{\prime }(x)=\frac{d}{dx}g(f(x))=g^{\prime }(f(x))f^{\prime }(x)$$

We can generalize this rule to higher dimensions!

Theorem: The Chain Rule

Let $O\subseteq {\mathbb{R}}^n$, and let $F:O\to {\mathbb{R}}^m$ be continuously differentiable. Also let $U\subseteq {\mathbb{R}}^m$ to define $g:U\to \mathbb{R}$ continuously differentiable.

Suppose that $F(O)\subseteq U$. Then, the composition $g\circ F$ is also continuously differentiable, and for $1\le i\le n$, we can find its partial derivative as

$$\frac{\partial }{\partial x_i}(g\circ F)(x)=\nabla (g\circ F)(x)=\nabla g(F(x))DF(x)$$

Theorem: The Chain Rule for General Mappings

Let $O\subseteq {\mathbb{R}}^n$ open. Let $F:O\to {\mathbb{R}}^m$, and let $U\subseteq {\mathbb{R}}^m$ open to define $G:U\to {\mathbb{R}}^k$. Let $F,G$ be continuously differentiable.

Suppose that $F(O)\subseteq U$. Then, their composition $G\circ F$ is also continuously differentiable, and for each $x$, we can find

$$D(G\circ F)(x)=DG(F(x))\cdot DF(x)$$

The Inverse Function Theorem §

Motivation

The Inverse Function Theorem provides a sufficiency condition for when a function is one-to-one and invertible, and when we can compute the inverse. We generalize this to higher dimensions here!

Inverse Function Theorem: 1D, 2D §

Recall the single-variable Inverse Function Theorem.

Theorem: Inverse Function Theorem (One Dimension)

Let $f:\mathbb{R}\to \mathbb{R}$ continuously differentiable, and let $x_0\in \mathbb{R}$ such that $f^{\prime }(x)\ne 0$.

Then, there is an open interval $I$ around $x_0$, and an open interval containing $f(x_0)$ such that the function

$$f:I\to J$$

Is 1-1 and onto. Furthermore, in these intervals, we can define the function inverse $f^{-1}:J\to I$ continuously differentiable as

$$f^{-1}(y{)}^{\prime }=\frac{1}{f^{\prime }(f^{-1}(y))}$$

for all $y\in V$.

This theorem is pretty important, as it tells us when we can find a function’s inverse! More importantly, it asserts that even if an entire function is not invertible, it may have smaller intervals where it is invertible.

We now generalize this theorem to higher dimensions!

We say that an open subset of ${\mathbb{R}}^n$ containing $x$ is a neighborhood of the point $x$. Using this definition, we will now generalize the Inverse Function Theorem to 2 dimensions.

Theorem: Inverse Function Theorem (Two Dimensions)

Let $F:{\mathbb{R}}^2\to {\mathbb{R}}^2$, $C^1$. Suppose at $(x_0,y_0)$, $DF(x_0,y_0)$ is invertible.

Then, there exists a neighborhood $U$ around $(x_0,y_0)$, and a neighborhood $V$ around $F(x_0,y_0)$ such that

$$F:U\to V$$

Is 1-1 and onto. Furthermore, $F^{-1}:V\to U$ is $C^1$, and for a point $F(x,y)=(u,v)$ where $(x,y)\in U,(u,v)\in V$, we can find the derivative matrix of the inverse as

$$D{F}^{-1}(u,v)=[DF(x,y){]}^{-1}$$

The inverse of the derivative matrix at $(x,y)$!

Example: Inverse Function Theorem (2D)

$$F(x,y)=(x^2-y^2,2xy)$$

We find that

$$\det DF(x,y)=\det \left[\begin{array}{cc}2x& -2y\\ 2y& 2x\end{array}\right]=4(x^2+y^2)$$

So for all $(x,y)\ne (0,0)$, our derivative matrix is non-zero! So, by the 2D Inverse Function Theorem, for any $(x_0,y_0)\ne (0,0)$, there exists a neighborhood $U$ of $(x_0,y_0)$, $V$ of $F(x_0,y_0)=(x_0^2-y_0^2,2x_0{y}_0)$ such that

$$F:U\to V$$

Is 1-1 and onto.

We ask, what happens at $(0,0)$? Does there exist a neighborhood $U$ of $(0,0)$ such that $F$ is 1-1 on $U$?

In fact, the answer is that this is impossible, because $F(x,y)=F(-x,-y)$. So, for any neighborhood around $(0,0)$, $F$ is not one-to-one.

Example: Inverse Function Theorem (2D, 2)

Let $\varphi :{\mathbb{R}}^2\to \mathbb{R},C^1$, and

$$F(x,y)=(\varphi (x,y),{\varphi }^2(x,y))$$

We find derivative matrix

$$F(x,y)=\left[\begin{array}{cc}\frac{\partial \varphi }{\partial x}& \frac{\partial \varphi }{\partial y}\\ 2\varphi \frac{\partial \varphi }{\partial x}& 2\varphi \frac{\partial \varphi }{\partial y}\end{array}\right]$$

As the determinant of this matrix is always 0, we find that $F$ is not invertible anywhere.

Example: Inverse Function Theorem (2D, 3)

Note that the Inverse Function Theorem provides a sufficiency condition, and is not a necessity.

To show this, we find a $F:{\mathbb{R}}^2\to {\mathbb{R}}^2,C^1$ such that

$$\det (DF(x_0,y_0))=0$$

Yet $F$ is 1-1 and onto. Let $F(x,y)=(x^3,y^3)$. Then, even though the determinant of the derivative matrix $0$ at $x=0$ or $y=0$, $F$ is 1-1 and onto.

However, we can prove that for all 3 conclusions of our theorem are to hold, then our our assumptions must hold. Let $F:{\mathbb{R}}^2\to {\mathbb{R}}^2,C^1$ and assume that $DF(0,0)$ is not invertible, ad $F$ is 1-1, $F$ is onto.

By way of contradiction, as

$$F\circ F^{-1}(x,y)=(x,y)$$

Then $(DF)F^{-1}(x,y)\circ (D{F}^{-1})(x,y)=I$, but $DF$ is not invertible at some point, which is not possible, as its inverse exists!

Example: Locality of Inverse Function Theorem (2D)

$$F(x,y)=(e^x\cos y,e^x\sin y)$$

We have

$$DF(x,y)=\left[\begin{array}{cc}{e}^x\cos y& -e^x\sin y\\ e^x\sin y& e^x\cos y\end{array}\right]$$

Where $\det (DF)=e^{2x}\ne 0$. So, the 2D Inverse Function Theorem holds for all $(x,y)$! In other words, for all $(x,y)$ we can find a local neighborhood such that $F$ is 1-1 and onto.

However, this note that this theorem does not hold globally, just locally.

• $F$ is not 1-1 globally as we can find $F(x,y)=F(x,y+2k\pi )$.
• $F$ is not onto globally, as there does not exist any $(x,y)$ such that $F(x,y)=(0,0)$.

Stability of Non-Linear Mappings §

We will now introduce concepts necessary to generalize the inverse function theorem.

Theorem

For an $n\times n$ matrix $A$, the following are equivalent:

• $A$ is invertible
• $\exists c>0$ such that $||Ah||\ge c||h||$, $\forall h\in {\mathbb{R}}^n$.

We say that a mapping $F:O\to {\mathbb{R}}^n$, $O$ open, is stable if $\exists c>0$ such that

$$||F(x)-F(y)||\ge c||x-y||\forall (x,y)\in O$$

Remark

$F$ stable implies that $F$ is 1-1.

As a brief proof, if $F(x)=F(y)$, then $||F(x)-F(y)||\ge ||x-y||\to x=y$.

Note that $F$ is stable if and only if $F^{-1}$ is Lipschitz (as the inequalities are flipped!)

Interestingly, we find that matrices that are sufficiently close to an invertible matrix are also invertible. In other words, matrices that are close to 1-1 matrices are also 1-1!

Lemma

Let $A$ be an $n\times n$ matrix, and assume that $\exists c>0$ such that

$$||Ah||\ge c||h||h\in {\mathbb{R}}^n$$

Now let $B$ be an $n\times n$ matrix such that $||A-B||\le \frac{c}{2}$. Then, $||Bh||\ge \frac{c}{2}||h||$.

Proof

$$||Bh||=||Ah+(B-A)h||\ge ||Ah||-||(B-A)h||\ge c||h||-\frac{c}{2}||h||\ge \frac{c}{2}||h||$$

This lets us prove the 1-1 condition on the General Inverse Function Theorem.

Theorem: Nonlinear Stability Theorem

Let $F:O\to {\mathbb{R}}^n,C^1$, $O$ open. Assume that we have a point $x^{\ast }$ such that $DF(x^{\ast })$, is invertible.

Then there exists a neighborhood of $x^{\ast }$ such that

• $F$ is stable on $V$ (implying $F$ is 1-1 on $U$).

• The derivative matrix of $F$ is invertible $\forall x\in U$.

Proof

For point 2, look at $\det DF(x^{\ast })\ne 0$. Thus, $\exists U$ neighborhood of $x^{\ast }$ such that $\det DF(x)\ne 0$ on $U$.

For point 1, look at $F:B_r(x^{\ast })\to {\mathbb{R}}^n$. If $x,y$ belong to the ball $B_r(x^{\ast })$, we have

$$F(x)-F(y)=\left[\begin{array}{c}\nabla F_1(p_1)\\ ⋮\\ \nabla F_n(p_n)\end{array}\right](x-y)$$

For some $p_1,\dots p_n$ on the line from $x$ to $y$.

As we know that $DF(x^{\ast })$ is invertible, then $\exists c>0$ such that

$$||DF(x^{\ast })h||\ge c||h||\forall h$$

If $r$ is so small that

$$||DF(x^{\ast })-\left[\begin{array}{c}\nabla F_1(p_1)\\ ⋮\\ \nabla F_n(p_n)\end{array}\right]||<\frac{c}{2}$$

For all $p_1,\dots p_n\in B_r(x^{\ast })$. Then,

$$||B(x-y)||\ge \frac{c}{2}||x-y||$$

Minimization Principle, General Inverse Function Theorem §

To prove the General Inverse Function Theorem, we will introduce an auxiliary function such that its minimizers are solutions of some given equation.

Suppose we have a $F:O\to {\mathbb{R}}^n,C^1$, $O$ open, and $x^{\ast }\in O$ where $DF(x^{\ast })$ is invertible. Then, from the previous section, we can find a neighborhood $U$ of $x^{\ast }$ such that $DF(x)$ is invertible for $x\in U$ and $\exists$c > 0$ such that

$$||F(x)-F(y)||\ge c||x||\forall x,y\in U$$

Using this, we can show the following.

Proposition: A Minimization Principle

Let $U$ open in ${\mathbb{R}}^n$, $F:U\to {\mathbb{R}}^n$, $C^1$. Assume that the derivative matrix of $F$ is invertible $\forall x\in U$.

Let $E(x)=||F(x)-y|{|}^2$, the distance between $F(x)$ and $y$. If $E$ has an (interior) minimizer at $x\in U$, then $F(x)=y$.

Proof

Suppose we have:

• A function $G$ that transforms $x$ to $F(x)-y$.
• A function $\varphi$ that takes the squared norm of its input, $\varphi (z)=⟨z,z⟩$

Then, $E(x)=(\varphi \circ G)(x)=\varphi (G(x))$. Assume $x$ is a minimizer of $E$. Then, $\nabla E(x)=0$

$$\nabla E(x)=(\nabla \varphi )(G(x))\ast DG(x)=(\nabla \varphi )(G(x))\ast DF(x)=0$$

Because $DF(x)$ is invertible, then

$$(\nabla \varphi )(G(x))=0$$

Note that because $\varphi (z)=⟨z,z⟩$, $\nabla \varphi (z)=2z$. So,

$$(\nabla \varphi )(G(x))=2G(x)=0⟹G(x)=0$$

And as $G(x)=F(x)-y=0,F(x)=y$.

Lemma: The Open-Image Lemma

Suppose we have a $F:O\to {\mathbb{R}}^n,C^1$, $O$ open, and $x^{\ast }\in O$ where $DF(x^{\ast })$ is invertible. Then, $F(O)$ is open.

Proof

Let $y_0\in F(U)$. By assumption, we know that $\exists x_0\in U$ such that $F(x_0)=y_0$.

Let $S=\{x\in U:||x-x_0||=R\}$, the sphere of radius $R$ centered around $x_0$. We know that all points along this sphere are greater than $||F(x)-F(x_0)||\ge c||x-x_0||=cR$.

We will show that if $||y-y_0||<\frac{cR}{2}$, then there exists an $x\in B_R(x_0)$ such that $F(x)=y$.

Look at ${\min }_{x\in {\bar{B}}_R(x_0)}||F(x)-y||$, where ${\bar{B}}_R(x_0)$ is the closed ball of radius $R$ around $x_0$ (includes the border). A minimizer exists, because we have a continuous function on a compact set. We furthermore rule out a boundary minimizer.

Let $x\in S$ (so, $||x-x_0||=R$). We know that $||F(x)-y_0||=||F(x)-F(x_0)||\ge cR$. So,

$$||F(x)-y||\ge ||F(x)-F(x_0)||-||F(x_0)-y||>\frac{cR}{2}$$

But $||F(x_0)-y||<\frac{cR}{2}$, $x_0\in B_R(x_0)$, so no point on $S$ can be the minimizer. So, the minimizer must be in the interior, so by the previous lemma, the minimizer must be such that $F(x)=y$.

Suppose we have a $F:O\to {\mathbb{R}}^n,C^1$, $O$ open, and $x^{\ast }\in O$ where $DF(x^{\ast })$ is invertible. Using the previous proofs, we have found that exists a neighborhood $U$ of $x^{\ast }$, a neighborhood $V$ of F(x^*)$, such that:

• $DF(x)$ is invertible for all $x\in U$
• $\exists c$ such that
  $$||F(x)-F(y)||\ge c||x-y||\forall x,y\in U$$
• $F(U)=V$.

By general proprties of functions, $F^{-1}:V\to U$ is well defined. Finally, to show that the inverse is $C^1$, we will prove that

$$(D{F}^{-1}(y))=(DF(x){)}^{-1}$$

Proof

To prove this, it suffices to show that

$$\underset{k\to 0}{\lim }\frac{||F^{-1}(y+k)-F^{-1}(y)-[DF(x){]}^{-1}(k)||}{||k||}=0$$

We use the notation $F(x)=y,F(x+h)=y+k$. On the LHS, we have

$$\begin{array}{rl}& \underset{k\to 0}{\lim }\frac{||(x+h)-x-[DF(x){]}^{-1}[F(x+h)-F(x)]||}{||k||}\\ & =\underset{k\to 0}{\lim }\frac{||[DF(x){]}^{-1}[DF(x)[h]-[F(x+h)-F(x)]]||}{||k||}\\ & \le \underset{k\to 0}{\lim }\frac{||[DF(x){]}^{-1}[F(x+h)-F(x)-DF(x)h]||}{||k||}\end{array}$$

We want to show that $||k||\ge C||h||$.

$$||F(x+h)-F(x)||\ge C||h||⟹||k||\ge C||h||$$

So, we have

$$\le \underset{k\to 0}{\lim }\frac{[DF(x){]}^{-1}}{c}\frac{||[F(x+h)-F(x)-DF(x)h]||}{||h||}\to 0$$

This is a first order approximation! So, this goes to 0 as $k\to \infty$, as then $h\to 0$.

This gives us the General Inverse Function Theorem.

Theorem: General Inverse Function Theorem

Let $O\subseteq {\mathbb{R}}^n$ open, and let $F:O\to {\mathbb{R}}^n$ be $C^1$. Now, let $DF(x^{\ast })$ be invertible for some $x^{\ast }\in O$.

Then, there is a neighborhood $U$ of $x^{\ast }$, a neighborhood $V$ of $F(x^{\ast })$, such that $F:U\to V$ is 1-1 and onto. Furthermore, $F^{-1}:V\to U$ is also $C^1$, and for $y\in V,x\in U$ such that $F(x)=y$,

$$D{F}^{-1}(y)=[DF(x){]}^{-1}$$

We also give a second proof of the inverse function theorem based on the contraction mapping principle.

Proof (Contraction Mapping Principle)

Let $F:{\mathbb{R}}^n\to {\mathbb{R}}^n,C^1$. Let $x^{\ast }\in {\mathbb{R}}^n$ where $DF(x^{\ast })$ is invertible. We will show that $\exists {\delta }_0>0$ such that if $||f(x^{\ast })-y||<\frac{{\delta }_0}{2||DF(x^{\ast }{)}^{-1}||}$, then $\exists !x\in {\bar{B}}_{{\delta }_0}(x^{\ast })$ such that $F(x)=y$.

In other words, $F$ is locally one-to-one and onto in a local neighborhood of $x^{\ast }$!

We want to solve $F(x)=y$ if and only if $x=x-(DF(x^{\ast }){)}^{-1}(F(x)-y)=T(x)$. We will use the contraction mapping principle to show that there exists a fixed point of $T(x)$.

Create a sequence

$$x_{k+1}=T(x_k)=x_k-(DF(x^{\ast }){)}^{-1}(F(x_k)-y)$$

Remark $f:\mathbb{R}\to \mathbb{R}$)

Notice the similarity to Newton’s method, which had root-finding formula (for

$$x_{n+1}=x_n-\frac{f(x_n)}{f^{\prime }(x_n)}$$

The main step is as follows: $\exists {\delta }_0>0$ such that

$$||x-z-DF(x^{\ast }{)}^{-1}(F(x)-F(z))||<\frac{1}{2}||x-z||\forall x,z\in {\bar{B}}_{{\delta }_0}(x^{\ast })$$

The left hand side equals

$$\begin{array}{rl}& ||(DF(x^{\ast }){)}^{-1}(F(x)-F(z)-DF(x^{\ast })(x-z))||\\ & \le ||DF(x^{\ast }{)}^{-1}||||\left(\left[\begin{array}{c}\nabla F_1(P_1)\\ ⋮\\ \nabla F_n(P_n)\end{array}\right]-DF(x^{\ast })\right)(x-z)||\end{array}$$

Choose ${\delta }_0>0$ such that

$$||DF(x^{\ast }{)}^{-1}||||\left(\left[\begin{array}{c}\nabla F_1(P_1)\\ ⋮\\ \nabla F_n(P_n)\end{array}\right]-DF(x^{\ast })\right)<\frac{1}{2}\forall P_1,\dots P_n\in B_{{\delta }_0}(x^{\ast })$$

So, we found a ${\delta }_0$ such tha

$$||T(x)-T(z)||\le \frac{1}{2}||x-z||$$

Next, we will show that $T$ maps its domain onto itself.

$$T:{\bar{B}}_{{\delta }_0}(x^{\ast })\to {\bar{B}}_{{\delta }_0}(x^{\ast })$$

Let $||x-x^{\ast }||\le {\delta }_0$. Look at $T(x)-x^{\ast }$. This is equal to

$$\begin{array}{rl}& ||x-x^{\ast }-(DF)(x^{\ast }{)}^{-1}[F(x)-F(x^{\ast })+F(x^{\ast })-y]||\\ & \le ||x-x^{\ast }-DF(x^{\ast }{)}^{-1}[F(x)-F(x^{\ast })]||+||DF(x^{\ast }{)}^{-1}[F(x^{\ast })-y]||\\ & \le \frac{1}{2}||x-x^{\ast }||+\frac{1}{2}||x-x^{\ast }||\le {\delta }_0\end{array}$$

So, $T$ is a contraction, and has a fixed point.

The Implicit Function Theorem §

We now discuss the Implicit Function Theorem. This lets us create local descriptions of the set of points $u$ where a function is equal to 0, $F(u)=0$, also known as a level-curve!

2D Case: Dini’s Theorem §

Let us have function $f:{\mathbb{R}}^2\to \mathbb{R},C^1$. We ask, when is the set

$$\{(x,y):f(x,y)=0\}$$

A $C^1$ curve?

Examples

$$f(x,y)=x^2+y^2+1=0$$

This will yield an empty set, so we don’t have a $C^1$ curve.

$$f(x,y)=x^2+y^2=0$$

This will yield 1 point, so we don’t have a $C^1$ curve.

What does it actually mean for a set to be a $C^1$ curve?

Intuitively, such a set is a $C^1$ curve if we can define a $C^1$ function to represent the points. More formally, we say $C\subseteq {\mathbb{R}}^2$ is a $C^1$ curve if for all points in the set $(x_0,y_0)\in C$, there is a $U$ neighborhood of $(x_0,y_0)$, and function $g:\mathbb{R}\to \mathbb{R},C^1$ such that

$$C\cap U=\{\text{Graph y = g(x) or x = g(y)}\}\cap U$$

In other words, the points in the $U$ neighborhood can be represented by some localized output of a $C^1$ function!

Theorem: Dini's Theorem

Let $O$ open in ${\mathbb{R}}^2$, $f:O\to \mathbb{R},C^1$. Let $(x_0,y_0)$ be a point in $O$, and assume $f(x_0,y_0)=0,\frac{\partial f}{\partial y}(x_0,y_0)\ne 0$.

Then, $\exists r,R>0$, and a function $g:(x_0-r,x_0+r)\to (y_0-R,y_0+R),C^1$ such that $f(x,g(x))=0,\forall |x-x_0|<r$, and if

$$(x,y)\in (x_0-r,x_0+r)\times (y_0-R,y_0+R)$$

And $f(x,y)=0$, then $y=g(x)$.

In this box, the 0-set of $f$ takes on a $C^1$ function $g(x)$.

Proof

We know

$$f(x_0,y_0)=0,\frac{\partial f}{\partial y}(x_0,y_0)\ne 0$$

Without loss of generality, suppose $\frac{\partial f}{\partial y}(x_0,y_0)>0$. Then, $\exists R>0,c>0$ such that

$$\frac{\partial f}{\partial y}\ge c>0$$

in the box $[x_0-R,x_0+R]\times [y_0-R,y_0+R]$. So, because $f(x_0,y_0)=0$, we know that along the vertical line $(x_0,y\pm k)$, $f$ is strictly increasing, so

$$f(x_0,y-R)<0f(x_0,y+R)>0$$

We can find an interval around this vertical line where it is always negative around $f(x,y-R)$ and positive around $f(x,y+R)$. In other words, $\exists r>0$ such that $f(x,y_0-R)<0$ if $|x-x_0|<r$, and $f(x,y_0+r)>0$ if $|x-x_0|<r$.

By IVT, for we can find a $y$ such that for all fixed $x$ in our interval $|x-x_0|<r$, we find a $y$ in the vertical line such that $f(x,y)=0$. Define $g(x)=y$, the unique $y$ such that $f(x,y)=0$, $y-y_0<R$.

Basically, we find a 0 by IVT for every vertical line in this interval!

We have constructed a $g:(x_0-r,x_0+r)\to (y_0-r,y_0+r)$ such that

$$f(x,g(x))=0\forall x$$

And if $f(x,y)=0$ in our box, then $y=g(x)$.

We have our function $g$. We must now show that $g$ is $C^1$, a we cannot guarantee our $y$’s on every sliver on the interval will form a continuous function or not.

To do this, we first show that $g$ is continuous. Let $x,x+h\in (x_0-r,x_0+r)$. We look at $f(x+h,g(x+h))-f(x,g(x))$ as the difference between two points $f(B)-f(A)$, and apply MVT. So, $\exists P$ on the line from $A$ to $B$ such that

$$\begin{array}{rl}0=f(x+h,g(x+h))-f(x,g(x))& =⟨\nabla f(P),(h,g(x+h)-g(x))⟩\\ & =\frac{\partial f}{\partial x}(P)\cdot h+\frac{\partial f}{\partial y}(P)(g(x+h)-g(x))\end{array}$$

So, we get

$$g(x+h)-g(x)=-\frac{\frac{\partial f}{\partial x}(P)}{\frac{\partial f}{\partial y}(P)}h$$

We can find an upper bound for the numerator as we have a continuous function on a compact set! As we assumed that the numerator is positive, we can find a bound for the fraction, so $\exists M$ such that

$$|g(x+h)-g(x)|\le \frac{|\frac{\partial f}{\partial x}(P)|}{|\frac{\partial f}{\partial y}(P)|}|h|\le M|h|$$

Thus, $g$ is continuous, as $h\to 0$, $P\to (x,g(x))$ and thus

$$\underset{h\to 0}{\lim }\frac{g(x+h)-g(x)}{h}=-\frac{\frac{\partial f}{\partial x}(x,g(x))}{\frac{\partial f}{\partial y}(x,g(x))}$$

This also gives us a formula for $g$!

If we know $g$ is $C^1$, differentiate

$$\frac{d}{dx}[f(x,g(x))]=0$$

To get

$$\frac{\partial f}{\partial x}(x,g(x))+\frac{\partial f}{\partial y}(x,g(x))\cdot g^{\prime }(x)=0$$

We can solve for $g^{\prime }(x)$ with this!

Implicit Function Theorem §

This can be generalized to higher dimensions!

Remark

We can generalize this!

Let $f:{\mathbb{R}}^{n+1}\to \mathbb{R},C^1$,

$$f(x_0,y_0)=0x_0\in {\mathbb{R}}^n,y_0\in \mathbb{R}$$

And assume the gradient at this point is not 0. Then, by the same proof, we can find $R,r>0$ such that $g:B_r(x_0)\to (y_0-R,y_0+R),C^1$ such that $f(x,g(x))=0$!

Theorem: The Implicit Function Theorem

We look at points $(x,y)\in {\mathbb{R}}^{n+k}$, where $x\in {\mathbb{R}}^n$, $y\in {\mathbb{R}}^k$. Let $O$ be open in ${\mathbb{R}}^{n+k}$, $F:O\to {\mathbb{R}}^k,C^1$.

Let $(x_0,y_0)\in O$ such that $F(x_0,y_0)=0$, $D_{y}F(x_0,y_0)$ invertible. Then, $\exists R,r>0$, and a function

$$G:B_r(x_0)\to B_R(x_0),C^1$$

Such that

$$F(x,G(x))=0\forall x\in B_r(x_0)$$

And if $x\in B_r(x_0),y\in B_R(y_0)$, and $F(x,y)=0$, then $y=G(x)$. Also, $DG(x)$ can be computed by the chain rule.

Proof

Let $H:O\to {\mathbb{R}}^{n+k}$,

$$H(x,y)=(x,F(x,y))$$

Clearly, $H(x_0,y_0)=(x_0,0)$, and

$$DH(x_0,y_0)=\left[\begin{array}{cc}{I}_n& 0\\ D_x(x_0,y_0)& D_y(x_0,y_0)\end{array}\right]$$

Because $D_{y}F(x_0,y_0)$ is invertible, this matrix has a non-zero determinant, so the inverse function theorem applies!

So, $\exists R>0$ and a neighborhood $V$ of $(x_0,0)$ such that $H:\mathbb{R}(x_0)\times B_R(y_0)\to V$ is is 1-1, onto with a $C^1$ inverse $H^{-1}=V\to {\mathbb{R}}^{n+k}$. We define this inverse as

$$H^{-1}(x,y)=(M(x,y),N(x,y))$$

We the fact that $H(M(x,y),N(x,y))=(x,y)$. By plugging $M,N$ into $H$, we get

$$(M(x,y),F(M(x,y),N(x,y)))=(x,y)$$

So, $M(x,y)=x$

$$(x,F(x,N(x,y)))=(x,y)$$

Define $G(x)=N(x,0)$. Pick $0<r<R$ such that $B_r(x_0)\times \{0,\}\subseteq V$. Because $G(x)=N(x,0)$ which is $C^1$, $G(x)$ is too $C^1$. Subbing this in, we get

$$F(x,N(x,0))=0$$

Now, if $x\in B_r(x_0),y\in B_R(y_0)$, and $F(x,y)=0$, we write that $H^{-1}(H(x,y))=0$

$$⟹(M(x,F(x,y)),N(x,F(x,y)))=(x,y)$$

If $F(x,y)=0$, then $y=N(x,0)=G(x)$.

Example: Implicit Function Theorem

Let $F:{\mathbb{R}}^3\to {\mathbb{R}}^2,C^1$. Assume $F(0,0)=(0,0)$ and

$$DF(0,0)=\left[\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\end{array}\right]$$

Which of the following is true? $\exists g,h\in C^1,g,h:(-r,r)\to \mathbb{R}$, $g(0)=h(0)=0$, such that

1. $F(x,g(x),h(x))=(0,0),\forall |x|<r$
2. $F(g(y),y,h(y))=(0,0),\forall |y|<r$
3. $F(g(z),h(z),z)=(0,0),\forall |z|<r$

The second one! In the implicit function theorem, we need a $Y$ such that $D_Y(F)$ is invertible. So, choose them to be $x,z$, with free variable $X=y$. Then, we can apply our implicit function theorem to get result (2).

We ask, is it possible for $F(x,g(x),h(x))=(0,0),\forall |x|<r$? No. If the above holds, then by the chain rule, we find

$$\begin{array}{rl}\frac{d}{dx}F(x,g(x),h(x))& =DF(x,g(x),h(x))\left[\begin{array}{c}1\\ g^{\prime }(x)\\ h^{\prime }(x)\end{array}\right]=0\end{array}$$

And at $(0,0,0)$,

$$DF(0,g(0),h(0))\left[\begin{array}{c}1\\ g^{\prime }(0)\\ h^{\prime }(0)\end{array}\right]=0$$

But this gives us $1=0$, which is impossible!

Finally, we will show a formula for $DG(x)$, $x\in B_r(x_0)$. We use the property that $F(x,G(x))=0$. We know that starting with $x$, we map

$$x\to (x,G(x))\to F(x,G(x))=0$$

So, by chain rule,

$$\begin{array}{r}{D}_{x,y}F(x,G(x))\cdot D(x,G(x))=0\\ (D_{x}F(x,G(x))D_y(x,G(x)))\left[\begin{array}{cc}I& DG(x)\end{array}\right]=0\\ (D_{x}F)(x,G(x))+D_y(x,G(x))DG(x)=0\end{array}$$

We can use this to solve for $DG(x)$!

$$DG(x)=-{\left[D_y(x,G(x))\right]}^{-1}{D}_{x}F(x,G(x))$$

Example

Describe solutions to

$$\begin{array}{r}(x^2+y^2+z^2{)}^3-x+z=0\\ \cos (x^2+y^2)+e^z-2=0\end{array}$$

On the LHS, we have $F(x,y,z),F:{\mathbb{R}}^{1+2}\to {\mathbb{R}}^2,F(0,0)=0$.

We expect this to be a curve through $(0,0,0)$. We will try to describe this curve locally. We find

$$DF(0)=\left[\begin{array}{ccc}\partial F_1/\partial x& \partial F_1/\partial y& \partial F_1/\partial z\\ \partial F_2/\partial x& \partial F_2/\partial y& \partial F_2/\partial z\end{array}\right]=\left[\begin{array}{ccc}-1& 0& 1\\ 0& 0& 1\end{array}\right]$$

Define $F(X,Y),X\in \mathbb{R},Y\in {\mathbb{R}}^2$. We need $D_{y}F(0)$ invertible, so we choose $X=(y),Y=(x,z)$ (as column $x$ and $z$ in $DF(0)$ will give us an invertible matrix).

We get $F(X,G(X))=0$, or in other words, $F(y,G(y))=0,G(y)\in {\mathbb{R}}^2$, so our solutions look like $(g_1(y),y,g_2(y))$.

Surfaces and Paths in ${\mathbb{R}}^3$ §

Let $f:{\mathbb{R}}^3\to \mathbb{R},C^1$. Look at the level set of this function, the set of points where the function is 0.

$$S=\{(x,y,z):f(x,y,z)=0\}$$

Assume $\nabla f(x,y,z)\ne 0$ for all $x,y,z\in S$. Then $S$ is a $C^1$ surface

Recall, that for $S\subseteq {\mathbb{R}}^3$ to be a $C^1$ surface, $\forall x\in S$, there exists a $W$ neighborhood of $x$ such that $S\cap W$ is a $C^1$ function.

Proof

Let $(x_0,y_0,z_0)\in S$. Without loss of generality, assume that $\frac{\partial f}{\partial z}(x_0,y_0,z_0)\ne 0$.

By the implicit function theorem, there exists a $r,R>0$ and a function $g:B_r(x_0,y_0)\to B_R(z_0)$ such that

$$f(x,y,g(x,y))=0\forall (x,y)\in B_r(x_0,y_0)$$

And furthermore, these are the only solutions to $f(x,y,z)=0$ in $B_r(x_0,y_0)\times (z_0-R,z_0+R)$.

So we have a transformation $(x,y)\to (x,y,g(x,y))$ paramterizing $S$ near $(x_0,y_0,z_0)$. To figure out the tangent vectors at $(x_0,y_0,z_0)$, look at the change in one variable along a particular direction.

1. Fixing $y$, we differentiate in the $x$ direction to get tangent
   $$T_1:(1,0,\frac{\partial g}{\partial x}(x_0,y_0))$$
2. Fixing $x$, we differentiate in the $y$ direction to get tangent
   $$T_2:(0,1,\frac{\partial g}{\partial y}(x_0,y_0))$$

We claim that $\nabla f(x_0,y_0,z_0)\perp T_1,T_2$. Look at $f(x,y,g(x,y))=0$ for all $(x,y)$. By the chain rule,

$$\begin{array}{rl}0& =\frac{\partial }{\partial x}{|}_{x=x_0}f(x,y_0,g(x,y_0))\\ & =\frac{\partial f}{\partial x}(x_0,y_0,z_0)+\frac{\partial f}{\partial z}(x_0,y_0,z_0)\cdot \frac{\partial g}{\partial x}(x_0,y_0)=0\\ & =⟨\nabla f(x_0,y_0,z_0),(1,0,\frac{\partial g}{\partial x}(x_0,y_0)⟩\end{array}$$

So, $\nabla f(x_0,y_0,z_0)\perp T_1$. By a similar argument, it is also orthogonal to $T_2$.

We can find another vector orthogonal to $T_1,T_2$ by taking the cross product! Take $T_1\times T_2\ne 0$. Then, $\exists \lambda \ne 0$ such that

$$\nabla f(x_0,y_0,z_0)=\lambda (T_1\times T_2)$$

Let $g,h:{\mathbb{R}}^3\to \mathbb{R},C^1$. Define the intersection of the two function’s level sets,

$$C=\{(x,y,z):g(x,y,z)=h(x,y,z)=0\}$$

Intuitively, we’re intersecting 2 2-dimensional surfaces. So we should expect a 1-dimensional curve!

A sufficient condition for $C$ to be a 1-dimensional curve in ${\mathbb{R}}^3$ is

$$\nabla g(x_0,y_0,z_0)\times \nabla h(x_0,y_0,z_0)\ne 0$$

Equivalently, let $G:{\mathbb{R}}^3\to {\mathbb{R}}^2,G(x,y,z)=(g(x,y,z),h(x,y,z))$. The, we require

$$DG(x_0,y_0,z_0)=\left[\begin{array}{ccc}\dots & \nabla g& \dots \\ \dots & \nabla h& \dots \end{array}\right]$$

Has rank 2 for all $(x_0,y_0,z_0)\in C$.

If so, without loss of generality, the derivative matrix with repect to $y,z$, $D_{y,z}G(x_0,y_0,z_0)$ is invertible. By the implicit function theorem, $\exists r,R$, and

$$\gamma :(x_0-r,x_0+r)\to B_R(y_0,z_0)$$

Such that $G(x,\gamma (x))=0$ for all $|x-x_0|<R$, and these are the only solutions in $B_r(x_0)\times B_R(y_0,z_0)$.

Thus, $C$ agrees with the graph $\{(x,\gamma (x)):|x-x_0|<r\}$ in $(x_0-r,x_0+r)\times B_R(x_0,y_0)$, and we can parameterize it as

$$x\to (x,\gamma (x))$$

With tangent vector at $(x_0,y_0,z_0)$ given as $T=(1,{\gamma }^{\prime }(x_0))$, and

$$\nabla g(x_0,y_0,z_0)\perp T\nabla h(x_0,y_0,z_0)\perp T$$

These are two normals to our curve!

So, $\nabla g(x_0,y_0,z_0)\times \nabla h(x_0,y_0,z_0)$ is a non-zero tangent vector to the curve, so $\exists \lambda \ne 0$ such that

$$\nabla g\times \nabla h=\lambda T$$

---

We generalize.

An $n$-dimentional manifold embedded in ${\mathbb{R}}^N$, $N=n+k$. Let $F:{\mathbb{R}}^{n+k}\to {\mathbb{R}}^k,C^1$, and assume that $k\times (n+k)$ matrix $DF(x_0)$ has maximimal rank $k$ if $F(x_0)=0$.

If so, we will represent the level set

$$M=\{X:F(X)=0\}$$

Locally, as a graph.

Let $X_0=(x_0,y_0)\in M$, $x_0\in {\mathbb{R}}^n,y_0\in {\mathbb{R}}^k$. Without loss of generality, $D_{y}F(x_0,y_0)$ (the rightmost $k\times k$ entries) is invertible. Thus, $\exists r,R>0$ and $G:B_r(x_0)\to B_R(y_0)$ such that

$$F(x,G(x))=0$$

And these are the only solutions if $x\in B_r(x_0),y\in B_R(y_0)$. Thus, $M\cap B_r(x_0)\times B_R(y_0)$ agrees with the graph $(x,G(x)):x\in B_r(x_0)$.

This is an $n$-dimentional manifold!

We need $n$ linearly independent tangent vectors at $(x_0,y_0)$. The process of doing this is the same— fix $n-1$ variable, and differentiate with respect to our last variable. These are our tangent vectors!

$$\begin{array}{rl}& (1,0,\dots ,\frac{\partial G}{\partial x_1}(x_0)\\ & (0,1,\dots ,\frac{\partial G}{\partial x_2}(x_0)\\ & ⋮\\ & (0,\dots ,1,\frac{\partial G}{\partial x_n}(x_0)\end{array}$$

The range of $DF(x,G(x))$ at $x_0$ is the tangent space above, as

$$\begin{array}{r}{D}_x(F(x,G(x))=0\\ (DF)(x_0,y_0)\cdot \left[\begin{array}{c}I\\ DG(x_0)\end{array}\right]=0\end{array}$$

So, the tangent space to $M$ at $(x_0,y_0)$ is the null space of $DF(x_0)$.

---

We have $f:{\mathbb{R}}^n\to {\mathbb{R}}^3,C^1$, and an open set $O\subseteq {\mathbb{R}}^2$. We want to know if $F(O)$ looks like a smooth surface.

Recall that we say that if

$$\frac{\partial F}{\partial x}(x,y)\times \frac{\partial F}{\partial y}(x,y)\ne 0$$

Then $F(O)$ is a smooth surface at $F(x,y)$.

This is equivalent to saying the derivative matrix of $F$ has rank 2, as the first and second row are linearly independent!

Theorem

In the general case, let $O\subseteq {\mathbb{R}}^k$,

$$F:O\to {\mathbb{R}}^N,C^1$$

Assume that $DF(x_0)$ has rank $k$ $(N\ge k)$. Denote $F=(F_1,F_2),F_1\in {\mathbb{R}}^k,F_2\in {\mathbb{R}}^N$. Without loss of generality, assume $D{F}_1(x_0)$ is invertible.

Then, $\exists U$ neighborhood of $x_0$ and $\exists V$ neighborhood of $F_1(x_0)$ such that

$$F(U)=\{(y,G(y)):y\in V\}$$

For some $G:V\to {\mathbb{R}}^{N-k},C^1$.

Proof

We know that $D{F}_1(x_0)$ is invertible. By the inverse function theorem, $\exists U$ neighborhood of $x_0$ and $V$ neighborhood of $F_1(x_0)$ such that $F_1:U\to V$ is one-to-one, onto, and has a $C^1$ inverse $F^{-1}:V\to U$.

We compose

$$F(x)=(F_1(x),F_2(x))=(F_1(F_1^{-1}(y)),F_2(F_1^{-1}(y)))=(y,G(y)),y\in V$$

Lagrange Multipliers §

Case 1: Surfaces in ${\mathbb{R}}^3$ §

Let $X=(x,y,z)\in {\mathbb{R}}^3$, and let $g:{\mathbb{R}}^3\to \mathbb{R},C^1$. Furthermore, define surface

$$S=\{x\in {\mathbb{R}}^3:g(X)=0\}$$

Where $\nabla g(X)\ne 0$ if $g(X)=0$ ($X\in S$).

Let $f:{\mathbb{R}}^3\to \mathbb{R},C^1$, and let $X_0$ be such that $f(X_0)\le f(X)$ (or $f(X_0)\ge f(X)$), $\forall X\in S$. Then, $\exists \lambda \in \mathbb{R}$ such that

$$\nabla f(x_0)=\lambda \nabla g(x_0)$$

Proof

Without loss of generality, we have that

$$\frac{\partial g}{\partial z}(x_0,y_0,z_0)\ne 0$$

By the implicit function theorem, there exists a function $h:B_r(x_0,y_0)\to \mathbb{R}$ such that

$$\{x,y,h(x,y)\}$$

Is equal to $S$ in the neighborhood of $X_0$. Then, look at the composition $\varphi :B_r(x_0,y_0)\to \mathbb{R}$, $\varphi (x,y)=f(x,y,h(x,y))$, which has an unconstrained (interior) minimum (or maximum) at $(x_0,y_0)$. Thus, $\nabla \varphi (x_0,y_0)=0$.

Let $H(x,y)=(x,y,H(x,y))$. By the Chain Rule,

$$\begin{array}{rl}\nabla \varphi (x_0,y_0)& =\nabla f(x_0,y_0,z_0)\cdot DH(x_0,y_0)\\ & =\nabla f(x_0,y_0,z_0)\left[\begin{array}{cc}1& 0\\ 0& 1\\ \frac{\partial h}{\partial x}(x_0,y_0)& \frac{\partial h}{\partial y}(x_0,y_0)\end{array}\right]\end{array}$$

Each column of the derivative matrix is a tangent vector! So, $\nabla f(x_0)$ is orthogonal to both columns.

We know that $\nabla g(x_0)$ is also normal to our surface at $X_0$. So,

$$\nabla g(X_0)\in (\text{span}\{T_1,T_2\}{)}^{\perp }$$

But the orthogonal set to the span of the tangent vectors is 1-dimensional! So, $\nabla g(X_0)$ is a basis for $(\text{span}\{T_1,T_2\}{)}^{\perp }$. As a basis, if $\nabla f(X_0)$ is in this space, then we can form $\nabla f(X_0)$ as a linear combination of $\nabla g(X_0)$.

$$\nabla f(X_0)=\lambda \nabla g(X_0)$$

Note that this same argument works for the case of $g:{\mathbb{R}}^n\to \mathbb{R},C^1$,

$$M=\{x:g(x)=0\}$$

Assuming $\nabla g(x)\ne 0$ if $g(x)=0$ (then $M$ is an $n-1$ dimensional manifold in ${\mathbb{R}}^n$).

If $f:{\mathbb{R}}^n\to \mathbb{R},C^1$ and $X_0$ is such that $f(x_0)\ge f(x)$ (or $f(x_0)\le f(x)$), then $\exists \lambda \in \mathbb{R}$ such that

$$\nabla f(x_0)=\lambda \nabla g(x_0)$$

Case 2: Curves in ${\mathbb{R}}^3$ §

Let $g,h:{\mathbb{R}}^3\to \mathbb{R},C^1$. Define curve

$$C=\{X\in {\mathbb{R}}^3:g(X)=h(X)=0\}$$

And assume

$$\text{Rank}\left[\begin{array}{c}\nabla g(X)\\ \nabla h(X)\end{array}\right]=2$$

If $X\in C$ (then $C$ is a $C^1$ curve in ${\mathbb{R}}^3$).

Let $f:{\mathbb{R}}^3\to \mathbb{R},C^1$. Let $X_0\in C$ such that $f(X_0)\le f(X)$ (or $f(X_0)\ge f(X)$) for all $x\in C$. Then there exists ${\lambda }_1,{\lambda }_2\in \mathbb{R}$ such that

$$\nabla f(X_0)={\lambda }_1\nabla g(X_0)+{\lambda }_2\nabla h(X_0)$$

Proof

Without loss of generality, say $D_{y,z}(g,h)(x_0,y_0,z_0)$ is invertible. By the implicit function theorem, $\exists \gamma :(x_0-r,x_0+r)\to {\mathbb{R}}^2,C^1$ such that $(x,\gamma (x))$ is equal to $C$ in a neighborhood $X_0$.

Let $\varphi (x)=f(x,\gamma (x))$, $\gamma :(x_0-r,x_0+r)\to \mathbb{R}$, $\gamma$ has an unconstrainer min (or max) at $x_0$, ${\gamma }^{\prime }(x_0)=0$.

By the Chain Rule,

$$\nabla f(x_0,\gamma (x_0))\left[\begin{array}{c}1\\ {\gamma }^{\prime }(x_0)\end{array}\right]=\nabla f(x_0,\gamma (x_0))\cdot T=0$$

Is a basis for the tangent space to $C$ at $x_0$.

Recall $\nabla g(x_0),\nabla h(x_0)$ are linearly independent vectors orthogonal to $T$. So,

$$\text{Span}(\nabla g(x_0),\nabla h(x_0))=(\text{Span}T{)}^{\perp }$$

We also know $\nabla f(x_0)$ is in this space. Thus, there exists a linear combination of $\nabla g(x_0),\nabla h(x_0)$ that form $\nabla f(x_0)$.

---

Let $A$ be an $n\times n$ symmetric real matrix. Let

$$\lambda =\underset{||x||=1}{\min }⟨Ax,x⟩$$

We look at the minimum of the quadratic function in the compact set given by the unit sphere.

Let $x_0$ be a minimizer ($||x_0||=1$). Then, $A{x}_0=\lambda x_0$

Proof

$g(x)=||x|{|}^2$. We try to minimize the function $f(x)=⟨Ax,x⟩$. By the above theorem, at a minimizer, $\exists \lambda$ such that $\nabla f(x_0)=\lambda \nabla g(x_0)=\lambda 2x$.

We show that $\nabla f(x_0)=2A{x}_0$. This completes our proof.

Lemma

If $f(x)=⟨Ax,x⟩$, then $\nabla f(x)=2Ax$.

We find

$$\begin{array}{rl}\underset{t\to 0}{\lim }\frac{f(x+t{e}_i)-f(x)}{t}& =\underset{t\to 0}{\lim }\frac{⟨A(x+t{e}_i),x+t{e}_i⟩-⟨Ax,x⟩}{t}\\ & =\underset{t\to 0}{\lim }\frac{⟨Ax,x⟩+t⟨Ax,e_i⟩+t⟨A{e}_i,x⟩+t^2⟨A{e}_i,e_i⟩-⟨Ax,x⟩}{t}\\ & =⟨Ax,e_i⟩+⟨A{e}_i,x⟩\\ & =2⟨Ax,e_i⟩\end{array}$$

This is the ith component of $2Ax$!

The last equality is because $⟨A{e}_i,x⟩=⟨e_i,Ax⟩$!