§ Minimising L2 norm with total constraint
- Suppose we are trying to minimize x2+y2 subject to x+y=10.
- We can think of (x,y) as two points located symmetrically about 5, suppose it is x=(5+ϵ) and y=(5−ϵ).
- See that the function f(k)=k2 is such that the output becomes larger as we go to the right / increase the argument than the rate at which the output becomes smaller as we go to the left / decrease the argument.
- This is clear by computing ∂kf=2k, which means that if kr>kl (right/left), then ∂krf=2kr, while ∂klf=2kl, so if we step to the left and the right by ϵ, keeping the total the same, the sum will change by (2kr−2kl)ϵ>0.
- Said differently, because the function is convex / f′′(x)>0, this means that ∂k∣rf>∂k∣lf, and thus we can trade the loss of the total from moving to the left (a −∂k∣lϵ for the gain of the total from moving to the right (a +∂k∣rϵ).
* dx=1.2
/|---->
- |
/ |
-- |
*---/ |
-dx=0.8 |
<-| |
| x=0.6
x=0.4
- We gain more by moving rightwards (in terms of f(r+dx)≃f(r)+f′(r)dx=f(r)+2f(r)dx than we lose by moving leftward (in terms of f(l−dx)≃f(l)−f′(l)dx=f(l)−2f(l)dx. Since f(r)>f(l), the total we gain is still net positive.
- Said differently again, we gain faster by moving from a point that is rightwards, than the rate at which we lose from a point that is leftwards.
- Said differently again, the elevation gain is larger towards the right, so a small motion rightwards gains us more elevation than a small motion leftwards loses elevation.
§ How does this relate to convexity?
- What is the geometric intution for this being related to "below a line"?