Homework 5: Cauchy-Schwarz and Independence vs Uncorrelation

A Quick Proof of the Cauchy-Schwarz Inequality

Let u and v be two vectors in Rⁿ. The Cauchy-Schwarz inequality states that:

|u · v| ≤ |u||v|.

Written out in coordinates, this says:

|u₁v₁ + u₂v₂ + ··· + uₙvₙ| ≤ √(u₁² + u₂² + ··· + uₙ²) √(v₁² + v₂² + ··· + vₙ²).

This equation ensures that vectors behave as we geometrically expect. For example, we know that:

|u + v|² = (u + v) · (u + v) = u · u + v · v + 2u · v = |u|² + |v|² + 2u · v.

Using the Cauchy-Schwarz inequality, we have:

|u|² + |v|² + 2u · v ≤ |u|² + |v|² + 2|u||v| = (|u| + |v|)².

So the Cauchy-Schwarz inequality tells us that:

|u + v|² ≤ (|u| + |v|)² or |u + v| ≤ |u| + |v|.

In other words, the length of the sum of two vectors is no more than the sum of their lengths. This result aligns with our geometric intuition, as u · v = |u||v| cos θ, and cos θ ≤ 1.

For a direct proof (avoiding geometric intuition), consider any vector w. We know:

w · w = w₁² + w₂² + ··· + wₙ² ≥ 0.

Rescale u and v to have the same length, and consider the vector |u|v - |v|u:

Compute its dot product with itself:

0 ≤ (|u|v - |v|u) · (|u|v - |v|u) = |u|²(v · v) - 2|u||v|(u · v) + |v|²(u · u).

Rearranging:

2|u||v|(u · v) ≤ 2|u|²|v|², or u · v ≤ |u||v|.

A similar argument using (|u|v + |v|u) shows that -u · v ≤ |u||v|, so:

|u · v| ≤ |u||v|, as promised.

Independence vs Uncorrelation

Conceptual Difference: Independence implies uncorrelation, but uncorrelation does not imply independence. If two random variables X and Y are independent, their joint probability distribution factorizes as:

P(X, Y) = P(X) * P(Y)

This independence guarantees that the covariance and correlation coefficient will both be zero. On the other hand, uncorrelation is defined purely in terms of the covariance:

Cov(X, Y) = E[XY] - E[X]E[Y] = 0

While uncorrelation indicates a lack of linear relationship, it does not rule out higher-order dependencies or nonlinear relationships.

Possible Measures

• For independence: Use metrics such as mutual information or tests for independence like the Chi-squared test or the Kolmogorov-Smirnov test.
• For uncorrelation: Check the correlation coefficient, or directly compute the covariance.

Independence is a stronger concept and often more difficult to verify, while uncorrelation provides a simpler, yet less comprehensive measure of relationship.

Exercise