Inner Products

Subtopic: Vector Spaces

An inner product generalizes the dot product to abstract vector spaces, providing notions of length, angle, and orthogonality. Any inner product induces a norm and allows geometric reasoning—even in spaces of functions or matrices. Inner product spaces are the setting for Fourier analysis, quantum mechanics, and least squares.

Introduction

The dot product on (R^n) gives us length, angle, and perpendicularity. But what about spaces of polynomials or functions? We need a generalization that works in abstract vector spaces while preserving these geometric concepts.

An inner product is that generalization. Once you define an inner product, you automatically get a notion of length (the induced norm) and angle (via the generalized cosine formula).

Definition

An inner product on a vector space V over R is a function <⋅,⋅>:V×V→R satisfying:

• Symmetry: <u,v>=<v,u>

• Linearity in the first argument: <a*u+b*w,v>=a<u,v>+b<w,v>

• Positive definiteness: <v,v>≥0, with equality if and only if v=0

For complex vector spaces, symmetry becomes conjugate symmetry: <u,v>=<v,u>∗ (complex conjugate).

A vector space with an inner product is called an inner product space.

Examples

Standard Inner Product on ℝⁿ

<x,y>=(x_1)*(y_1)+(x_2)*(y_2)+⋯+(x_n)*(y_n)=xT*y

This is just the dot product.

Weighted Inner Product

<x,y>W=xT*W*y where W is a positive definite matrix. Different components get different weights.

Inner Product on Functions

For continuous functions on [a,b]: <f,g>=(∫_a^b)(f(x)*g(x)*d(x))

This inner product underlies Fourier series—sine and cosine functions are orthogonal under this product.

Frobenius Inner Product on Matrices

<A,B>=tr(AT*B)=(∑_^)((A_i*j))*(B_i*j)

The sum of element-wise products, treating the matrix as a long vector.

Induced Norm

Every inner product induces a norm: ‖v‖=√(,<v,v>)

This generalizes length. For the standard inner product on (R^n), this gives the Euclidean norm.

Angle and Orthogonality

The angle θ between vectors u and v is defined by:

cos(θ)=<u,v>/(‖u‖*‖v‖)

Vectors are orthogonal (perpendicular) if <u,v>=0.

A set of vectors is orthogonal if every pair is orthogonal. It's orthonormal if also each vector has norm 1.

Cauchy-Schwarz Inequality

For any inner product: |<u,v>|≤‖u‖*‖v‖

Equality holds iff u and v are linearly dependent. This is perhaps the most important inequality in analysis—it guarantees the angle formula makes sense (the ratio is between -1 and 1).

Worked Example

In the function space C[0, 1] with inner product ⟨f, g⟩ = ∫₀¹ f(x)g(x) dx, are f(x) = 1 and g(x) = x − 1/2 orthogonal?

In the function space C[0,1] with inner product <ƒ,g>=(∫_0^1)(ƒ(x)*g(x)*d(x)),
are ƒ(x)=1 and g(x)=x−1/2 orthogonal?

Compute:

<ƒ,g>=(∫_0^1)(1⋅(x−1/2)*d(x))=(∫_0^1)((x−1/2)*d(x))=[(x2)/2−x/2]10=1/2−1/2=0

Yes, ƒ and g are orthogonal. The constant function 1 is perpendicular to the centered linear function x-1/2.

Orthogonal Projection

The projection of v onto u is:

(proj_u)(v)=<v,u>/<u,u>*u

This is the component of v in the direction of u. The residual v−(proj_u)(v) is orthogonal to u.

Applications

In Fourier analysis, functions are decomposed into orthogonal sine/cosine components using the function inner product.

In quantum mechanics, states are vectors in a Hilbert space (complete inner product space). The inner product gives probability amplitudes.

In least squares, the best approximation minimizes the norm of the residual—finding the point in a subspace closest to the target.

In machine learning, kernel methods implicitly work in high-dimensional inner product spaces.

Summary

An inner product generalizes the dot product to abstract spaces, satisfying symmetry, linearity, and positive definiteness. It induces a norm and defines orthogonality. The Cauchy-Schwarz inequality ensures angles are well-defined. Common inner products include the standard dot product, weighted products, function integrals, and matrix trace products. Inner product spaces enable geometric reasoning in settings far beyond (R^n).