Linear Algebra for Graphics Geeks (SVD-VIII)

The SVD and the Moore-Penrose Pseudoinverse

Let's look at the SVD and its inverse.

A = USV^T

A^-1 = (USV^T)^{- 1} =
(V^T)^-1S^-1U^-1 =
VS^-1U^T

Here's the question for this post:

If we substitute the pseudoinverse of S for S^-1 in the product above, will we wind up with the pseudoinverse of A?

In other words, is A⁺ = VS⁺U^T?

The quick answer is yes.

I've read quite a bit on the subject in the past few months, but I don't recall seeing this worked out, so I'm going to do it just for the sake of sanity and entertainment. :)

A few reminders for the sake of clarity:

First, because U and V are orthogonal matrices their inverses and tranposes are the same, which is why either matrix cancels out its transpose (UU^T=U^TU=VV^T=V^TV=I). Second, there are several simplifications involving S and S⁺ that exploit the identities of pseudoinverses. I'll note these when they are applied. Also, the inverse of a matrix product is the reverse product of the individual inverses (likewise with transposes). cf. The Matrix Cookbook.

Matrix A needs to meet four criteria to be a pseudoinverse. Let's substitute USV^T for A and VS⁺U^T for A+ and see if everything works out.

Rule #1:

AA⁺A = A

(USV^T)(VS⁺U^T) (USV^T) =
US(V^TV)S⁺(U^TU) SV^T =
U(SS⁺S)V^T =
USV^T

note: The last step applies this rule (#1) to (SS⁺S), which we can do because S+ is a pseudoinverse.


Rule #2:

A⁺AA⁺ = A⁺
(VS⁺U^T)(USV^T)(VS⁺U^T) =
VS⁺(U^TU)S(V^TV) S⁺U^T =
V(S⁺SS⁺)U^T =
VS⁺U^T

note: The last step applies this rule (#2) to (S⁺SS⁺), which we can do because S+ is a pseudoinverse.


Rule #3:

(AA⁺)^T = AA⁺

AA⁺ = (USV^T)(VS⁺U^T)
AA+ = USV^TVS⁺U^T
AA+ = US(V^TV)S⁺U^T
AA+ = USS⁺U^T
AA+ = U(SS⁺)U^T

(AA+)^T = (U(SS⁺)U^T)^T
(AA+)^T = U^TT(SS⁺)^TU^T
(AA+)^T = U(SS⁺)^TU^T
(AA+)^T = U(SS⁺)U^T

note: The last step applies this rule (#3) to (SS⁺)^T, which we can do because S⁺ is a pseudoinverse.


Rule #4:

(A⁺A)^T = A⁺A

A⁺A = (VS⁺U^T)(USV^T)
A⁺A = VS⁺(U^TU)SV^T
A⁺A = V(S⁺S)V^T

(A⁺A)^T = (V(S⁺S)V^T)^T
(A⁺A)^T = V^TT(S⁺S)^TV^T
(A⁺A)^T = V(S⁺S)^TV^T
(A⁺A)^T = V(S⁺S)V^T

note: The last step applies this rule (#4) to (S⁺S)^T, which we can do because S⁺ is a pseudoinverse.

Finally, let's look at an identity mentioned in a my last post on the subject. This is an important one relating to least squares.

A⁺ = (A^TA)^-1A^T
A⁺ = ((USV^T)^T(USV^T))^-1(USV^T)^T
A⁺ = ((VS^TU^T)(USV^T))^-1(VS^TU^T)
A⁺ = (USV^T)^-1(VSU^T)^-1(VSU^T)
A⁺ = (USV^T)^-1
A⁺ = (V^T)^-1S⁺U^-1
A⁺ = VS⁺U<sup>T

fin</sup>

This is post #10 of Linear Algebra for Graphics Geeks. Here are links to posts #1, #2, #3, #4, #5, #6, #7, #8, #9.

Next: Some insights relating to rotation matrices.