Linear algebra · information loss

When matrices cannot be inverted

An inverse must recover one unique input from every output. A singular matrix prevents that recovery because it collapses at least one direction.

Four signs of the same problem

For a square n × n matrix, these statements are equivalent. Each describes the same loss of a dimension from a different mathematical viewpoint.

  1. 1

    Determinant is zero

    The transformed unit square or cube has zero area or volume.

  2. 2

    Columns are dependent

    At least one column can be constructed from the others.

  3. 3

    Rank is below n

    The outputs span fewer dimensions than the inputs.

  4. 4

    Zero is an eigenvalue

    Some non-zero direction is mapped to the zero vector.

See a dimension disappear

Compare an invertible transformation with matrices that flatten every point onto a line or a single point.

Transformation matrix A

Edit any entry or choose a preset. Values update immediately.

Before: a 2D square
Original unit square A square with positive area and two independent basis directions.
After: A transforms the square
Transformed square The selected matrix collapses the square onto a line, so its area becomes zero.
det(A)0
rank(A)1 of 2
InverseDoes not exist

The square collapses to a line. Distinct inputs along the lost direction share an output, so the transformation cannot be reversed uniquely.

Recognize common singular matrices

In each example, one direction contributes no new information.

Zero row

1  2
0  0

The second output coordinate is always zero.

Proportional rows

2  4
1  2

Row one is twice row two, so it adds no independent constraint.

Zero matrix

0  0
0  0

Every possible input maps to the same output: zero.

What singularity means in linear regression

The familiar normal-equation expression uses an inverse, but singularity is a design-matrix diagnosis—not a dividing line between “statistics” and “machine learning.”

β̂ = (XTX)−1XTy

If columns of X are exactly redundant—for example, a duplicated predictor, all category indicators plus an intercept, or more columns than independent observations—then XTX is singular and that inverse form is unavailable.

Repair the design

Remove or recode exact redundancy when it comes from duplicated variables, constants, or dummy-variable coding.

Use stable computation

QR, SVD, or a pseudoinverse can identify an estimable solution without explicitly forming (XTX)−1.

Regularize deliberately

Ridge adds a positive diagonal penalty so the penalized system is typically invertible; lasso or elastic net may be chosen for different modeling goals.