Skip to main content

Singular Value Decomposition (SVD)


Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) is a powerful matrix factorization that generalizes the concept of eigendecomposition to any \( m \times n \) matrix. Geometrically, SVD decomposes a linear transformation into three distinct steps: a rotation in the input space, a rescaling along the principal axes, and a second rotation in the output space.

Definition

For any matrix \( A \in \mathbb{R}^{m \times n} \), the SVD is defined as:

\[ A = U \Sigma V^T \]

Where:

  • \( U \): An \( m \times m \) orthogonal matrix. Its columns form an orthonormal basis of the output space.
  • \( \Sigma \): An \( m \times n \) rectangular diagonal matrix. The diagonal entries \( \sigma_1 \geq \sigma_2 \geq \dots \geq 0 \) are the singular values, representing the magnitude of scaling along each axis.
  • \( V \): An \( n \times n \) orthogonal matrix. Its columns form an orthonormal basis of the input space.

The Role of \( A^T A \) and \( A A^T \)

To compute the SVD, we analyze the symmetric matrices \( A^T A \) and \( A A^T \). These matrices are positive semi-definite and share the same non-zero eigenvalues, which are the squares of the singular values (\( \sigma_i^2 \)):

  • \( A^T A \) (size \( n \times n \)) has eigenvectors that form the columns of \( V \).
  • \( A A^T \) (size \( m \times m \)) has eigenvectors that form the columns of \( U \).

Example

Consider the matrix:

\[ A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \]

Step 1: Compute \( A^T A \) and \( A A^T \)

First, compute the transpose of \( A \):

\[ A^T = \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} \]

Now compute:

\[ A^T A = \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 10 & 14 \\ 14 & 20 \end{bmatrix}, \quad A A^T = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} = \begin{bmatrix} 5 & 11 \\ 11 & 25 \end{bmatrix} \]

These symmetric matrices are used to compute singular values and the columns of \( U \) and \( V \).

Step 2: Compute Singular Values

Singular values are the positive square roots of the eigenvalues of \( A^T A \). Solve:

\[ \det(A^T A - \lambda I) = 0 \] \[ \begin{vmatrix} 10 - \lambda & 14 \\ 14 & 20 - \lambda \end{vmatrix} = (10-\lambda)(20-\lambda)-196 = \lambda^2 - 30\lambda + 4 \]

Solving this quadratic equation gives:

\[ \lambda_1 \approx 29.866, \quad \lambda_2 \approx 0.134 \]

Hence, the singular values are:

\[ \sigma_1 = \sqrt{29.866} \approx 5.466, \quad \sigma_2 = \sqrt{0.134} \approx 0.366 \]

Step 3: Compute Matrix \( V \)

The columns of \( V \) are the normalized eigenvectors of \( A^T A \).

\[ (A^T A - \lambda_1 I)\vec{v}_1 = 0 \] \[ \begin{bmatrix} -19.866 & 14 \\ 14 & -9.866 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = 0 \quad \Rightarrow \quad y = 1.419 x \]

Normalize:

\[ \vec{v}_1 \approx \begin{bmatrix} 0.576 \\ 0.817 \end{bmatrix}, \quad \vec{v}_2 \approx \begin{bmatrix} 0.817 \\ -0.576 \end{bmatrix} \] \[ V = \begin{bmatrix} 0.576 & 0.817 \\ 0.817 & -0.576 \end{bmatrix} \]

Step 4: Compute Matrix \( U \)

The eigenvalues of the matrices \( A^T A \) and \( A A^T \) are equal to the squares of the singular values of \( A \). This follows from the relation \( A \vec{v}_i = \sigma_i \vec{u}_i \). Multiplying both sides by \( A^T \) gives \( A^T A \vec{v}_i = \sigma_i^2 \vec{v}_i \), which shows that \( \vec{v}_i \) is an eigenvector of \( A^T A \) with eigenvalue \( \lambda_i = \sigma_i^2 \). Similarly, multiplying by \( A \) leads to \( A A^T \vec{u}_i = \sigma_i^2 \vec{u}_i \), so \( \vec{u}_i \) is an eigenvector of \( A A^T \) with the same eigenvalue. Hence, the eigenvalues of both matrices are the squares of the singular values.

The columns of \( U \) are obtained via:

\[ \vec{u}_i = \frac{1}{\sigma_i} A \vec{v}_i \] \[ A \vec{v}_1 = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 0.576 \\ 0.817 \end{bmatrix} \approx \begin{bmatrix} 2.210 \\ 4.996 \end{bmatrix}, \quad \vec{u}_1 = \frac{1}{5.466} \begin{bmatrix} 2.210 \\ 4.996 \end{bmatrix} \approx \begin{bmatrix} 0.404 \\ 0.915 \end{bmatrix} \] \[ \vec{u}_2 \approx \begin{bmatrix} -0.915 \\ 0.404 \end{bmatrix}, \quad U = \begin{bmatrix} 0.404 & -0.915 \\ 0.915 & 0.404 \end{bmatrix} \]

Step 5: Construct the SVD

\[ \Sigma = \begin{bmatrix} 5.466 & 0 \\ 0 & 0.366 \end{bmatrix}, \quad A = U \Sigma V^T \]

This decomposition can be verified by multiplying \( U \Sigma V^T \), which reconstructs \( A \).

Why SVD Matters: Real-World Applications

1. Image Compression

By keeping only the largest singular values (low-rank approximation), SVD can reduce image file size significantly while maintaining visual quality.

2. Recommender Systems

SVD is the foundation of Collaborative Filtering, used by companies like Netflix to predict user ratings for movies.

3. Latent Semantic Analysis (NLP)

In Natural Language Processing, SVD helps identify hidden relationships between documents and terms.

4. Principal Component Analysis (PCA)

SVD provides a numerically stable way to perform PCA for dimensionality reduction in high-dimensional datasets.

Implementing SVD in MATLAB

MATLAB is highly optimized for linear algebra. You can compute the Singular Value Decomposition using the built-in svd() function. This is widely used in engineering and signal processing applications.

% Define the matrix A
A = [1, 2; 3, 4];

% Compute the SVD
% U: Left singular vectors
% S: Diagonal matrix of singular values
% V: Right singular vectors (Note: MATLAB returns V, not V-transpose)
[U, S, V] = svd(A);

% Display results
disp('U Matrix:');
disp(U);

disp('Singular Values (Diagonal Matrix S):');
disp(S);

disp('V Matrix:');
disp(V);

% Verification: Reconstruct A
A_reconstructed = U * S * V';
disp('Reconstructed Matrix A:');
disp(A_reconstructed);

Pro Tip: For very large, non-square matrices, use [U, S, V] = svd(A, 'econ') to perform an "economy-size" decomposition, which saves memory by removing unnecessary zero-padding in the Σ matrix.

Implementing SVD in Python (NumPy)

In practice, you rarely compute SVD by hand. Here is how to do it using Python's NumPy library:


import numpy as np

# Define the matrix A
A = np.array([[1, 2], [3, 4]])

# Perform SVD
U, s, Vt = np.linalg.svd(A)

print("U Matrix:\n", U)
print("Singular Values:", s)
print("V Transpose:\n", Vt)

SVD vs. Eigendecomposition

Feature Eigendecomposition SVD
Matrix Shape Only Square Matrices Any m x n Matrix
Existence Not always exists Always exists
Orthogonality Not necessarily orthogonal U and V are orthogonal

Frequently Asked Questions

What is the difference between SVD and PCA?

PCA is a specific application of SVD where the data is centered around its mean. SVD is the mathematical engine that makes PCA possible.

Can SVD be used for non-square matrices?

Yes, unlike eigendecomposition, SVD is defined for any rectangular matrix, making it much more versatile for real-world data.

Summary

  • Existence: SVD exists for every matrix, unlike eigendecomposition.
  • Uniqueness: Singular values are unique. Singular vectors are unique up to a sign flip, but their subspaces are fixed.
  • Applications: SVD is used in PCA, pseudoinverse computation, and low-rank matrix approximation.
Return to Matrix Decomposition Main Page →

Try Interactive Online Simulators


Further Reading

  1. Eigen Value and Eigen Vector (Eigendecomposition)
📊 MATLAB & ONLINE SIMULATIONS (Web-Based)
🎓 GATE & UGC - NET SOLUTIONS
📡 WIRELESS & SIGNAL PROCESSING


Contact Us

Name

Email *

Message *

Popular Posts

CMOS Logic Gates: Inverter, NAND, and NOR

CMOS Logic Gates: Inverter, NAND, and NOR Complementary metal–oxide–semiconductor (CMOS) technology forms the backbone of modern digital IC design. Patented by Frank Wanlass in 1963 ( US patent 3,356,858 ), CMOS enables low-power, high-speed, and noise-immune logic circuits. CMOS gates, including inverters, NAND, and NOR, are widely used in both large-scale and small-scale integrated circuits due to their efficiency and versatility. CMOS Inverter A CMOS inverter consists of a complementary pair of PMOS and NMOS transistors. The PMOS transistor connects to the supply voltage V DD , while the NMOS connects to ground. The input voltage V IN controls both gates, and the output voltage V OUT is taken from the connected drains. CMOS inverters are highly energy-efficient and produce minimal heat due to their comple...
Read more

UGC NET Electronic Science Previous Year Question Papers with Solutions

Download Papers and Solutions Exam Pattern Preparation Tips FAQs More Home / Engineering & Other Exams / UGC NET 2026 PYQ 📊 Exam Highlights: Electronic Science (88) Feature Details Junior Research Fellowship (JRF) ₹37,000 + HRA per month Eligibility M.Sc/M.Tech in Electronics (55%) Validity of Certificate JRF (3 Years) | Lectureship (Lifetime) 📥 Download UGC NET Electronics PDFs Complete collection of previous year question papers, answer keys and explanations for Subject Code 88. Start Downloading 📂 View All Question Papers June 2026 - Question Paper Download PDF June 202...
Read more

Design of CMOS Flip-Flops (SR, D, JK)

Design of CMOS Flip-Flops (SR, D, JK) A flip-flop or latch is a circuit with two stable states, used to store state information. It is the basic storage element in sequential logic and a fundamental building block in digital electronics systems, including computers and communication devices. Flip-flops and latches act as data storage elements for states, pulse counting, and synchronization of variably-timed input signals to a reference clock. Flip-flops can be transparent/opaque (latches) or clocked (synchronous, edge-triggered). Latches are level-sensitive, while flip-flops are edge-sensitive. In sequential logic, the output depends on current inputs and previous states. Fig.1 shows a sequential circuit combining a combinational block and a memory element. ...
Read more

Flat vs Frequency Selective Online Simulator

Flat vs Frequency Selective Online Simulator Channel Type Without Fading Flat Fading Multipaths Nakagami m SNR(dB) Run Simulation Input Signal Signal After Fading Constellation Diagram BER vs SNR Explore Advanced Flat vs Frequency-Selective Fading Simulator Want to see these equations in action? Visualize it. Launch Simulator Tool Flat Fading and Frequency-Selective Fading Effects of Multipath Components and the Nakagami-m Parameter 1. Flat Fading Flat fading occurs when the signal bandwidth is smaller than the channel's coherence bandwidth, meaning that all frequency components of the transmitted signal experience approximately the same fading. It is typically associated with narrowband signals, where the multipath delay spread is small relative to the symbol duration. ...
Read more

Linear vs Circular Convolution

Linear vs Circular Convolution Linear convolution vs circular convolution Linear convolution (what nature does) If: x[n] = transmitted signal h[n] = channel impulse response (multipath) Then the received signal is: y[n] = x[n] * h[n] This means: Past symbols spill into future symbols Causes inter-symbol interference (ISI) DFT of y[n] is not a simple product Circular convolution (what DFT assumes) The DFT assumes periodic signals : y[n] = x[n] ⊛ h[n], where indices wrap around modulo N Only under circular convolution does the DFT satisfy: Y[k] = X[k] · H[k] This is what enables one-tap frequency-domain equalization . What happens in the real channel (linear convolution) The channel performs linear convolution: y[n] = Σ l=0 Lh−1 h[l] x[n-l] But because: CP absorbs the “spillover” Receiver discards CP before DFT The useful part of the received symbol becomes: y[n] = Σ l=0 Lh−1 h[l] x[(n-l) mod N] Modulo inde...
Read more

MUSIC Algorithm Explained (with MATLAB + Simulator)

Practical Implementation of the MUSIC Algorithm The focus is on how the algorithm works computationally , not just theory, and it explains the denominator (a H E n E n H a) mathematically and intuitively. 1. Introduction The MUSIC (Multiple Signal Classification) algorithm is a high-resolution method used in signal processing and array processing to estimate the Direction of Arrival (DOA) of signals received by a sensor array. Unlike classical beamforming methods, MUSIC uses eigenvector decomposition of the covariance matrix to separate the signal subspace and noise subspace , allowing it to achieve much higher angular resolution. In practical implementations, MUSIC works by: Simulating or collecting array signals Computing the covariance matrix Performing eigenvalue decomposition Separating signal and noise subspaces Scanning possible angles using a steering vector Constructing a pseudo-spectrum where peaks indicate signal directions 2. Signal Mo...
Read more

Decision Feedback Equalizer (DFE) in MATLAB

  MATLAB Code M = 4; % QPSK numSymbols = 10000; numTrainingSymbols = 5000; chtaps = [1 0.5*exp(1i*pi/6) 0.1*exp(-1i*pi/8)]; % Generate QPSK symbols data = randi([0 M-1], numSymbols, 1); tx = pskmod(data, M, pi/4); % DFE setup eq = comm.DecisionFeedbackEqualizer; eq.ReferenceTap = 1; eq.NumForwardTaps = 5; eq.NumFeedbackTaps = 3; % SNR sweep snrVec = 0:5:25; % 0,5,10,...25 dB berBefore = zeros(size(snrVec)); berAfter = zeros(size(snrVec)); for k = 1:length(snrVec) % Apply channel and AWGN rx = awgn(filter(chtaps,1,tx), snrVec(k), 'measured' ); % BER before equalization rxDataNoEq = pskdemod(rx, M, pi/4); [~, berBefore(k)] = biterr(data, rxDataNoEq); % Equalize [y,~,~] = eq(rx, tx(1:numTrainingSymbols)); % Phase correction phaseOffset = angle(mean(conj(tx(1:numTrainingSymbols)) .* y(1:numTrainingSymbols))); yCorrected = y * exp(-1i*phaseOffset); % BER after equalization rxDataEq = pskdemod(yCorrected, M, pi/4); [...
Read more

Superheterodyne Receiver Online Simulator

Superheterodyne Receiver Simulator Simulation Controls Message Frequency (Hz): Carrier Frequency (Hz): Intermediate Frequency (Hz): Sampling Frequency (Hz): Simulation Duration (s): Run Simulation Signal Plots Numerical Values (first 10 samples) Superheterodyne Receiver Workflow The superheterodyne receiver converts a received RF signal to an intermediate frequency (IF) to simplify filtering and amplification, and finally recovers the original message signal. Steps: Input Message Signal: A low-frequency signal m(t) (e.g., sine wave). RF Modulation (AM): The message modulates the carrier with amplitude modulation: RF(t) = [1 + K_a * m(t)] * cos(2Ï€ f_c t) Mixing with Local Oscillator (LO): The RF signal is multiplied by a local oscillator LO(t) = cos(2Ï€ f_LO t) to produce: IF(t) = RF(t) * LO(t) Intermediate Frequency Filtering: A low-pass filter extracts t...
Read more