Singular Value Decomposition (SVD) Singular Value Decomposition (SVD) is a powerful matrix factorization that generalizes the concept of eigendecomposition to any \( m \times n \) matrix. Geometrically, SVD decomposes a linear transformation into three distinct steps: a rotation in the input space, a rescaling along the principal axes, and a second rotation in the output space. Definition For any matrix \( A \in \mathbb{R}^{m \times n} \), the SVD is defined as: \[ A = U \Sigma V^T \] Where: \( U \): An \( m \times m \) orthogonal matrix. Its columns form an orthonormal basis of the output space. \( \Sigma \): An \( m \times n \) rectangular diagonal matrix. The diagonal entries \( \sigma_1 \geq \sigma_2 \geq \dots \geq 0 \) are the singular values, representing the magnitude of scaling along each axis. ...

Eigenvalue Decomposition Eigenvalue decomposition is a fundamental concept in linear algebra that expresses a square matrix as a product of its eigenvectors and a diagonal matrix of its eigenvalues. This decomposition allows a matrix to be diagonalized and simplifies computations involving matrix powers, linear transformations, and analysis of system stability. Definitions Eigenvalue: For a square matrix \( A \), a scalar \( \lambda \) is called an eigenvalue if there exists a nonzero vector \( \vec{v} \) such that: \[ A \vec{v} = \lambda \vec{v} \] Intuitively, this means that multiplying \( A \) by \( \vec{v} \) only scales the vector by \(\lambda\) without changing its direction. Eigenvalues provide important information about the matrix, such as scaling factors in transformations, stability in dynamical systems, and the mod...

LU Decomposition using Doolittle Factorization We can write an m X n matrix A as a product of two matrices, L and U . And A = L*U L = $\ \begin{bmatrix} 1 & 0 & 0 & 0 \\ l21 & 1 & 0 & 0 \\ l31 & l32 & 1 & 0 \\ l41 & l42 & l43 & 1 \end{bmatrix}$ ; U = $\begin{bmatrix} u11 & u12 & u13 & u14 \\ 0 & u22 & u23 & u24 \\ 0 & 0 & u33 & u34 \\ 0 & 0 & 0 & u44 \end{bmatrix}$ L= lower triangular matrix; U= upper triangular matrix Procedure- Choose a matrix ( m X n) (e.g., 3X 3, 3 X 4, 4 X 4, etc.,) Initialize the L and U matrices. For L matrix, take a matrix with all diagonal elements assigned to 1, and the matrix elements above the diagonal are zeroes. L matrix size will be ( m X m ). The values of matrix elements below the main diagonal can be assigned to l 21 , l 31 , etc., and so on. $$\ \begin{bmatrix} 1 & 0 & 0 & 0 \\ l21 & 1 & 0 & 0 \\ l31 & l32 ...

LU Decomposition using Crout’s Method Procedure- Choose a matrix ( m X n) (e.g., 3X 3, 3 X 4, 4 X 4, etc.,) The Crout’s matrix decomposition algorithm differs slightly from the Doolittle method. Doolittle's method returns a unit lower triangular matrix and an upper triangular matrix, while the Crout’s method returns a lower triangular matrix and a unit upper triangular matrix. Initialize the L and U matrices. L matrix size will be ( m X m ). The values of matrix elements below the main diagonal can be assigned to l 21 , l 31 , etc., and so on. And the matrix elements above the diagonal are zeroes. $$\ \begin{bmatrix} l11 & 0 & 0 & 0 \\ l21 & l22 & 0 & 0 \\ l31 & l32 & l33 & 0 \\ l41 & l42 & l43 & l44 \end{bmatrix} $$ l 21 , l 31 , etc. are unknown For matrix U , take a matrix with all diagonal elements assigned to 1, and the matrix elements below the diagonal are zeroes. Size of matrix U will be as same as matrix A (...

Jordan Decomposition The goal of a Jordan decomposition is to diagonalize a given square matrix. If there is an invertible n×n matrix C and a diagonal matrix D such that A=CDC-1, then an n×n matrix A is diagonalizable. Procedure- Choose a square matrix ( m X m) (e.g., 3 X 3, 4 X 4, 5 X 5, etc.,) Otherwise-Pop up error – select number of rows and Columns should be same (or matrix dimension mismatched) For a given matrix, A = $\begin{bmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{bmatrix}$ The aim of Jordan decomposition is to diagonalize a given square matrix A, if A=PDP -1 is possible, where P is an invertible matrix and D is diagonal matrix. We'll go into the specifics of how matrix P and matrix D are formed later. Matrix P and D are derived from matrix A. Firstly, we’ll find the eigen values of the matrix A | A – λ*I | = 0 (I = identity matrix) Or, $\begin{bmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{bma...

Gaussian Elimination with Back Substitution Method Gaussian elimination  is a method in which an augmented matrix is subjected to row operations until the component corresponding to the coefficient matrix is reduced to triangular form. Procedure Choose an n X n matrix Otherwise- Show Pop up – please select number of rows equal to number of Columns Here, we can perform two different types of operations to convert a given matrix into the REF (row echelon) form (a) modify a row by adding or subtracting multiples of another row. (b) multiply/divide a row by a scalar Construct an upper triangular matrix from the given matrix. In the next step convert the diagonal elements to 1s. Example Solve Equations x+2y-3z=1, 2x-y+z=1, x+4y-2z=9 using Gaussian Elimination with Back Substitution method Solution: Total Equations are 3 x+2y-3z=1 … (i) 2x-y+z=1 … (ii) x+4y-2z=9 … (iii) From the aforementioned equations, A = $\begin{bmatrix} 1 & 2 & - 3 \\ 2 & - 1 ...

Gauss-Seidel Method In numerical linear algebra, the Gauss–Seidel method, also known as the Liebmann method or the method of successive displacement, is an iterative method used to solve a system of a strictly diagonally dominant system of linear equations. Procedure Choose an n X n matrix Otherwise- Show Pop up – please select the number of rows equal to the number of Columns Verify that the magnitude of the diagonal item in each row of the matrix is greater than or equal to the sum of the magnitudes of all other (non-diagonal) values in that row so that Otherwise- Show Pop up – entered matrix is not diagonally dominant Ensure that all of the diagonal elements are non-zero as well. a ii ≠ 0 Otherwise- Show Pop up – all of the diagonal elements must be non-zero Decompose the given matrix A into a lower triangular matrix L* , and a strictly upper triangular matrix U : Let’s assume, A linear system of the form  ��=�Ax = b L* = $\begin{bmatrix} a11 & 0 &...

Gauss Jordan Elimination Method Often Gauss Jordan Elimination (GJE) is used to get a matrix to reduced echelon form so it is easy to solve a linear equation. Linear systems can have many variables. These systems can be solved as long as we have one unique equation/variable. For example- two variables need two equations Three variables need three equation to find a unique solution. And ten variables need ten equation and so on. In the same way 4 variables need 4 equation to find a unique solution. Although, Gauss-Jordan elimination works on matrices of any size, they don’t have to be square. But the number of independent linear equations must not be less than number of unknown variables. However, we actually don't need. For solving ‘n’ number of unknown variables ‘n’ number of independent linear equations are enough. On the other hand, the given matrix needs to be square if you are using it to calculate the inverse of the matrix. Procedure Choose an n X n matrix Othe...

Gauss Jacobi Method In numerical linear algebra, the Gauss-Jacobi method (the Jacobi iteration method) is an iterative algorithm for determining the solutions of a strictly diagonally dominant system of linear equations. Procedure Choose an n X n matrix Otherwise- Show Pop up – please select the number of rows equal to the number of Columns Verify that the magnitude of the diagonal item in each row of the matrix is greater than or equal to the sum of the magnitudes of all other (non-diagonal) values in that row so that Otherwise- Show Pop up – entered matrix is not diagonally dominant Ensure that all of the diagonal elements are non-zero as well. a ii ≠ 0 Otherwise- Show Pop up – all of the diagonal elements must be non-zero Decompose the given matrix into a diagonal matrix D, a lower triangular matrix L, and an upper triangular matrix U: Let’s assume, A linear system of the form ��=�Ax = b A = $\begin{bmatrix} a11 & a12 & \cdots & a1n \\ a21 &...

Let’s assume a square matrix A The characteristic equation, | A – λ* I | = 0 (where I is an identity matrix) After calculating the values of λs we attempt to find eigenvectors for corresponding eigenvalues like this For eigenvalue, λ = λ 1 A*x = λ 1 * I*x (where, x is an unknown vector) Or, ( A - λ 1 * I )* x = 0 The value of x is the corresponding eigenvector of λ 1 Power Method for Dominant Eigenvalue Let λ 1 , λ 2 , λ 3 , and λ n be the eigenvalues of an n X n matrix A . λ 1 is called the dominant eigenvalue of A if | λ 1 | > | λ i |, i = 2, 3, ... , n The eigenvectors corresponding to λ 1 are called dominant eigenvectors of A . Procedure Choose an n X n matrix The number of rows and columns should be the same (or matrix dimension mismatched) Like the Jacobi and Gauss-Seidel methods, the power method for approximating eigenvalues is iterative. First, we assume that matrix A has a dominant eigenvalue with corresponding dominant eigenvectors. ...

Determinant using Sarrus rule Procedure Choose an n X n matrix Otherwise-Pop up error – the number of rows and columns should be the same (or matrix dimension mismatched) Although, Sarrus rule is not applicable for a square matrix which size is greater than 3 X 3, but also we can apply it to obtained 3 X 3 matrix after co-factor expansion of a given matrix. For a given matrix, A = $\begin{bmatrix} a11 & a12 & a13 \\ a21 & a22 & a23 \\ a31 & a32 & a33 \end{bmatrix}$ If the number of rows = number of columns = 3 , then, The determinant of this matrix using the Sarrus rule is Write out the first two columns of the matrix to the right of the third column, giving five columns in a row. Then add the products of the diagonals going from top to bottom (solid) and subtract the products of the diagonals going from bottom to top (dashed). det( A ) = a 11 *a 22 *a 33 + a 12 *a 23 *a 31 + a 13 *a 21 *a 32 – a 31 *a 22 *a 13 – a 32 *a 23 *a 11 – a 3...

Cholesky Decomposition Procedures Choose an n X n matrix The chosen matrix should be symmetric Let’s assume, A = $\begin{bmatrix} a1 & a2 & a3 \\ a2 & a4 & a5 \\ a3 & a5 & a6 \end{bmatrix}$ a matrix A is symmetric if A is a square matrix and A = A T Otherwise-Pop up error – entered matrix should be symmetric The chosen matrix should be positive definite as well. a matrix A is positive definite if determinants of all upper-left sub-matrices are positive or, $\left| \begin{matrix} a1 \end{matrix} \right|$ > 0 $\left| \begin{matrix} a1 & a2 \\ a2 & a4 \end{matrix} \right|$ > 0 $\left| \begin{matrix} a1 & a2 & a3 \\ a2 & a4 & a5 \\ a3 & a5 & a6 \end{matrix} \right|\ $ > 0 Otherwise-Pop up error – entered matrix should be positive definite Every symmetric positive definite matrix A can be decomposed into a product of a unique lower triangular matrix L and its transpose. A = L*L T There...

Rows: Cols: Edit Matrix A Edit Matrix B Matrix A Matrix B A + B A × B