AR, MA, and ARMA Explained

 

Wide-Sense Stationarity (WSS)

A process {X_t} is Wide-Sense Stationary (or covariance stationary) if it possesses a finite second moment (E[X_t²] < ∞) and satisfies two conditions:

1. Constant Mean: E[X_t] = μ for all t.
2. Time-Invariant Covariance: The autocovariance between any two observations depends solely on the lag τ:
   Cov(X_t, X_t+τ) = γ(τ).
   (This implies a constant variance, where Var(X_t) = γ(0)).

White Noise (WN)

A process {ε_t} is defined as white noise if it is a WSS process with E[ε_t] = 0, constant variance σ², and Cov(ε_t, ε_t+τ) = 0 for all τ ≠ 0. If the variables are also independent and identically distributed (I.I.D.), it is termed "Independent White Noise."

LTI Systems and Discrete Convolution

A stationary time series is characterized as the output of a Linear Time-Invariant (LTI) system driven by a white noise input. In discrete time, the output y_t is determined by the convolution sum of the input sequence ε_t and the system’s impulse response h_k:

y_t = ∑_k=-∞^∞ h_k ε_t-k

Linear Models: AR, MA, and ARMA

The Wold Decomposition Theorem provides the theoretical framework for these models, stating that any WSS process can be expressed as the sum of a deterministic component and a stochastic component (represented as an infinite moving average).

Autoregressive (AR) Model

An AR(p) model expresses the current value of a series as a linear combination of its p previous values plus a stochastic shock.

Mathematical Form:
Y_t = c + φ₁Y_t-1 + φ₂Y_t-2 + ... + φ_pY_t-p + ε_t

• Stationarity Constraint: For an AR process to be stationary, the roots of the characteristic polynomial Φ(z) = 1 - φ₁z - φ₂z² - ... - φ_pz^p must lie outside the unit circle in the complex plane.

Moving Average (MA) Model

An MA(q) model represents the current value as a linear combination of the current and q previous white noise error terms.

Mathematical Form:
Y_t = μ + ε_t + θ₁ε_t-1 + θ₂ε_t-2 + ... + θ_qε_t-q

• Invertibility Constraint: While finite MA models are inherently stationary, they must be invertible (roots of the MA polynomial Θ(z) = 1 + θ₁z + ... + θ_qz^q outside the unit circle) to ensure a unique AR(∞) representation.

Autoregressive Moving Average (ARMA) Model

The ARMA(p, q) model integrates both AR and MA components to provide a parsimonious representation of a stationary process.

Mathematical Form:
Y_t = c + ∑_i=1^p φ_iY_t-i + ε_t + ∑_j=1^q θ_jε_t-j

Summary Table

Process Type	Always WSS?	Stationarity Condition
MA (Moving Average)	Yes	Always WSS as long as coefficients are finite.
AR (AutoRegressive)	Conditional	Roots of AR polynomial must lie outside the unit circle.
ARMA	Conditional	Determined strictly by the AR part stability.

Applications

These models serve as the standard for forecasting and signal analysis in:

• Econometrics: Modeling asset returns, inflation, and GDP.
• Meteorology: Predicting climate patterns and temperature fluctuations.
• Signal Processing: System identification and noise filtering in discrete-time controllers.
• Operations Research: Demand and retail sales forecasting.

The robustness of these applications is derived from the convergence of linear system theory and the statistical properties of WSS processes.

Return to Signal Processing Main Page →