Information Theory vs. Physical Reality
Distinguishing Shannon's Noisy-Channel Coding Theorem from the Shannon-Hartley Formula.
In the pantheon of communication engineering, Claude Shannon’s 1948 paper, "A Mathematical Theory of Communication," serves as the bedrock. However, students and engineers often conflate his general theorem on error-free communication with the specific formula used to calculate bandwidth capacity. To understand modern telecommunications, one must distinguish between the Existence Proof and the Physical Bound.
A. Shannon's Noisy-Channel Coding Theorem
The Fundamental Existence Proof
This theorem is the universal law of reliability. It states that for any communication channel, there exists a capacity $C$ such that if the information rate $R$ is less than $C$, then an error-correcting code exists that allows the probability of error to approach zero ($P_e \to 0$).
The General Mutual Information Capacity:
C = maxP(x) I(X; Y)
Key Mathematical Implications:
- Entropy and Mutual Information: $I(X; Y) = H(X) - H(X|Y)$. This defines capacity as the maximum reduction in uncertainty about the transmitter $X$ given the received signal $Y$.
- The $P_e$ Threshold: If $R > C$, the probability of error is bounded away from zero; reliable communication is impossible regardless of coding complexity.
- Non-Constructive: The theorem proves a code exists but does not define the algorithm to build it.
B. The Shannon-Hartley Theorem
The Physical Channel Limit
This is a specific application of the general theorem. It applies the concept of capacity to a continuous-time analog channel subject to Additive White Gaussian Noise (AWGN).
The AWGN Capacity Formula:
C = B log2(1 + S/N)
Key Mathematical Implications:
- Logarithmic Scaling: Doubling the Signal Power ($S$) does not double the capacity; it only increases it logarithmically.
- Linear Bandwidth Scaling: Doubling the Bandwidth ($B$) provides a linear increase in capacity (assuming $S/N$ remains constant).
- The Gaussian Assumption: This formula assumes noise follows a normal distribution, which represents the "worst-case" interference for a given power.
Comparative Analysis
| Feature | Shannon’s Coding Theorem | Shannon-Hartley Theorem |
|---|---|---|
| Scope | Abstract / Information Logic | Physical / Electromagnetic |
| Channel Type | Any (Discrete or Continuous) | AWGN (Analog Band-limited) |
| Variables | Probability Distributions ($P(x)$) | Watts, Hertz, Joules |
| Operational Goal | Reliability (Error Correction) | Efficiency (Spectral Bounds) |
The Mathematical Bridge: Spectral Efficiency
To bridge these two concepts, we analyze Spectral Efficiency ($\eta$), measured in bits per second per Hertz (bps/Hz).
The relationship between the Shannon-Hartley bound and energy efficiency is expressed by:
η = C / B = log2(1 + S/N)
By substituting $S/N = (E_b/N_0) \cdot (R/B)$, and setting $R = C$ (at the limit):
Eb/N0 = (2η - 1) / η
Physical Interpretation:
This equation allows us to visualize the trade-off between power and bandwidth. As bandwidth $B \to \infty$ (and thus $\eta \to 0$), we apply L'Hôpital's rule to the expression above, which yields the Ultimate Shannon Limit: