Shannon Theory

Information theory sets the ultimate limits. Learn information and entropy, source coding (Huffman), channel capacity C = B log₂(1 + SNR), the Shannon limit Eb/N₀ ≥ −1.6 dB, bandwidth vs power trade-offs, and how modern codes approach capacity.

Claude Shannon's 1948 paper proved that reliable communication is possible at any rate below the channel capacity — and impossible above it. Every modem, code and compression algorithm is judged against these limits.

An analogy

Channel capacity is like the maximum flow of a pipe: widening it (bandwidth) or increasing pressure (SNR) raises flow, but no clever trick can exceed it.

Information and entropy

Rare events carry more information. Entropy is the average information per symbol — the minimum average number of bits needed to encode a source (e.g. English text ≈ 1–1.5 bits/letter).

Source coding

Huffman and arithmetic coding approach entropy; practical compressors (ZIP, JPEG, MP3) exploit redundancy and perceptual limits.

Channel capacity

Capacity grows linearly with bandwidth but only logarithmically with SNR. At high SNR, +3 dB gives ≈ +1 bit/s/Hz; at low SNR, capacity is almost proportional to power.

Approaching capacity

Turbo codes (1993), LDPC (DVB-S2, Wi-Fi, 5G data) and polar codes (5G control) operate within ≈ 0.5–1 dB of the Shannon limit. MIMO multiplies capacity with multiple spatial streams.

Formulas

• Information of an event: I = −log₂ p
• Entropy: H = −Σ p_i log₂ p_i
• Shannon–Hartley: C = B log₂(1 + SNR)
• Shannon limit: E_b/N₀ ≥ (2^η − 1)/η → ln 2 = −1.59 dB as η → 0

Worked example: Capacity

Given: B = 1 MHz, SNR = 20 dB

Find: C.

• SNR = 100
• C = 10⁶ × log₂ 101 = 6.66 Mbps

Answer: 6.66 Mbps

Worked example: Telephone line

Given: B = 3.1 kHz, SNR = 30 dB

Find: C.

• C = 3100 × log₂ 1001 = 30.9 kbps

Answer: ≈ 31 kbps — why analog modems topped out near 33.6 kbps

Common mistakes

• Wrong: Using SNR in dB inside log₂(1 + SNR). — Right: Convert SNR to linear first.
• Wrong: Expecting real systems to reach capacity. — Right: Practical systems are a few dB away.

Applications

• System design: Upper bounds for link throughput.
• Coding research: Benchmark for LDPC/polar codes.
• Compression: Entropy limits for data.

Key takeaways

• H = −Σp log₂ p.
• C = B log₂(1 + SNR).
• Eb/N₀ ≥ −1.6 dB.
• Modern codes approach capacity.

Going further

• MIMO capacity: log₂ det(I + H H*·SNR/N_t).
• Rate–distortion: Limits of lossy compression.
• Finite block length: Capacity penalties for short packets.