Information and signals

information theory formulaschannel capacitysampling and quantisationShannon and Nyquist

Entropy, compression, error-correcting bits, channel capacity and sampling: what a signal carries and what a channel will pass.

Shannon Entropy of a Binary Source

H = -p\log_2 p - (1 - p)\log_2(1 - p)

Average information carried by each symbol of a two-outcome source, in bits. It peaks at exactly 1 bit for a fair coin and falls to zero as the outcome becomes certain.

Compression Ratio

\mathit{CR} = \frac{S_u}{S_c}

How many times smaller a file got: original size divided by compressed size. A ratio of 4 means the file shrank to a quarter, which is a 75% saving, not a 25% one.

Hamming Code Parity Bits

2^{r} \ge m + r + 1

Parity bits r needed to correct any single-bit error in m data bits. The r check bits must be able to name every bit position plus the no-error case, which is where the +1 comes from.

Shannon-Hartley Channel Capacity

C = B\log_2\!\left(1 + \frac{S}{N}\right)

Highest error-free bit rate a channel of bandwidth B can carry at a given signal-to-noise power ratio. Bandwidth in hertz gives a capacity in bits per second.

Nyquist Bit Rate (Noiseless Channel)

C = 2B\log_2 M

Maximum bit rate through a perfectly noiseless channel of bandwidth B using M distinct signal levels. It is the theoretical best case, and noise is what stops you reaching it.

Bit Rate from Baud Rate

R = B\,n

Bit rate from a symbol rate and the number of bits each symbol carries. Baud counts symbols per second; a 16-QAM symbol carries 4 bits, so 2400 baud is 9600 bits per second.

Nyquist Sampling Rate

f_s = 2 f_{max}

Minimum sampling rate that captures a signal without aliasing: twice the highest frequency present. Sample any slower and the lost detail reappears as a false low-frequency tone.

Quantization Signal-to-Noise Ratio

\mathit{SNR} = 6.02n + 1.76

Best signal-to-noise ratio in decibels an ideal n-bit converter can achieve on a full-scale sine wave. Every extra bit is worth almost exactly 6 dB.

How they fit together

Read this set in three movements. First, how much information is there. Shannon entropy is the answer for a two-outcome source, and it peaks at exactly one bit per symbol for a fair coin, falling to zero as the outcome becomes predictable. That number is a hard floor: no lossless compressor beats it, ever, which is why a compression ratio of 10:1 on text and 1.02:1 on an already-compressed video are both correct answers about the same algorithm. Hamming parity bits then run the trade in reverse, spending redundancy on purpose so a single flipped bit can be found and fixed.

Second, how much will the wire carry. Shannon-Hartley is the real ceiling because it admits noise, and Nyquist's bit rate is the fantasy version for a channel with none. Compare them on the same line and the gap is what noise costs you. Bit rate from baud is the practical middle: pick a modulation, get bits per symbol, multiply. The classic confusion lives here, so it is worth stating plainly. Baud counts symbols per second and bit rate counts bits, and they are only equal when each symbol carries one bit.

Third, how do you digitise something analogue. Nyquist's sampling rate fixes how often, at twice the highest frequency present, and sampling slower does not merely lose detail. It returns the lost detail as a false low tone that no later filter can remove. Quantisation SNR fixes how finely, at roughly 6 dB of dynamic range per bit, which is where 16-bit audio's 98 dB comes from. Rate and depth are independent choices, and a converter generous on one and stingy on the other is wasting whichever it spent more on.