Shannon Entropy of a Binary Source

Entropy measures surprise. A fair coin gives $H = 1$ bit per flip, the maximum, because you learn the most possible from the answer. A coin that lands heads 75% of the time gives only $0.811$ bits, and a coin that lands heads 99.9% of the time gives $0.0114$ bits: you already knew what was going to happen, so being told adds almost nothing. At $p = 0$ or $p = 1$ the entropy is zero and there is nothing to transmit at all.

That gap between 0.811 bits of content and the 1 bit of storage a naive encoding would use is the entire business model of lossless compression. Shannon's source coding theorem says you can encode a source at its entropy and no lower, which means a 10/90 stream carrying $0.469$ bits per symbol can in principle be squeezed to under half its size, and cannot be squeezed further no matter how clever the algorithm. When a vendor claims 100:1 lossless compression on arbitrary data, this is the theorem they are violating.

Notice the shape of the function rather than any single value. It is symmetric about $p = 0.5$, so 0.25 and 0.75 give exactly the same entropy, and it is flat at the top. Moving from a 50/50 source to a 60/40 source costs you only 0.029 bits, about 3%. Most of the compressible slack lives out at the extremes, which is why real compressors spend their effort on long runs and rare symbols rather than on mildly biased ones. That symmetry also means the formula cannot be run backwards here: every entropy below 1 bit has two probabilities that produce it, and neither has a closed form.