Entropy Calculator

Calculate Shannon entropy, joint entropy, and information content for probability distributions

Home Categories Math Entropy Calculator

H(X) = -∑ p(x) × log(p(x))

Shannon entropy measures uncertainty in a probability distribution

Probabilities must sum to 1.0

Binary entropy: H(p) = -p×log(p) - (1-p)×log(1-p)

Frequencies will be normalized to probabilities

Quick Examples

Shannon Entropy

Entropy Scale

Predictable (0%) Moderate Maximum (100%)

Maximum Entropy

Normalized

of max

Perplexity

effective states

Redundancy

unused capacity

Probability Breakdown

Event Probability -p×log(p) % of H
Total 1.0000 100%

Step-by-Step Solution

1

Identify probabilities

[]

2

Apply Shannon entropy formula

H = -∑ p × log(p)

3

Calculate each term

...

4

Sum all contributions

H =

Common Entropy Values (bits)

Distribution Entropy Interpretation
Fair coin (50/50) 1.000 1 bit per flip
Biased coin (90/10) 0.469 More predictable
Fair 6-sided die 2.585 log₂(6) bits
DNA bases (A,T,C,G) 2.000 2 bits per base
English letters ~4.11 Non-uniform freq.
Byte (256 values) 8.000 8 bits if uniform

If you like this calculator

Please help us simply by sharing it. It will help us a lot!

Share this Calculator

Related Calculators

Other calculators you might find useful.

Scientific Notation Calculator

Convert numbers to and from scientific notation with step-by-step explanations

Gamma Function Calculator

Calculate the gamma function Γ(z) for real numbers with step-by-step solutions

On Base Percentage Calculator

Calculate OBP - how often a batter reaches base safely

Inverse Variation Calculator

Calculate inverse variation relationships where xy = k (constant)

Antilog Calculator

Calculate antilogarithms (inverse logarithms) with any base including base 10 and natural antilog

Variance Calculator

Calculate population and sample variance with detailed statistics

About Entropy Calculator

What is Entropy?

Entropy in information theory is a measure of the uncertainty or randomness in a probability distribution. It was introduced by Claude Shannon in 1948 and is fundamental to data compression, cryptography, and machine learning.

Shannon Entropy Formula

For a discrete random variable X with probability distribution P:

H(X) = -∑ p(x) × log₂(p(x))

Where:

  • H(X) is the entropy in bits
  • p(x) is the probability of each outcome
  • The sum is over all possible outcomes

Key Properties

Entropy Bounds

  • Minimum entropy: 0 (when one outcome has probability 1)
  • Maximum entropy: log₂(n) (when all n outcomes are equally likely)

Additivity Property

For independent variables X and Y:

H(X,Y) = H(X) + H(Y)

Conditioning Reduces Entropy

H(X|Y) ≤ H(X)

Types of Entropy

1. Shannon Entropy

Measures the average information content per symbol.

2. Joint Entropy

H(X,Y) measures the total uncertainty of two variables together.

3. Conditional Entropy

H(X|Y) measures the remaining uncertainty in X given Y.

4. Mutual Information

I(X;Y) = H(X) + H(Y) - H(X,Y) measures the shared information.

Common Entropy Values

Distribution Entropy (bits)
Fair coin (50/50) 1.0
Fair die (1/6 each) 2.585
Biased coin (90/10) 0.469
Certain outcome 0

Applications

  1. Data Compression - Optimal encoding uses ≈H bits per symbol
  2. Cryptography - High entropy = hard to predict
  3. Machine Learning - Decision trees, information gain
  4. Communications - Channel capacity limits
  5. Statistical Mechanics - Thermodynamic entropy

Example Calculation

For a coin with P(heads) = 0.7, P(tails) = 0.3:

H = -[0.7 × log₂(0.7) + 0.3 × log₂(0.3)] H = -[0.7 × (-0.515) + 0.3 × (-1.737)] H = -[-0.360 + (-0.521)] H = 0.881 bits

Entropy vs Randomness

  • High entropy = More uncertainty = More random
  • Low entropy = Less uncertainty = More predictable
  • Zero entropy = Completely deterministic