Mutual Information Calculator

Mutual information quantifies how much two variables tell you about each other. It ties together the rest of information theory: it is the shared region between the two entropies H(X) and H(Y), and it equals the drop in one variable's uncertainty once the other is known. Following Cover & Thomas, for a joint distribution p(x,y) over discrete X (rows) and Y (columns) it is defined as:

I(X;Y) = Σₓ Σᵧ p(x,y) · log_b[ p(x,y) / (p(x)·p(y)) ]

The base of the logarithm sets the unit — base 2 gives bits, base e (the natural log) gives nats, converting with 1 nat = 1/ln2 ≈ 1.4427 bits. The calculator follows the plug-in estimator used by scikit-learn:

1. Read the grid into a matrix. In counts mode, divide every cell by the grand total N to get the joint probabilities p(x,y). In probabilities mode, optionally rescale so the table sums to 1. Reject negative or empty cells and tables smaller than 2×2.
2. Sum across each row for the marginal p(x), and down each column for p(y). These are the distributions of X and Y on their own.
3. Compute the entropies with the convention 0·log0 = 0: H(X) = −Σ p(x) log p(x), likewise H(Y), and the joint H(X,Y) = −Σ p(x,y) log p(x,y).
4. Sum p(x,y)·log[p(x,y)/(p(x)·p(y))] over every cell for I(X;Y). Whenever p(x,y) > 0 both marginals are at least that large, so the ratio is positive and the result is always finite — unlike KL divergence, mutual information never diverges.
5. Derive the conditional entropies H(X|Y) = H(X,Y) − H(Y) and H(Y|X) = H(X,Y) − H(X), and the normalized MI: the arithmetic-mean form 2I/(H(X)+H(Y)) and the min form I/min(H(X),H(Y)), both clamped to 0 when a variable is degenerate.

As a built-in correctness check, the tool computes I(X;Y) two independent ways — the direct per-cell sum above and the entropy identity I = H(X) + H(Y) − H(X,Y) — and shows them reconciling. Two facts double as sanity signals: mutual information is always ≥ 0, and it is exactly 0 if and only if X and Y are independent. Because it is a plug-in estimate, MI is biased slightly upward on small samples; this version applies no Miller–Madow or other bias correction, matching mutual_info_score.

• Cover & Thomas, Elements of Information Theory (2nd ed., 2006), ch. 2 — entropy, joint/conditional entropy and mutual information
• Shannon (1948), “A Mathematical Theory of Communication”, Bell System Technical Journal 27
• scikit-learn — sklearn.metrics.mutual_info_score (reference implementation, natural log)
• scikit-learn — normalized_mutual_info_score (arithmetic-mean normalization)

The formulas and conventions on this page were last cross-checked against Cover & Thomas and scikit-learn on 2026-07-10. The worked examples reproduce to the displayed precision against sklearn.metrics.mutual_info_score.