Information Gain Calculator

Information gain answers one question a decision tree asks at every node: which attribute should I split on? It measures how much a candidate split lowers the uncertainty about the class label, measured in entropy. The metric is the core of Quinlan's ID3 algorithm (1986) and, with the gain-ratio correction, of its successor C4.5 (1993).

Let the parent node hold class counts c₁, …, c_K with total N. Every step builds on Shannon's entropy (1948), using log base 2 so the unit is bits, with the convention 0·log 0 = 0 so pure groups never produce NaN:

1. Parent entropy. H(parent) = −Σ (cᵢ/N)·log₂(cᵢ/N). This is the uncertainty before the split. It is 0 when the node is pure and maximal (log₂K) when all classes are equally frequent.
2. Child entropies. Each group j the split produces has its own counts and total Nⱼ. Compute H(childⱼ) with the same entropy formula, and its weight wⱼ = Nⱼ / N.
3. Weighted child entropy.Σ wⱼ·H(childⱼ) — also called the conditional entropy or "remainder". It is the uncertainty that is still left after you know which branch a sample fell into.
4. Information gain. Gain = H(parent) − Σ wⱼ·H(childⱼ). This is the drop in uncertainty, and it equals the mutual information between the class and the split — which is why it can never be negative. This tool cross-checks the gain by recomputing it directly as mutual information from the joint count table.
5. Split information and gain ratio. C4.5 adds SplitInfo = −Σ wⱼ·log₂ wⱼ, which grows with the number of branches, and GainRatio = Gain / SplitInfo. This penalises attributes with many values (a unique-ID column would otherwise win with a perfect but meaningless gain). Gain ratio is undefined when split information is 0 — only one group holds any samples.

The parent distribution is derived by summing the child columns, so you never type it separately. Switching the entropy unit from bits to nats (natural log) scales every entropy and the gain by the same factor, so the gain ratio — a ratio of two same-unit quantities — is unchanged.

• Quinlan, J. R. (1986) — Induction of Decision Trees (ID3, defines information gain)
• Quinlan, J. R. (1993) — C4.5: Programs for Machine Learning (split information & gain ratio)
• Shannon, C. E. (1948) — A Mathematical Theory of Communication (entropy)
• scikit-learn — Decision Trees, Mathematical formulation (entropy criterion)

The formulas on this page were last cross-checked against the sources above on 2026-07-12. These are standard, uncontested textbook definitions with no rates or policy that drift.