Activation Function Calculator

An activation functionis applied element-wise to a neuron's pre-activation (the weighted sum plus bias) to introduce the non-linearity a neural network needs to learn. This tool implements the six most common ones exactly as their originating papers define them, and reproduces the behaviour of PyTorch's torch.nn.functional.

• ReLU — max(0, x). Zeroes every negative input; derivative 1 for x > 0, else 0 (Nair & Hinton, 2010).
• Leaky ReLU — x if x>0 else α·x, default slope α = 0.01, so negatives leak a small gradient instead of dying (Maas et al., 2013).
• GELU — x·Φ(x)with Φ the standard-normal CDF, Φ(x) = ½(1 + erf(x/√2)). The tanh approximation ½x(1 + tanh(√(2/π)(x + 0.044715x³))) is the form used inside GPT and BERT (Hendrycks & Gimpel, 2016).
• SiLU / Swish — x·σ(x), sigmoid-gated and smooth; Swish with β = 1 is identical (Elfwing 2017; Ramachandran 2017).
• Tanh — (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ), a zero-centred squashing function bounded in (−1, 1) with derivative 1 − tanh²(x).
• ELU — x if x>0 else α(eˣ−1), default α = 1, which saturates at −α for large negatives (Clevert et al., 2015).

The exact GELU needs the error function erf, computed here with the Abramowitz & Stegun 7.1.26 rational approximation (maximum error 1.5×10⁻⁷), which is comfortably tighter than the six decimals shown. The sigmoid inside SiLU uses the numerically-stable sign-aware branch — 1/(1+e⁻ˣ) when x ≥ 0 and eˣ/(1+eˣ) when x < 0 — so extreme inputs never overflow to Infinity or NaN; SiLU(1e9) returns 1e9 and ELU(−1000) returns −1 cleanly.

To prove the implementation is self-consistent, the module recomputes two of the functions by independent algebraic identities — tanh(x) = 2σ(2x) − 1 and SiLU(x) = x·½(1+tanh(x/2)) — and reports the largest disagreement, which is 0 (≈1e-16) for every input. All arithmetic is double-precision JavaScript; results are rounded only for display, never for chained computation, and nothing is sent to a server.

• Hendrycks & Gimpel (2016) — Gaussian Error Linear Units (GELUs)
• Ramachandran, Zoph & Le (2017) — Searching for Activation Functions (Swish / SiLU)
• Clevert, Unterthiner & Hochreiter (2015) — Exponential Linear Units (ELUs)
• Maas, Hannun & Ng (2013) — Rectifier Nonlinearities (Leaky ReLU)
• PyTorch torch.nn documentation — reference implementations and default constants

The formulas on this page were last cross-checked against these papers and PyTorch on 2026-07-13. Activation functions are stable mathematical definitions, so this tool needs no rate or schedule updates — only the worked examples are periodically re-reconciled.