LoRA Parameter Calculator — Trainable Params & Adapter Size

Low-Rank Adaptation (LoRA) freezes the base model and inserts a small, trainable low-rank update beside each targeted weight matrix. For a linear layer with weight W ∈ ℝ^(out × in), LoRA learns two matrices — A ∈ ℝ^(r × in) and B ∈ ℝ^(out × r) — and adds their product to the frozen weight. Only A and B are trained, so each module contributes exactly r · (in + out) trainable parameters (Hu et al., arXiv:2106.09685).

1. Resolve the architecture dimensions from the chosen preset (taken from each model's official config.json) or from your custom inputs: hidden size, layers, attention heads, KV heads, head dim, and MLP intermediate size.
2. Find each projection's in/out size. Attention: q_proj is d_model → n_heads·head_dim; k_proj/v_proj are d_model → n_kv_heads·head_dim (narrower under Grouped-Query Attention); o_proj is n_heads·head_dim → d_model. MLP: gate_proj/up_proj are d_model → intermediate and down_proj is intermediate → d_model.
3. Compute per-module LoRA parameters as r · (in + out), sum across the selected modules, then multiply by the number of layers. Alpha is excluded — it only scales ΔW = (α/r)·B·A at inference and does not change the count.
4. Derive the base-model parameter count from the same dimensions (embeddings, per-layer attention and MLP weights, and norms) so the trainable percentage is internally consistent.
5. Adapter file size = trainable × bytes-per-parameter (fp16/bf16 = 2, fp32 = 4), reported in MB (÷ 1024²). Training memory = trainable × (param + gradient at adapter precision + 8 bytes for the Adam fp32 moment states) — 12 bytes per parameter in fp16. The frozen base weights and activations are counted separately.

The result reproduces what Hugging Face PEFT's print_trainable_parameters() prints for the same rank and target_modules. The tool also cross-checks the total two independent ways — the r·(in + out) identity and counting the A and B matrices separately — which must agree to the parameter.

• Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
• Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
• Hugging Face PEFT documentation — reference implementation
• Meta Llama 3 8B — official config.json (architecture dims)

The model architecture dimensions and formulas on this page were last cross-checked against the cited papers and config.json files on 2026-07-02. LoRA sizing is architecture-driven, so figures change only when a model's config changes.