Reference

Poros — Official Validation Results (NVIDIA A100 80GB PCIe)

These are the official, reproducible benchmark numbers for the architecture families measured in this A100 run (qwen2, qwen3, gemma3, gemma4) on an A100 80GB PCIe (sm_80, PCIe Gen4). Every cell is measured — no number is interpolated or estimated.

This is the clean, same-attention-path matrix the B300 report planned: all three methods (poros / resident / unsloth) run on the same default fast attention path (deterministic: false), so the slowdown column is the pure cost of blockwise residency with no math-vs-flash gap mixed in. It is also fully standardized on the synthetic dataset (license-free, zero download).

Configuration

Field	Value
Hardware	NVIDIA A100 80GB PCIe (80 GB VRAM, PCIe Gen4)
Precision	NF4 (the only bitwise path in the validated matrix)
Steps × passes	200 steps × 3 passes (warmup discarded; median ± std across passes)
Sequence length	512
Batch / grad-accum	1 / 1 (so 1 step = 1 optimizer step for every method)
LoRA	rank 16, alpha 32, dropout 0.0
Block size (poros)	4
Dataset	`synthetic` (deterministic, license-free random tokens)
Methods	poros (blockwise residency), resident (full QLoRA), unsloth 2026.6.1
Software stack	torch 2.10.0+cu128, transformers 5.5.0, bitsandbytes 0.49.2, peft 0.19.1
Attention path	same default fast path for all methods (`deterministic: false`)

Bitwise parity is a separate gate (tests/test_gpu_parity.py), run on the deterministic path that the parity guarantee ships on. It asserts torch.equal(resident, blockwise) — exact, not allclose. 0.00e+00 = bit-for-bit identical.

Bitwise parity

Full bitwise parity is gated separately per the paper's Table 3: Qwen2.5 and Qwen3 across the datacenter/consumer matrix, and Gemma3/Gemma4 on RTX 5090 (block-streamed loader; receipts under validation/streamed-load/sweep/). This A100 run reports throughput and VRAM.

Benchmark matrix

Model	Arch	poros s/step (median±std)	resident s/step	unsloth s/step	Slowdown vs resident	vs unsloth	poros VRAM	resident VRAM	unsloth VRAM	VRAM cut vs resident	vs unsloth
Qwen2.5-32B	qwen2	2.0489±0.0091	0.7662	0.9086	2.67×	2.25×	10.95 GB	34.72 GB	20.82 GB	68.4%	47.4%
Qwen3-32B	qwen3	2.1832±0.0040	0.8157	0.9287	2.68×	2.35×	11.03 GB	36.00 GB	20.77 GB	69.4%	46.9%
Gemma3-27B	gemma3	1.8933±0.0048	0.7835	0.9365	2.42×	2.02×	9.95 GB	33.17 GB	22.06 GB	70.0%	54.9%
Gemma4-31B	gemma4	2.2537±0.0055	0.9889	1.0255	2.28×	2.20×	10.65 GB	38.75 GB	20.57 GB	72.5%	48.2%

The raw per-cell artifacts (with full provenance: torch/cuda/transformers/bitsandbytes versions, GPU name, per-pass timings) are in validation/official_a100_consolidated.jsonl.

Takeaways

~68–73% VRAM cut vs resident, ~47–55% vs unsloth. Every 27–32B model fine-tunes in 10–11 GB instead of 33–39 GB resident / 20–22 GB unsloth.
Same-path slowdown is 2.3–2.7× vs resident on the A100 — measured with every method on the identical attention path, so this is the pure streaming cost. It confirms the slowdown ratio is hardware-dependent: ~2.3–2.7× on A100 (PCIe Gen4) vs ~3.5–4.3× on the B300 (whose resident baseline is so fast that the fixed PCIe streaming cost dominates the ratio).
This run adds the first published 3-method A100 benchmark numbers for the Gemma rows. Their bitwise parity is gated on RTX 5090 (see Bitwise parity above).

Reproduce

# bitwise parity gate (per arch)
POROS_GPU_TEST_MODEL=Qwen/Qwen2.5-32B  pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=Qwen/Qwen3-32B    pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=google/gemma-3-27b-pt pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=google/gemma-4-31B-it pytest -m gpu tests/test_gpu_parity.py

# 3-method benchmark matrix (per model; gemma repos are gated — needs HF_TOKEN)
poros bench matrix configs/bench/official_a100_qwen25_32b.yaml -o out/
poros bench matrix configs/bench/official_a100_qwen3_32b.yaml  -o out/
poros bench matrix configs/bench/official_a100_gemma3_27b.yaml -o out/
poros bench matrix configs/bench/official_a100_gemma4_31b.yaml -o out/

Downstream capability check

One Qwen2.5-32B NF4 adapter was trained end-to-end with Poros Blockwise (oasst1, 100 steps, rank-16 LoRA; the recorded run is on the RTX PRO 6000) and evaluated with the LM Evaluation Harness: zero-shot MMLU is preserved (81.6% with the adapter vs 83.0% base on a fixed 1,824-question subset, about 1.1 combined standard errors). Raw harness JSONs, the per-step training record, and the full protocol (including a note on format-sensitive generative benchmarks) are in docs/validation/downstream-eval/.