# Poros — Official Validation Results (NVIDIA A100 80GB PCIe)

These are the official, reproducible benchmark numbers for the **architecture
families measured in this A100 run** (`qwen2`, `qwen3`, `gemma3`, `gemma4`) on an
A100 80GB PCIe (sm_80, PCIe Gen4). Every cell is **measured** — no number is
interpolated or estimated.

This is the **clean, same-attention-path matrix** the B300 report planned:
all three methods (poros / resident / unsloth) run on the same default fast
attention path (`deterministic: false`), so the slowdown column is the pure
cost of blockwise residency with no math-vs-flash gap mixed in. It is also
fully standardized on the `synthetic` dataset (license-free, zero download).

## Configuration

| Field | Value |
|---|---|
| Hardware | NVIDIA A100 80GB PCIe (80 GB VRAM, PCIe Gen4) |
| Precision | NF4 (the only bitwise path in the validated matrix) |
| Steps × passes | 200 steps × 3 passes (warmup discarded; median ± std across passes) |
| Sequence length | 512 |
| Batch / grad-accum | 1 / 1 (so **1 step = 1 optimizer step** for every method) |
| LoRA | rank 16, alpha 32, dropout 0.0 |
| Block size (poros) | 4 |
| Dataset | `synthetic` (deterministic, license-free random tokens) |
| Methods | poros (blockwise residency), resident (full QLoRA), unsloth 2026.6.1 |
| Software stack | torch 2.10.0+cu128, transformers 5.5.0, bitsandbytes 0.49.2, peft 0.19.1 |
| Attention path | same default fast path for **all** methods (`deterministic: false`) |

**Bitwise parity is a separate gate** (`tests/test_gpu_parity.py`), run on the
deterministic path that the parity guarantee ships on. It asserts
`torch.equal(resident, blockwise)` — exact, not `allclose`. 0.00e+00 =
bit-for-bit identical.

## Bitwise parity

Full bitwise parity is gated separately per the paper's Table 3: Qwen2.5 and Qwen3
across the datacenter/consumer matrix, and **Gemma3/Gemma4 on RTX 5090**
(block-streamed loader; receipts under
[`validation/streamed-load/sweep/`](validation/streamed-load/sweep/)). This A100
run reports throughput and VRAM.

## Benchmark matrix

| Model | Arch | poros s/step (median±std) | resident s/step | unsloth s/step | Slowdown vs resident | vs unsloth | poros VRAM | resident VRAM | unsloth VRAM | VRAM cut vs resident | vs unsloth |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-32B | qwen2 | 2.0489±0.0091 | 0.7662 | 0.9086 | 2.67× | 2.25× | 10.95 GB | 34.72 GB | 20.82 GB | 68.4% | 47.4% |
| Qwen3-32B | qwen3 | 2.1832±0.0040 | 0.8157 | 0.9287 | 2.68× | 2.35× | 11.03 GB | 36.00 GB | 20.77 GB | 69.4% | 46.9% |
| Gemma3-27B | gemma3 | 1.8933±0.0048 | 0.7835 | 0.9365 | 2.42× | 2.02× | 9.95 GB | 33.17 GB | 22.06 GB | 70.0% | 54.9% |
| Gemma4-31B | gemma4 | 2.2537±0.0055 | 0.9889 | 1.0255 | 2.28× | 2.20× | 10.65 GB | 38.75 GB | 20.57 GB | 72.5% | 48.2% |

The raw per-cell artifacts (with full provenance: torch/cuda/transformers/bitsandbytes
versions, GPU name, per-pass timings) are in
[`validation/official_a100_consolidated.jsonl`](validation/official_a100_consolidated.jsonl).

## Takeaways

- **~68–73% VRAM cut vs resident, ~47–55% vs unsloth.** Every 27–32B model
  fine-tunes in **10–11 GB** instead of 33–39 GB resident / 20–22 GB unsloth.
- **Same-path slowdown is 2.3–2.7× vs resident on the A100** — measured with
  every method on the identical attention path, so this is the pure streaming
  cost. It confirms the slowdown ratio is hardware-dependent: ~2.3–2.7× on
  A100 (PCIe Gen4) vs ~3.5–4.3× on the B300 (whose resident baseline is so
  fast that the fixed PCIe streaming cost dominates the ratio).
- **This run adds the first published 3-method A100 benchmark numbers** for the
  Gemma rows. Their bitwise parity is gated on RTX 5090 (see Bitwise parity above).

## Reproduce

```bash
# bitwise parity gate (per arch)
POROS_GPU_TEST_MODEL=Qwen/Qwen2.5-32B  pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=Qwen/Qwen3-32B    pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=google/gemma-3-27b-pt pytest -m gpu tests/test_gpu_parity.py
POROS_GPU_TEST_MODEL=google/gemma-4-31B-it pytest -m gpu tests/test_gpu_parity.py

# 3-method benchmark matrix (per model; gemma repos are gated — needs HF_TOKEN)
poros bench matrix configs/bench/official_a100_qwen25_32b.yaml -o out/
poros bench matrix configs/bench/official_a100_qwen3_32b.yaml  -o out/
poros bench matrix configs/bench/official_a100_gemma3_27b.yaml -o out/
poros bench matrix configs/bench/official_a100_gemma4_31b.yaml -o out/
```

## Downstream capability check

One Qwen2.5-32B NF4 adapter was trained end-to-end with Poros Blockwise
(oasst1, 100 steps, rank-16 LoRA; the recorded run is on the RTX PRO 6000)
and evaluated with the LM Evaluation Harness: zero-shot MMLU is preserved
(81.6% with the adapter vs 83.0% base on a fixed 1,824-question subset,
about 1.1 combined standard errors). Raw harness JSONs,
the per-step training record, and the full protocol (including a note on
format-sensitive generative benchmarks) are in
[docs/validation/downstream-eval/](validation/downstream-eval/README.md).