# Poros on consumer GPUs

One page for everyone fine-tuning on a desktop card: what fits, what you need
besides the GPU, and what to do when you hit OOM.

## What fits where (NF4, rank-16 LoRA, seq_len 512, block_size 4)

| Model | Poros peak VRAM | Fits on 12 GB | Fits on 16 GB | Fits on 24 GB | Resident QLoRA needs ‡ |
|---|---:|:---:|:---:|:---:|---:|
| Qwen2.5-7B  | 6.90 GB  | yes | yes | yes | 15.24 GB |
| Qwen2.5-32B | 11.61 GB | borderline* | fits the budget§ | yes | 44.88 GB |
| Qwen2.5-72B | 18.87 GB | no | no | yes† | 81.57 GB |

§ "Fits the budget" is derived from peak VRAM, not measured: the smallest
card with a committed end-to-end run is 24 GB.

\*11.61 GB peak leaves little margin on a 12 GB card; lower `block_size` and/or
`seq_len` (see OOM playbook below) to trade speed for headroom.

†72B: the recorded end-to-end run (RTX 5090, 32 GB card) measured 1.63 GB
load and 18.13 GB training peak; the sizing figure above (18.87 GB) is the
validation-platform (RTX PRO 6000) 200-step peak. Both fit a 24 GB budget,
but a measured 24 GB end-to-end run is future work. See
[limitations.md](limitations.md).

‡ The "Resident QLoRA needs" references are measured on the RTX PRO 6000
validation platform; consumer-card resident numbers differ slightly — e.g. the
README's RTX 3090 bench records 16.37 GB for 7B PEFT QLoRA (vs 15.24 GB here).

All values are measured (200-step canonical protocol; raw artifacts in
[`validation/`](validation/README.md)). Peak VRAM depends on model, `block_size`,
and `seq_len` far more than on the card: measured 32B peaks span 10.95 GB
(A100) to 11.61 GB (3090 / PRO 6000), so size to the high end. The 32B run
was independently
reproduced on an RTX 5090 at 11.02 GB peak, including a padding-masked real-data pass on oasst1
([`validation/consumer-5090/`](validation/consumer-5090/README.md)).

## What you need besides VRAM

- **Host RAM**: the frozen NF4 base lives in CPU RAM — roughly **~4 GB at 7B,
  ~18 GB at 32B, ~40 GB at 72B** steady-state (from the NF4 storage format).
  Provision to the measured peak process RSS of the **default streamed
  loader: 12.1 GB at 7B, 34.7 GB at 32B, 48.2 GB at 72B**
  ([validation/rss/](validation/rss/), [validation/streamed-load/](validation/streamed-load/README.md)).
  If a run falls back to the standard loader, the one-time checkpoint
  load/quantization transient raises the high-water mark to 15.3 / 62.0 /
  136.5 GB. The streamed base copy is allocated in pinned host memory
  (required for async transfers), plus transient per-block transfer slabs and
  normal process overhead.
- **Disk**: the one-time Hugging Face checkpoint download (e.g. ~65 GB for
  Qwen2.5-32B). `poros doctor` checks free disk for you.
- **PCIe bandwidth**: VRAM savings are portable, but step time is not — blocks
  stream over PCIe every step. Measured: about 1.6× the resident step time
  at 32B on an RTX PRO 6000 and 1.7× at 7B on an RTX 3090 (at 32B no
  consumer card fits the resident baseline at all); slower PCIe links (e.g.
  Gen4 x8) will be proportionally slower per step. Poros trades time for
  memory by design.

## Quick start on a consumer card

```bash
pip install "poros-train[ml]"
poros doctor                              # checks GPU, deps, disk
poros init train32b.yaml -m Qwen/Qwen2.5-32B -d ./my_data.jsonl
poros train train32b.yaml
```

(Everything else — NF4, rank-16 LoRA, block size 4 — is already the validated
default. Ready-made configs live in the repository's `examples/`; the wheel
does not include them.)

or in Python:

```python
from poros import PorosTrainer
trainer = PorosTrainer.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization="nf4", lora_rank=16, lora_alpha=32,
    block_size=4,
)
trainer.train("OpenAssistant/oasst1", steps=200, seq_len=512)
trainer.save("./poros_output/qwen32b-lora")
```

Use NF4. It is the default, the only precision with bitwise (0.00e+00) parity
to a fully resident run, and the only one validated on consumer hardware.

The alternatives are not shortcuts. `int8` doesn't stream at all, so it saves
no VRAM. `bf16_bitwise` does stream — it genuinely uses less memory than a
resident bf16 run — but it is unoptimized, slow, and outside the validated
matrix: a research path, not a way to fit a bigger model. See
[limitations.md](limitations.md).

## OOM playbook

`PorosOOMError` messages tell you the offending sizes. In order:

1. **Halve `block_size`** (4 → 2 → 1). Fewer frozen layers resident at once;
   lowest VRAM at `block_size=1`. Cost: more transfers, slower steps. Never
   affects parity.
2. **Lower the sequence length** (512 → 256): `max_seq_length` in a YAML
   config, `seq_len=` on `PorosTrainer.train()`. Activation memory at the rematerialization
   boundary scales with sequence length.
3. **Lower LoRA rank** if you adjusted it upward; rank-16 is the validated
   default.
4. Close other VRAM consumers (browser, desktop compositor, inference servers)
   — on a desktop the display itself can hold 0.5–1.5 GB.

If the host runs out of RAM instead (process killed, swap thrash), pick a
smaller model — the streamed base is held in pinned CPU memory and must fit.

The streamed NF4 loader is the **default** (`nf4_load_strategy="auto"`). It bounds
load-time GPU residency to under 2 GB at any scale (0.81 GB at 32B, 1.63 GB at 72B),
is byte-identical to the standard load, and **lowers** host RSS (12.1 / 34.7 / 48.2 GB
at 7B / 32B / 72B, vs 15.3 / 62.0 / 136.5 standard). It is what lets a 72B model load
and train on an RTX 5090. See
[validation/streamed-load/](validation/streamed-load/README.md).

## Windows / WSL2

Poros is developed and validated on Linux. CUDA on WSL2 generally works with
recent drivers, but Poros has not been separately validated there; pinned
host-memory transfer bandwidth can be lower under WSL2, which raises step time
(never affects parity or peak VRAM). If you try it, run `poros doctor` first
and a short `poros train` smoke run before a long job. Native Windows is
untested.

## More

- [quickstart.md](quickstart.md) — full API tour
- [limitations.md](limitations.md) — what Poros does not do
- [validation/](validation/README.md) — raw artifacts behind every number above