Reference

One page for everyone fine-tuning on a desktop card: what fits, what you need besides the GPU, and what to do when you hit OOM.

What fits where (NF4, rank-16 LoRA, seq_len 512, block_size 4)

§ "Fits the budget" is derived from peak VRAM, not measured: the smallest card with a committed end-to-end run is 24 GB.

*11.61 GB peak leaves little margin on a 12 GB card; lower block_size and/or seq_len (see OOM playbook below) to trade speed for headroom.

†72B: the recorded end-to-end run (RTX 5090, 32 GB card) measured 1.63 GB load and 18.13 GB training peak; the sizing figure above (18.87 GB) is the validation-platform (RTX PRO 6000) 200-step peak. Both fit a 24 GB budget, but a measured 24 GB end-to-end run is future work. See limitations.md.

‡ The "Resident QLoRA needs" references are measured on the RTX PRO 6000 validation platform; consumer-card resident numbers differ slightly — e.g. the README's RTX 3090 bench records 16.37 GB for 7B PEFT QLoRA (vs 15.24 GB here).

All values are measured (200-step canonical protocol; raw artifacts in validation/). Peak VRAM depends on model, block_size, and seq_len far more than on the card: measured 32B peaks span 10.95 GB (A100) to 11.61 GB (3090 / PRO 6000), so size to the high end. The 32B run was independently reproduced on an RTX 5090 at 11.02 GB peak, including a padding-masked real-data pass on oasst1 (validation/consumer-5090/).

What you need besides VRAM

• Host RAM: the frozen NF4 base lives in CPU RAM — roughly ~4 GB at 7B, ~18 GB at 32B, ~40 GB at 72B steady-state (from the NF4 storage format). Provision to the measured peak process RSS of the default streamed loader: 12.1 GB at 7B, 34.7 GB at 32B, 48.2 GB at 72B (validation/rss/, validation/streamed-load/). If a run falls back to the standard loader, the one-time checkpoint load/quantization transient raises the high-water mark to 15.3 / 62.0 / 136.5 GB. The streamed base copy is allocated in pinned host memory (required for async transfers), plus transient per-block transfer slabs and normal process overhead.
• Disk: the one-time Hugging Face checkpoint download (e.g. ~65 GB for Qwen2.5-32B). poros doctor checks free disk for you.
• PCIe bandwidth: VRAM savings are portable, but step time is not — blocks stream over PCIe every step. Measured: about 1.6× the resident step time at 32B on an RTX PRO 6000 and 1.7× at 7B on an RTX 3090 (at 32B no consumer card fits the resident baseline at all); slower PCIe links (e.g. Gen4 x8) will be proportionally slower per step. Poros trades time for memory by design.

Quick start on a consumer card

pip install "poros-train[ml]"
poros doctor                              # checks GPU, deps, disk
poros init train32b.yaml -m Qwen/Qwen2.5-32B -d ./my_data.jsonl
poros train train32b.yaml

(Everything else — NF4, rank-16 LoRA, block size 4 — is already the validated default. Ready-made configs live in the repository's examples/; the wheel does not include them.)

or in Python:

from poros import PorosTrainer
trainer = PorosTrainer.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization="nf4", lora_rank=16, lora_alpha=32,
    block_size=4,
)
trainer.train("OpenAssistant/oasst1", steps=200, seq_len=512)
trainer.save("./poros_output/qwen32b-lora")

Use NF4. It is the default, the only precision with bitwise (0.00e+00) parity to a fully resident run, and the only one validated on consumer hardware.

The alternatives are not shortcuts. int8 doesn't stream at all, so it saves no VRAM. bf16_bitwise does stream — it genuinely uses less memory than a resident bf16 run — but it is unoptimized, slow, and outside the validated matrix: a research path, not a way to fit a bigger model. See limitations.md.

OOM playbook

PorosOOMError messages tell you the offending sizes. In order:

1. Halve block_size (4 → 2 → 1). Fewer frozen layers resident at once; lowest VRAM at block_size=1. Cost: more transfers, slower steps. Never affects parity.
2. Lower the sequence length (512 → 256): max_seq_length in a YAML config, seq_len= on PorosTrainer.train(). Activation memory at the rematerialization boundary scales with sequence length.
3. Lower LoRA rank if you adjusted it upward; rank-16 is the validated default.
4. Close other VRAM consumers (browser, desktop compositor, inference servers) — on a desktop the display itself can hold 0.5–1.5 GB.

If the host runs out of RAM instead (process killed, swap thrash), pick a smaller model — the streamed base is held in pinned CPU memory and must fit.

The streamed NF4 loader is the default (nf4_load_strategy="auto"). It bounds load-time GPU residency to under 2 GB at any scale (0.81 GB at 32B, 1.63 GB at 72B), is byte-identical to the standard load, and lowers host RSS (12.1 / 34.7 / 48.2 GB at 7B / 32B / 72B, vs 15.3 / 62.0 / 136.5 standard). It is what lets a 72B model load and train on an RTX 5090. See validation/streamed-load/.

Windows / WSL2

Poros is developed and validated on Linux. CUDA on WSL2 generally works with recent drivers, but Poros has not been separately validated there; pinned host-memory transfer bandwidth can be lower under WSL2, which raises step time (never affects parity or peak VRAM). If you try it, run poros doctor first and a short poros train smoke run before a long job. Native Windows is untested.

More

• quickstart.md — full API tour
• limitations.md — what Poros does not do
• validation/ — raw artifacts behind every number above