# Limitations and roadmap Poros is designed around a specific tradeoff: memory for time. The limitations below are stated plainly because the project's credibility depends on honesty about what it does and does not do. ## Current limitations ### Throughput Poros is slower per step than fully-resident NF4 QLoRA because streaming frozen blocks over PCIe costs wall-clock time — the fundamental memory-for-time tradeoff. The slowdown ratio is **hardware-dependent**: measured 1.7× at 7B on an RTX 3090, about 1.6× at 32B on an RTX PRO 6000, 2.28–2.68× on an A100, and ~3.5–4.2× on a B300 (measured — see [official-b300-results.md](official-b300-results.md)). The ratio is *larger* on faster GPUs because the resident baseline gets faster while the PCIe transfer (the streaming cost) does not — a B300 does a 32B resident step in ~0.27 s, so the fixed transfer cost dominates the ratio. Poros competes on feasibility (fits where others OOM), not raw speed. At 32B, fully-resident QLoRA needs ~37–47 GB depending on the GPU, so it OOMs on a 24 GB card. Poros trains the same model at ~11 GB: | Platform | Peak | Protocol | |---|---:|---| | RTX 3090 | 11.61 GB | same-card comparison table | | RTX PRO 6000 | 11.61 GB | canonical campaign (reproduces the 3090 figure) | | RTX 5090 | 11.02 GB | single-pass validation | Peak VRAM follows the model, block size, and sequence length rather than the card, which is why the first two agree to the reported precision. ### What 0.00e+00 means — and what it does not The bitwise claim compares two runs of **Poros itself**: the same model, config, seed, and kernels, once fully resident and once streamed blockwise. Those are `torch.equal` on the loss stream, adapter weights, and optimizer state — streaming adds zero numerical difference. That is the whole claim. It is **not** a claim that Poros reproduces the numbers of a *different* implementation. An unpatched PEFT/HF training loop dispatches its own kernel sequence and reduction order, so its losses differ from a Poros run of the same recipe by normal cross-implementation floating-point noise — measured ~2.6×10⁻³ on a first-step loss at 0.5B (RTX 5090), the same band as the ≤4×10⁻⁴ cross-method final-loss spread in the benchmark table. Neither number is "wrong"; they are different valid FP execution orders. If you need your existing loop's exact trajectory, Poros cannot promise it — no alternative implementation can. What it promises instead is that choosing streaming over residency costs you nothing numerically, and that the guarantee is checked with `torch.equal`, never `allclose`. The claim is also per-platform (see the parity matrix for which GPU/architecture pairs are gated) and NF4-only — bf16/fp16 are experimental and carry no bitwise claim. ### bf16 LoRA (EXPERIMENTAL) Dense-model bf16 streaming ships as `quantization="bf16_bitwise"`: - **Parity is bitwise.** The parity gate measures `max_abs_diff = 0.0` across per-step losses, adapter gradients, adapter weights, and optimizer state — validated up to Qwen2.5-32B on A100 ([receipts](validation/bf16/README.md)). - **Throughput is bandwidth-bound.** bf16 is 4× the bytes of NF4, and the frozen base streams twice per step (forward + remat backward); at 32B the parity probe moved ~440 GB host→GPU per step. Expect several× slower than resident on PCIe — bf16 streaming benefits most from datacenter-class links (Grace-Blackwell NVLink-C2C). NF4 remains the recommended path. `bf16_bitwise` emits an EXPERIMENTAL warning until the full official matrix runs at this precision. fp16 remains experimental and non-bitwise. Arches that are structurally non-bitwise in bf16 (MXFP4-native MoE dequantized to bf16, e.g. gpt-oss) run under a structural self-check tolerance instead — see `poros/core/precision.py`. ### int8 (EXPERIMENTAL, not exposed) int8 is bitwise (0.00e+00 parity) but bitsandbytes caches its compute state (`state.CB`, `state.SCB`, `state.CxB`) as module attributes on the GPU. The block manager offloads parameters but not these cached buffers, so the frozen base remains GPU-resident. Result: Poros int8 uses more VRAM than Unsloth int8 at the same scale (no committed measurement — the structural reason is above). int8 is not exposed in the `quantization` field. The roadmap tracks either fixing the streaming or documenting int8 as non-streaming. ### Single-GPU only The paper validates single-GPU blockwise residency, and Poros ships single-GPU only. Multi-GPU support is **deferred to a future release**: it has not been validated against the blockwise engine, so the current release exposes no distributed surface (no `launch` command, no `poros.distributed`). When multi-GPU lands it will be data-parallel — each GPU runs Poros locally and syncs adapter gradients — improving throughput and effective batch size but NOT reducing per-GPU VRAM. Sharded block residency (splitting the frozen base across GPUs), FSDP, ZeRO, tensor parallel, pipeline parallel, and multi-node remain future work. ### Consumer-card caveats VRAM savings are portable: they depend on the model and block size, not the GPU. But per-step latency depends on PCIe bandwidth. The measured numbers in the README are from an RTX 3090 and an RTX 5090; on narrower links (e.g. Gen4 x8), per-step time is higher. ### Validated architectures only Poros works on the validated dense families — Qwen2.5, Qwen3, Qwen3.5/Qwen3.6, Gemma3, and Gemma4 (see the registry via `poros doctor`, or `poros check ` for one model). Other architectures fail at startup with a clear error. This is by design: the architecture gate (`blockwise_arch_check`) refuses to train on an architecture that is neither validated nor registered by you as `user_asserted`, where parity is unknown. See [architecture-support.md](architecture-support.md) for the validation protocol and how to add support for a new family. ### No custom kernels Poros ships no mandatory custom Triton kernels. It uses standard PyTorch operations. Kernel-level speed parity with tools like Unsloth is not a goal of the current release. ### Pretraining is out of scope Poros exploits a read-only frozen base. The offload is a near-free pointer release, and only tiny adapter optimizer state stays resident. Pretraining breaks these properties: mutable weights require bidirectional PCIe traffic (gradient sync), offload becomes a costly host memory sync, and full-model Adam state is large. Full-parameter blockwise training would effectively become ZeRO, not Poros. --- ## Roadmap Everything below ships only after its own 0.00e+00 parity gate passes — the validated path stays bitwise or it does not ship as validated. No dates. **Precision** - **fp8 support.** Streamed fp8 base weights on hardware with native fp8 compute; halves the transfer volume bf16 pays today. - **int8 streaming.** Release the full bitsandbytes compute state on offload so int8 streams like NF4, or document it explicitly as non-streaming. **Throughput** - **bf16 prefetch.** Coalesce per-block transfers and deepen prefetch overlap for `bf16_bitwise`; bf16 stays a high-bandwidth-bus recommendation until then. - **Faster steps at NF4.** Deeper prefetch and selective recompute. - **The Poros kernel.** A custom-kernel program held to the same bitwise standard as the engine: only fusions that are bit-exact by construction qualify (pointwise chains — SwiGLU multiply, residual adds, rotary with precomputed tables — produce the same per-element values in the same order), and nothing ships without passing the 0.00e+00 parity gates. Reduction-reordering fusions are excluded by the spec, not deferred. **Architectures** - More dense families as their parity gates pass. - **MoE support.** First-class mixture-of-experts streaming, including Gemma 4 MoE if routing under blockwise residency holds parity (today MXFP4-native MoE runs dequantized to bf16 under a structural, non-bitwise self-check — see the bf16 section above). - Multimodal backbones. **Scale-out** - **Multi-GPU support.** FSDP / ZeRO / tensor parallel, multi-node training, and sharded block residency (per-GPU VRAM reduction). **Further out** - Custom Triton kernels.