skip to content

Getting started

Quickstart

copy markdown

Get from zero to a trained adapter in under 5 minutes.

Option 1: PorosTrainer (5 lines)

from poros import PorosTrainer

trainer = PorosTrainer.from_pretrained(
    "Qwen/Qwen2.5-32B",
    quantization="nf4",
    lora_rank=16, lora_alpha=32,
)
trainer.train("OpenAssistant/oasst1", steps=200, seq_len=512)  # any HF dataset, or "synthetic"
trainer.save("./poros_output/qwen32b-lora")   # adapter + tokenizer + config

block_size defaults to the validated 4; pass it explicitly only to tune. For full configuration use PorosConfig (see the API reference).

Use your trained adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load the 32B base in NF4 4-bit — a dense load would need ~65 GB of VRAM.
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization_config=bnb, device_map="auto"
)
model = PeftModel.from_pretrained(base, "./poros_output/qwen32b-lora")
tok = AutoTokenizer.from_pretrained("./poros_output/qwen32b-lora")

Or merge into a standalone model for vLLM / TGI: poros export ./poros_output/qwen32b-lora -o ./merged. See examples/inference.py.

Merging dequantizes the base back to bf16, so poros export needs the whole model resident on the GPU — the one thing blockwise streaming does not help with, since it is a training technique. If you trained a model bigger than your card, keep serving the adapter on top of the 4-bit base (Option A in examples/inference.py); Poros tells you this with a clear error rather than an OOM.

from_pretrained loads the model in NF4 (4-bit), attaches LoRA adapters, and sets up blockwise residency. block_size=4 means 4 transformer layers are resident at a time; the rest stream from CPU pinned memory. If you omit block_size, Poros uses the validated default 4 — it is not guessed from available VRAM. block_size is a memory<->speed knob (larger = fewer transfers = faster but more resident VRAM; smaller = less VRAM but slower); it never affects parity.

Option 2: HF drop-in (one line)

If you already have a Hugging Face training script:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import get_peft_model, LoraConfig
import poros

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B", quantization_config=bnb
)
model = get_peft_model(
    model, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
)

model = poros.prepare(model)   # the only Poros line

# standard HF Trainer now trains blockwise (the canonical 32B config peaks at 11.61 GB)

This is the same one line the README leads with. The prepared model is still a compliant nn.Module — HF Trainer, eval callbacks, loss handlers, and AMP all work unchanged. prepare() also detects precision, gates the adapter, and reports which guarantee applies; poros.patch_hf_model is the lower-level engine call it wraps, for when you want to skip the detection and gating.

Option 3: CLI

The CLI trains from a YAML config, and writes you a ready-to-run one:

poros init              # writes train.yaml: model, dataset, output_dir
poros train train.yaml

The generated file has three active fields — everything omitted uses the validated defaults (NF4, rank-16 LoRA, block size 4) — plus the common knobs as comments, ready to uncomment. poros init -m <model> -d <data> pre-fills it; poros schema config prints every available field.

To train on your own .jsonl/.json/.csv file, see poros-trainer.md — a local path in the dataset: field works as-is. If a run is interrupted, poros train train.yaml --resume-from latest resumes from the newest checkpoint in output_dir (written every save_steps steps), bitwise-faithful to an uninterrupted run.

What to expect

The 32B configuration trains at 11.61 GB peak (measured), which fits a 16 GB VRAM budget with 4.39 GB of headroom; the smallest physical card we have measured on directly is 24 GB. The per-step slowdown vs fully-resident QLoRA is hardware- and scale-dependent — measured 1.7× at 7B on an RTX 3090, about 1.6× at 32B on an RTX PRO 6000, 2.28–2.68× on an A100, and ~3.5–4.3× on a B300 — the deliberate memory-for-time tradeoff. On a 24 GB card the resident 32B does not fit at all; Poros does.

The training output is bitwise identical to the same setup run fully resident: 0.00e+00 max difference on loss and adapter weights. (That is a claim about Poros's streaming, not about matching a different framework's numbers — see limitations.md.)

Next steps