How to fine-tune a small model on your own data
Most teams reach for fine-tuning before they’ve exhausted prompting and retrieval, and end up burning a week and a few hundred dollars in GPU rental to get a model that performs about the same as a well-written system prompt. Fine-tuning solves a narrower problem than people think: it’s for teaching a model a consistent style, a structured output format, or a domain vocabulary it keeps getting wrong, not for injecting facts it should instead retrieve. If your problem is “the model doesn’t know about our internal product catalog,” that’s retrieval, not fine-tuning. I wrote more on that distinction in fine-tuning vs RAG: which do you actually need.
This tutorial is for operators and solo builders who’ve already decided fine-tuning is the right tool. Maybe you have a support-ticket classifier that needs to speak in your company’s tone, a code assistant that should follow your team’s specific conventions, or a small model you want to specialize for one task so you can run it cheaply instead of paying per-token for a frontier API. I’ll walk through fine-tuning a small open-weight model (1-3 billion parameters) using LoRA on a single rented GPU, from raw data to a model you can actually run.
By the end you’ll have a working LoRA adapter trained on your own examples, merged into a standalone model, and tested locally. Expect to spend $5-30 in compute and half a day of wall-clock time for a first pass on a dataset of a few hundred to a few thousand examples.
what you need
- A base model in the 1B-3B parameter range. I’ll use
meta-llama/Llama-3.2-1B-Instructfrom Hugging Face as the example; Qwen2.5-1.5B-Instruct works the same way. - A Hugging Face account (free) to download gated models like Llama, plus an access token.
- A dataset of at least 200-500 examples in prompt/response pairs. More matters less than quality here, bad examples teach the model bad habits faster than good examples teach good ones.
- A GPU with at least 16GB VRAM. A rented instance works fine, I use a single RTX 4090 or A10 on RunPod or Lambda, roughly $0.40-$0.80/hour.
- Python 3.10+, and comfort running pip installs and Python scripts from a terminal.
- About $10-40 in compute budget for a first training run plus a few retries.
- Optional: Ollama or vLLM installed locally if you want to serve the finished model without renting more GPU time.
step by step
1. Decide the exact task and write 20 examples by hand first
Before touching any code, write out 20 real input/output pairs for the exact task you want. Not a description of the task, actual examples in the format you want the model to produce. This forces you to notice inconsistencies in your own mental model of the task before you’ve burned GPU hours on it.
Expected output: a short file, seed_examples.jsonl, with 20 lines like:
{"prompt": "Classify this support ticket: 'my invoice shows the wrong plan tier'", "response": "category: billing_discrepancy\npriority: medium"}
If it breaks: if you can’t write 20 consistent examples by hand, the task isn’t well-specified enough for fine-tuning yet. Go back and narrow the scope.
2. Assemble the full dataset in prompt/response JSONL format
Expand your 20 seed examples into your full dataset, aiming for 300-2,000 examples depending on task complexity. Keep everything in one JSONL file, one JSON object per line, with consistent prompt and response keys.
python - <<'EOF'
import json
with open("train.jsonl", "w") as f:
for row in your_data: # replace with your actual data source
f.write(json.dumps({"prompt": row["input"], "response": row["output"]}) + "\n")
EOF
Expected output: train.jsonl with one clean JSON object per line, no trailing commas or malformed rows.
If it breaks: run python -c "import json; [json.loads(l) for l in open('train.jsonl')]" to catch bad lines before you start training. A single malformed line will crash the trainer partway through a run, wasting the GPU time already spent.
3. Set up the training environment
Rent a GPU instance (RunPod, Lambda Labs, or Vast.ai all work) and install the training stack.
pip install transformers peft trl accelerate bitsandbytes datasets huggingface_hub
huggingface-cli login # paste your HF access token
Expected output: no import errors when you run python -c "import transformers, peft, trl".
If it breaks: version mismatches between transformers, peft, and trl are the most common failure. Pin versions if you hit an error: pip install transformers==4.46.0 peft==0.13.2 trl==0.11.4 is a combination known to work together as of mid-2026.
4. Load the base model and configure LoRA
LoRA (Low-Rank Adaptation) freezes the base model’s weights and trains a small set of additional matrices instead, which is why you can fine-tune a 1B model on a single consumer GPU rather than needing a multi-GPU cluster. The original method is described in the LoRA paper from Microsoft Research.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
model_name = "meta-llama/Llama-3.2-1B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
Expected output: a printout showing trainable parameters are roughly 0.1-1% of the total, something like trainable params: 2,097,152 || all params: 1,235,814,400.
If it breaks: an out-of-memory error here means drop to a smaller r value (try 8) or add load_in_4bit=True to the model load call using bitsandbytes, which is the QLoRA approach described in the QLoRA paper.
5. Run the training job with TRL’s SFTTrainer
Hugging Face’s TRL library handles the training loop, including tokenization and packing.
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
dataset = load_dataset("json", data_files="train.jsonl", split="train")
def format_example(example):
return f"### Instruction:\n{example['prompt']}\n\n### Response:\n{example['response']}"
config = SFTConfig(
output_dir="./output",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
logging_steps=10,
save_strategy="epoch",
)
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
args=config,
formatting_func=format_example,
)
trainer.train()
Expected output: a decreasing loss curve logged every 10 steps, typically starting around 2.0-3.0 and dropping toward 0.5-1.0 over 3 epochs on a few hundred examples.
If it breaks: if loss doesn’t move at all, your learning rate is likely too low, or your formatting function is producing empty strings. Print a few formatted examples before training starts to sanity-check them.
6. Save the adapter and merge it into the base model
model.save_pretrained("./lora-adapter")
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(model_name)
merged = PeftModel.from_pretrained(base, "./lora-adapter").merge_and_unload()
merged.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")
Expected output: a ./merged-model directory with .safetensors weight files, a config, and tokenizer files, roughly 2-6GB for a 1B model.
If it breaks: if the merge script errors on a shape mismatch, you likely changed target_modules between training runs without clearing the old adapter directory first. Delete ./lora-adapter and retrain.
7. Test the model locally
Copy the merged model down, or convert it to GGUF for Ollama:
ollama create my-finetuned-model -f Modelfile
ollama run my-finetuned-model "your test prompt here"
Expected output: responses that follow your target format and tone on prompts similar to your training examples, and reasonable (not necessarily perfect) behavior on prompts slightly outside that distribution.
If it breaks: if the model repeats training examples verbatim regardless of input, you’ve overfit, drop epochs to 1-2 or add more diverse examples. If it ignores your fine-tuning entirely, check that your merge step actually loaded the adapter weights and didn’t silently fall back to the base model.
common pitfalls
- Training on too few, too similar examples. 50 nearly-identical examples teach the model to memorize a pattern, not generalize a task. Vary phrasing and edge cases in your dataset.
- Skipping a held-out eval set. Always keep 10-15% of your data out of training and check the model against it. Watching training loss alone hides overfitting.
- Fine-tuning to fix a prompting problem. If a better system prompt or a few-shot example fixes the behavior, fine-tuning is unnecessary cost and maintenance overhead.
- Ignoring the base model’s license. Llama models carry Meta’s community license with use restrictions above certain scale thresholds and acceptable-use terms. Read them before deploying commercially, this isn’t legal advice, check with counsel if your use case is ambiguous.
- Not versioning your dataset. When a fine-tune underperforms, you need to know exactly which data produced which adapter. Keep
train.jsonlfiles in git or a similar system, tagged by run.
scaling this
At 10x your original dataset size (a few thousand examples), the workflow doesn’t change, but training time per epoch grows linearly, so budget more GPU hours and consider dropping to 2 epochs since more data reduces the overfitting risk that extra epochs were compensating for. If your source data is scraped from the web rather than hand-written, a tool like the ones covered on proxyscraping.org/blog becomes relevant for collecting that raw data at volume without getting your requests blocked.
At 100x (tens of thousands of examples), single-GPU training starts to strain, both in VRAM for larger batch sizes and in wall-clock time. This is where you’d move to multi-GPU training with accelerate config for data parallelism, and where data quality filtering becomes a real engineering task rather than a manual review, you can’t eyeball 50,000 examples.
At 1000x (hundreds of thousands to millions of examples), you’re no longer really doing lightweight LoRA fine-tuning, you’re approaching full fine-tuning or continued pretraining territory, which needs a training framework built for distributed multi-node jobs and a much larger compute budget, often into the thousands of dollars. At that scale it’s worth asking again whether the underlying problem is better solved with a bigger base model plus retrieval, since dataset curation and infrastructure costs both grow faster than the marginal quality gain.
where to go next
If you’re still deciding whether fine-tuning or retrieval fits your problem, read fine-tuning vs RAG: which do you actually need first. Once you have a fine-tuned model in production, how to choose an embedding model in 2026 is useful if you’re pairing it with a retrieval layer, and how to cut your LLM API bill in half covers the cost side of running a small fine-tuned model instead of a frontier API for repetitive tasks. For more tutorials like this one, browse the blog index.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-18.