Series idea: fine-tuning small local models on domain-specific data — worth exploring?
Pitch: a hands-on series about fine-tuning small local models (1-8B) on your own domain-specific dataset instead of paying per-token for a frontier API.
Rough shape:
- When does this actually make sense? Honest cost/quality math — API calls vs. a fine-tuned Qwen/Llama/Phi running on a single consumer GPU or a MacBook.
- Building the dataset — turning internal docs, support tickets, or logs into instruction pairs; how much data is actually enough (spoiler: less than people think with LoRA).
- Training — LoRA/QLoRA on a single 24GB card, what breaks, real loss curves.
- Serving it locally — llama.cpp / Ollama / vLLM, quantization choices, latency numbers on real hardware.
- Evals — how to know your 3B fine-tune actually beats the base model on your domain and not just vibes.
Would you read this? Anything you would add or cut — e.g. is the dataset-building part the interesting piece, or the serving part?
0 replies