Fine-Tuning (LLM)
Fine-Tuning (LLM) is a training process that takes a pretrained foundation model and continues updating its weights on a curated dataset tailored to a specific task or domain; Fine-tuning supplies the model with labeled input-output pairs and runs gradient descent to minimize prediction error on those examples; Fine-tuning is appropriate when prompt engineering alone cannot achieve required consistency, when latency demands shorter prompts than few-shot examples allow, or when proprietary style and tone must be enforced at scale
Fine-Tuning (LLM) is a training process that takes a pretrained foundation model and continues updating its weights on a curated dataset tailored to a specific task or domain. The goal is to encode behavior, tone, format preferences, or specialized knowledge directly into the model rather than engineering prompts to elicit that behavior at inference time.
How it works
Fine-tuning supplies the model with labeled input-output pairs and runs gradient descent to minimize prediction error on those examples. Parameter-efficient methods like LoRA (Low-Rank Adaptation) freeze most weights and train only small adapter matrices, drastically reducing GPU memory requirements. Supervised fine-tuning is often followed by RLHF to align outputs with human preferences.
Key facts
- Data requirement: Effective fine-tuning typically requires hundreds to thousands of high-quality examples, not millions.
- LoRA: Low-Rank Adaptation reduces trainable parameters by injecting small rank-decomposition matrices into attention layers.
- Catastrophic forgetting: Aggressive fine-tuning on narrow data can degrade general capabilities present in the base model.
- Hosted fine-tuning: OpenAI, Anthropic, and Google offer fine-tuning APIs that handle the infrastructure.
For builders
Fine-tuning is appropriate when prompt engineering alone cannot achieve required consistency, when latency demands shorter prompts than few-shot examples allow, or when proprietary style and tone must be enforced at scale. For most product teams, prompt engineering and RAG should be exhausted before investing in fine-tuning, as the data curation and evaluation overhead is substantial.
Sources
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback (InstructGPT). arXiv:2203.02155. arxiv.org
- Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. arxiv.org
- Christiano, P., et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741. arxiv.org
- Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org
- Hugging Face. Training Transformers documentation. huggingface.co
