The Alignment Problem

Base language models are trained to predict the next token — not to be helpful, harmless, or honest. Without alignment, even a perfectly capable model may generate toxic content, hallucinate confidently, or refuse to follow reasonable instructions. Alignment shapes model behavior to match human values and preferences.

RLHF Pipeline

Step 1 — SFT: Fine-tune the base model on high-quality human demonstrations of desired behavior. Step 2 — Reward Model: Train a model to predict which of two responses humans prefer, creating a proxy for human judgment. Step 3 — PPO: Use reinforcement learning (PPO) to optimize the SFT model against the reward model, with a KL penalty to prevent drifting too far from the base distribution. This is how ChatGPT, Claude, and Gemini were trained.

DPO: Simplifying Alignment

Direct Preference Optimization (DPO) eliminates the need for a separate reward model and RL training. It reparameterizes the RLHF objective to directly optimize the policy from preference pairs — simpler, more stable, and often more effective. DPO has become the default for many open-source alignment efforts (Zephyr, Llama 3).

Constitutional AI and RLAIF

Anthropic's Constitutional AI uses a written constitution of principles to guide model behavior, with the model itself generating critiques and revisions. RLAIF (RL from AI Feedback) replaces human preference labels with AI-generated feedback, dramatically reducing annotation costs while maintaining alignment quality.

探索人工智能的无限可能

📚 技术文章