llms.blog
RLHF2 articles

RLHF

Articles

  • Reward Model Overoptimization in Large Language Models: How Goodhart's Law, Proxy Exploitation, and KL Drift Degrade Alignment

    Post-training alignment of large language models relies on optimizing a policy toward objectives defined by human intent and preferences. Because querying human evaluators during every step of continuous reinforcement learning or high-throughput rejection sampling is computationally and logistically infeasible, alignment workflows construct a parameterised proxy reward model. Trained on pairwise preference datasets through formulations such as the Bradley-Terry model, this proxy acts as a surrog

    1 min

  • Reinforcement Learning from Human Feedback (RLHF): How Reward Models, PPO, and KL Penalties Align LLMs

    Pre-training large language models on internet-scale text corpora equips them with general linguistic patterns, world knowledge, and broad reasoning heuristics. However, pre-training optimizes next-token prediction: $\mathbb{E}_{x \sim \mathcal{D}} [\log P_\theta(x_t \mid x_{<t})]$. A base model trained purely on next-token prediction reflects the entirety of its web corpus, reproducing hallucinations, toxic phrasing, incorrect code, and unhelpful conversational patterns. To transform an uncons

    1 min