llms.blog
post-training2 articles

post-training

Articles

  • Distributed RLHF Frameworks in Production: Comparing OpenRLHF, verl, and TRL Architecture, Ray Resource Scheduling, and Rollout-Training Co-Location

    Post-training alignment has shifted from offline preference tuning to large-scale, online reinforcement learning. Modern post-training loops for reasoning models, agentic workflows, and conversational alignment require coordinating multiple distinct neural network roles simultaneously. Under standard Proximal Policy Optimization (PPO), an RL infrastructure pipeline must manage up to four distinct model instances: the Actor (the active policy undergoing gradient updates), the Critic (the value mo

    1 min

  • Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity

    Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity Rejection sampling fine-tuning (RAFT) has emerged as the practical workhorse of LLM alignment. While PPO-based RLHF dominated early literature, production systems from Llama 2 to DeepSeek-R1 rely on a simpler loop: generate multiple completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. The technique converts the reinforcement learning problem into s

    1 min