Better Credit Assignment Boosts LLM Search Agent Training
Researchers propose Contribution-Weighted GRPO (CW-GRPO), a reinforcement learning framework that improves how LLM-based search agents learn to retrieve and reason over information. The method uses an LLM judge to score the utility of each retrieval step and reasoning correctness, then applies these per-round scores to guide credit assignment across entire trajectories. Experiments show 5-6% improvements over standard GRPO on multiple knowledge-intensive benchmarks, with analysis indicating that successful searches concentrate their value in specific rounds rather than spreading it evenly.
Researchers propose Contribution-Weighted GRPO (CW-GRPO), a reinforcement learning framework that improves how LLM-based search agents learn to retrieve and reason over information. The method uses an LLM judge to score the utility of each retrieval step and reasoning correctness, then applies these per-round scores to guide credit assignment across entire trajectories. Experiments show 5-6% improvements over standard GRPO on multiple knowledge-intensive benchmarks, with analysis indicating that successful searches concentrate their value in specific rounds rather than spreading it evenly.
- CW-GRPO integrates process supervision (per-step feedback) into group relative policy optimization to improve search agent training stability and credit assignment
- An LLM judge evaluates retrieval utility and reasoning correctness at each search round, producing contribution scores that rescale trajectory-level rewards
- Tested on Qwen3-8B and Qwen3-1.7B models, CW-GRPO achieves 5.0-6.3% improvements over standard GRPO on knowledge-intensive benchmarks
- Analysis reveals successful search trajectories concentrate contributions in specific rounds, offering insight into how agents should allocate effort during multi-step retrieval tasks
Search-augmented LLMs are becoming critical for production systems that need access to current and specialized information beyond training data. Current RL training methods struggle with either unstable value estimation or poor credit assignment, limiting how well agents learn to search effectively. This work addresses a fundamental training challenge that affects the reliability and performance of retrieval-augmented generation systems at scale.
- Per-round contribution scoring from an LLM judge offers a practical middle ground between expensive process supervision and sparse outcome-only rewards, potentially applicable beyond search agents
- The finding that successful trajectories concentrate value in specific rounds suggests agents may benefit from adaptive search strategies that allocate compute differently across steps
- Improvements on smaller models (1.7B and 8B) indicate the method scales to resource-constrained deployments, relevant for edge and on-device search applications
Our Briefing
Weekly signal. No noise. Built for founders, operators, and AI-curious professionals.
No spam. Unsubscribe any time.
