Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek-v3-0324· DeepSeekHugging Face LMArena Human Preference 140K

How would you improve LLMs with RL for multi-turn tool use? You can try using tools…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-preview-04-17’s answer.
  1. User

    How would you improve LLMs with RL for multi-turn tool use? You can try using tools within or outside of reasoning traces so that reasoning/cot process has the chance to try different tools and navigate the search space. But what about Q-Learning, or other RL approaches – which one could potentially work apart from PPO/DPO/GRPO and could we automatically/manually design reliable reward functions in single agent or multi-agent training settings?

  2. DeepSeek Assistant

    Improving LLMs with Reinforcement Learning (RL) for multi-turn tool use is an exciting and challenging problem. Below is a structured approach to address your questions, covering RL methods beyond PPO/DPO/GRPO, reward design, and single vs. multi-agent settings:


    1. RL Approaches for Multi-Turn Tool Use

    Beyond PPO/DPO/GRPO, the following RL methods could be effective:

    a) Q-Learning & Deep Q-Networks (DQN)

    • Why? Q-learning is well-suited for discrete action spaces (e.g., selecting tools or reasoning steps).
    • How?
      • Represent the LLM's policy as a Q-function where actions are tool calls or reasoning steps.
      • Use a hybrid approach: LLM generates candidate actions (e.g., tool calls), and the Q-function selects the best one.
    • Challenges:
      • High-dimensional state space (dialog history + tool outputs).
      • Requires careful exploration (e.g., ε-greedy or Boltzmann).

    b) Actor-Critic Methods (A2C, A3C, SAC)

    • Why? More stable than pure value-based methods, handles continuous and discrete actions.
    • How?
      • Actor: LLM generates tool-use trajectories.
      • Critic: Evaluates the value of states/trajectories to guide the actor.
    • Advantage: Can handle partial observability (e.g., dialog history as state).

    c) Monte Carlo Tree Search (MCTS)

    • Why? Explicitly plans over tool-use trajectories, balancing exploration/exploitation.
    • How?
      • Integrate MCTS with LLM rollouts (e.g., AlphaZero-style).
      • Use LLM to propose actions, then simulate future tool interactions.
    • Challenge: Computationally expensive, but feasible for small tool sets.

    d) Hierarchical RL (HRL)

    • Why? Break down multi-turn tool use into high-level planning (goal setting) and low-level execution (tool calls).
    • How?
      • High-level policy selects subgoals (e.g., "retrieve data" → "analyze").
      • Low-level policy picks tools to achieve subgoals.

    e) Imitation Learning + RL

    • Why? Start with expert demonstrations (e.g., human tool-use trajectories), then fine-tune with RL.
    • How? Use behavioral cloning first, then refine with RL (e.g., DQN or PPO).

    2. Reward Design for Tool Use

    Designing rewards is critical. Options include:

    a) Manual Reward Shaping

    • Task Success: +1 for correct final answer, 0 otherwise.
    • Efficiency Penalty: -0.1 per tool call to discourage redundancy.
    • Relevance Bonus: +0.2 for using correct tools in context.

    b) Learned Reward Models

    • Train a reward model (RM) on human preferences (like RLHF):
      • Input: (State, Action, Tool Output).
      • Output: Scalar reward (e.g., correctness, efficiency).
    • Challenge: Requires high-quality preference data.

    c) Multi-Objective Rewards

    • Combine:
      • Task success (primary).
      • Tool efficiency (secondary).
      • User satisfaction (e.g., via feedback).

    d) Intrinsic Rewards

    • Curiosity: Reward novel tool-use combinations.
    • Uncertainty Reduction: Reward actions that reduce entropy in the answer.

    3. Single-Agent vs. Multi-Agent RL

    Single-Agent

    • Pros: Simpler, focuses on tool-use policy.
    • Cons: Harder to model competition/coordination (e.g., if tools are agents).

    Multi-Agent RL (MADRL)

    • Why? Useful if tools are independent agents (e.g., API services).
    • How?
      • Centralized Training, Decentralized Execution (CTDE): Train a central critic, but tools act independently.
      • Cooperative Setting: Tools share rewards (e.g., team success).
    • Challenge: Non-stationarity (tools' policies change concurrently).

    4. Automatic Reward Design

    • Inverse RL (IRL): Infer rewards from expert tool-use trajectories.
    • Adversarial Rewards: Use a GAN-like setup where a discriminator learns to distinguish good/bad tool use.
    • Meta-RL: Optimize the reward function itself via meta-learning.

    5. Practical Recommendations

    1. Start with Imitation Learning: Collect human tool-use trajectories to bootstrap.
    2. Use Hybrid RL: Combine Q-learning (for tool selection) with actor-critic (for reasoning).
    3. MCTS for Exploration: Plan over tool calls when the action space is small.
    4. Multi-Objective Rewards: Balance correctness, efficiency, and user feedback.
    5. Single-Agent First: Only use multi-agent if tools are truly independent actors.

    Key Challenges

    • Credit Assignment: Which tool call led to success/failure?
    • Partial Observability: The LLM may not see the full state of tools.
    • Exploration: Tools may have sparse rewards (e.g., only final answer matters).

    By combining these ideas, you can create a robust RL system for multi-turn tool use that goes beyond PPO/DPO. Would you like to dive deeper into any specific area?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.