Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash-preview-04-17· GoogleHugging Face LMArena Human Preference 140K

..what's the difference between dqn and double dqn? why is it named like this? pros and…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-2025-04-16’s answer.
  1. User

    ..what's the difference between dqn and double dqn? why is it named like this? pros and cons? be concise.

  2. Gemini Assistant

    Okay, let's break down DQN vs Double DQN concisely:

    Difference:

    • DQN: Calculates the target Q value using the same network to both select the best action in the next state (argmax) and evaluate its value (max operation on that same network). This leads to overestimation bias.
      • Target: r + γ * max_a' Q(s', a'; θ)
    • Double DQN (DDQN): Decouples action selection and value evaluation for the target. It uses the current (online) network to select the best action in the next state, but uses the target network to evaluate the value of that chosen action. This reduces overestimation bias.
      • Target: r + γ * Q(s', argmax_a' Q(s', a'; θ); θ_target)

    Naming:

    • "Double" refers to the idea of using two sets of weights (or effectively two roles for the networks – the online and target network) to decouple the action selection and the value evaluation steps when calculating the target Q value, inspired by Double Q-learning which uses two networks.

    Pros and Cons:

    • DQN:
      • Pros: Conceptually simpler target calculation; a foundational algorithm that showed RL could work with deep networks.
      • Cons: Suffers from significant overestimation bias which can lead to suboptimal policies and training instability.
    • Double DQN:
      • Pros: Significantly reduces overestimation bias; leads to more stable training and often better performance/policies.
      • Cons: Slightly more complex target calculation (though computationally similar); doesn't completely eliminate overestimation.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.