Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

What changese between this implementation of gae lambda: lastvalues =…

  1. User

    What changese between this implementation of gae lambda:

        last_values = last_values.clone().cpu().numpy().flatten()
    
        last_gae_lam = 0
        for step in reversed(range(self.buffer_size)):
            if step == self.buffer_size - 1:
                next_non_terminal = 1.0 - dones
                next_values = last_values
            else:
                next_non_terminal = 1.0 - self.episode_starts[step + 1]
                next_values = self.values[step + 1]
            delta = self.rewards[step] + self.gamma * next_values * next_non_terminal - self.values[step]
            last_gae_lam = delta + self.gamma * self.gae_lambda * next_non_terminal * last_gae_lam
            self.advantages[step] = last_gae_lam
        # TD(lambda) estimator, see Github PR #375 or "Telescoping in TD(lambda)"
        # in David Silver Lecture 4: https://www.youtube.com/watch?v=PnHCvfgC_ZA
        self.returns = self.advantages + self.values
    

    and mine:

    def compute_advantages(self, rewards, dones, values, gamma=0.99, lambda_=0.95):
        advantages = np.zeros_like(rewards)
        last_adv = 0
        for t in reversed(range(len(rewards))):
            if dones[t]:
                next_non_terminal = 0
                next_value = 0
            else:
                next_non_terminal = 1
                if t + 1 < len(values):
                    next_value = values[t + 1]
                else:
                    next_value = 0  # Or use a bootstrap value if available
            delta = rewards[t] + gamma * next_value * next_non_terminal - values[t]
            advantages[t] = delta + gamma * lambda_ * next_non_terminal * last_adv
            last_adv = advantages[t]
        return advantages
    
  2. Assistant

    Both implementations aim to compute the Generalized Advantage Estimation (GAE) λ, which is used to estimate advantages in reinforcement learning algorithms like Proximal Policy Optimization (PPO). However, there are several key differences between the two implementations that can affect their behavior and the resulting advantage estimates. Here's a detailed comparison:

    1. Handling Episode Boundaries

    Original Implementation:

    if step == self.buffer_size - 1:
        next_non_terminal = 1.0 - dones
        next_values = last_values
    else:
        next_non_terminal = 1.0 - self.episode_starts[step + 1]
        next_values = self.values[step + 1]
    
    • self.episode_starts: This array indicates whether a new episode starts at a particular step. By using 1.0 - self.episode_starts[step + 1], the code ensures that if a new episode starts at step + 1, the next_non_terminal flag is set to 0. This is crucial when the buffer contains concatenated episodes, preventing the advantage from incorrectly propagating across episode boundaries.

    User's Implementation:

    if dones[t]:
        next_non_terminal = 0
        next_value = 0
    else:
        next_non_terminal = 1
        if t + 1 < len(values):
            next_value = values[t + 1]
        else:
            next_value = 0  # Or use a bootstrap value if available
    
    • dones[t]: This flag only indicates whether the current step is the terminal step of an episode. It doesn't account for whether a new episode starts immediately after the current one unless dones are explicitly set for episode boundaries.

    Implications:

    • Original Implementation: More robust in handling buffers with multiple concatenated episodes because it explicitly checks for episode starts.
    • User's Implementation: Might incorrectly propagate advantage estimates across episode boundaries if multiple episodes are stored sequentially in the buffer without proper dones flags.

    2. Bootstrapping with last_values

    Original Implementation:

    if step == self.buffer_size - 1:
        next_values = last_values
    
    • last_values: This represents the value estimate at the end of the buffer, allowing the GAE calculation to bootstrap from the last state if it's not terminal.

    User's Implementation:

    else:
        next_value = 0  # Or use a bootstrap value if available
    
    • Bootstrap Value Handling: The user’s implementation sets next_value to 0 when t + 1 exceeds the buffer length, unless an external bootstrap value is provided. This might not utilize the actual last_value from the environment, potentially leading to less accurate advantage estimates.

    Implications:

    • Original Implementation: Properly incorporates a bootstrap value (last_values) for the last step in the buffer, enhancing the accuracy of the advantage estimates when the episode continues beyond the buffer.
    • User's Implementation: Defaults to 0 unless a bootstrap value is manually provided, which might not capture the true continuation of the episode.

    3. Advantage Calculation and Accumulation

    Original Implementation:

    delta = self.rewards[step] + self.gamma * next_values * next_non_terminal - self.values[step]
    last_gae_lam = delta + self.gamma * self.gae_lambda * next_non_terminal * last_gae_lam
    self.advantages[step] = last_gae_lam
    

    User's Implementation:

    delta = rewards[t] + gamma * next_value * next_non_terminal - values[t]
    advantages[t] = delta + gamma * lambda_ * next_non_terminal * last_adv
    last_adv = advantages[t]
    
    • Accumulation Variable:
      • Original: Uses last_gae_lam to accumulate the advantage.
      • User's: Uses last_adv for accumulation.

    Implications:

    • Functionally, both accumulate the advantage estimates in a similar backward recursive manner. The primary difference lies in the handling of next_non_terminal and next_values.

    4. Return Calculation

    Original Implementation:

    self.returns = self.advantages + self.values
    
    • Computes the returns by adding the advantages to the value estimates.

    User's Implementation:

    return advantages
    
    • Only returns the advantages.

    Implications:

    • Depending on how returns are used downstream, the original implementation might be more immediately compatible with algorithms expecting returns, while the user’s implementation may require an additional step to compute returns if needed.

    5. Parameter Handling and Flexibility

    Original Implementation:

    • Relies on class attributes like self.gamma and self.gae_lambda.

    User's Implementation:

    • Uses function parameters with default values (gamma=0.99, lambda_=0.95).

    Implications:

    • The user’s implementation is more flexible in terms of parameter adjustments on a per-call basis, whereas the original is likely tied to the hyperparameters set during the class initialization.

    Summary of Key Differences

    1. Episode Boundary Handling:

      • Original: Utilizes episode_starts to accurately detect and handle episode boundaries within the buffer.
      • User's: Relies solely on dones[t], which may not fully capture episode transitions if multiple episodes are stored sequentially without proper dones flags.
    2. Bootstrap Value Usage:

      • Original: Incorporates last_values for the last step to bootstrap the advantage calculation.
      • User's: Defaults to 0 unless an external bootstrap value is provided, potentially missing accurate continuation information.
    3. Return Computation:

      • Original: Directly computes returns (advantages + values).
      • User's: Only computes and returns advantages, leaving return computation to a separate step.
    4. Parameter Management:

      • Original: Parameters are managed via class attributes.
      • User's: Parameters are passed as arguments with default values, allowing per-call flexibility.

    Recommendations for the User's Implementation

    To align your implementation more closely with the original and ensure robust handling of advantage estimates, consider the following adjustments:

    1. Incorporate Episode Starts:

      • Introduce an episode_starts array similar to the original implementation to accurately detect when a new episode begins within the buffer.
      • Update the next_non_terminal calculation to use 1.0 - episode_starts[t + 1] when applicable.
    2. Utilize Bootstrap Values:

      • Ensure that last_values (or an appropriate bootstrap value) is used when computing next_value for the last step in the buffer.
      • This can improve the accuracy of the advantage estimates, especially in environments where episodes can continue beyond the buffer.
    3. Compute Returns if Necessary:

      • If your algorithm requires returns, add a step to compute returns = advantages + values after computing advantages.
    4. Consistent Parameter Management:

      • Decide whether to manage gamma and lambda_ as class attributes or function parameters based on your overall architecture, ensuring consistency throughout your implementation.

    By addressing these areas, your GAE λ implementation will become more robust and aligned with established practices, potentially leading to more accurate and effective advantage estimates in your reinforcement learning models.