What changese between this implementation of gae lambda: lastvalues =…
- User
What changese between this implementation of gae lambda:
last_values = last_values.clone().cpu().numpy().flatten() last_gae_lam = 0 for step in reversed(range(self.buffer_size)): if step == self.buffer_size - 1: next_non_terminal = 1.0 - dones next_values = last_values else: next_non_terminal = 1.0 - self.episode_starts[step + 1] next_values = self.values[step + 1] delta = self.rewards[step] + self.gamma * next_values * next_non_terminal - self.values[step] last_gae_lam = delta + self.gamma * self.gae_lambda * next_non_terminal * last_gae_lam self.advantages[step] = last_gae_lam # TD(lambda) estimator, see Github PR #375 or "Telescoping in TD(lambda)" # in David Silver Lecture 4: https://www.youtube.com/watch?v=PnHCvfgC_ZA self.returns = self.advantages + self.valuesand mine:
def compute_advantages(self, rewards, dones, values, gamma=0.99, lambda_=0.95): advantages = np.zeros_like(rewards) last_adv = 0 for t in reversed(range(len(rewards))): if dones[t]: next_non_terminal = 0 next_value = 0 else: next_non_terminal = 1 if t + 1 < len(values): next_value = values[t + 1] else: next_value = 0 # Or use a bootstrap value if available delta = rewards[t] + gamma * next_value * next_non_terminal - values[t] advantages[t] = delta + gamma * lambda_ * next_non_terminal * last_adv last_adv = advantages[t] return advantages - Assistant
Both implementations aim to compute the Generalized Advantage Estimation (GAE) λ, which is used to estimate advantages in reinforcement learning algorithms like Proximal Policy Optimization (PPO). However, there are several key differences between the two implementations that can affect their behavior and the resulting advantage estimates. Here's a detailed comparison:
1. Handling Episode Boundaries
Original Implementation:
if step == self.buffer_size - 1: next_non_terminal = 1.0 - dones next_values = last_values else: next_non_terminal = 1.0 - self.episode_starts[step + 1] next_values = self.values[step + 1]self.episode_starts: This array indicates whether a new episode starts at a particular step. By using1.0 - self.episode_starts[step + 1], the code ensures that if a new episode starts atstep + 1, thenext_non_terminalflag is set to0. This is crucial when the buffer contains concatenated episodes, preventing the advantage from incorrectly propagating across episode boundaries.
User's Implementation:
if dones[t]: next_non_terminal = 0 next_value = 0 else: next_non_terminal = 1 if t + 1 < len(values): next_value = values[t + 1] else: next_value = 0 # Or use a bootstrap value if availabledones[t]: This flag only indicates whether the current step is the terminal step of an episode. It doesn't account for whether a new episode starts immediately after the current one unlessdonesare explicitly set for episode boundaries.
Implications:
- Original Implementation: More robust in handling buffers with multiple concatenated episodes because it explicitly checks for episode starts.
- User's Implementation: Might incorrectly propagate advantage estimates across episode boundaries if multiple episodes are stored sequentially in the buffer without proper
donesflags.
2. Bootstrapping with
last_valuesOriginal Implementation:
if step == self.buffer_size - 1: next_values = last_valueslast_values: This represents the value estimate at the end of the buffer, allowing the GAE calculation to bootstrap from the last state if it's not terminal.
User's Implementation:
else: next_value = 0 # Or use a bootstrap value if available- Bootstrap Value Handling: The user’s implementation sets
next_valueto0whent + 1exceeds the buffer length, unless an external bootstrap value is provided. This might not utilize the actuallast_valuefrom the environment, potentially leading to less accurate advantage estimates.
Implications:
- Original Implementation: Properly incorporates a bootstrap value (
last_values) for the last step in the buffer, enhancing the accuracy of the advantage estimates when the episode continues beyond the buffer. - User's Implementation: Defaults to
0unless a bootstrap value is manually provided, which might not capture the true continuation of the episode.
3. Advantage Calculation and Accumulation
Original Implementation:
delta = self.rewards[step] + self.gamma * next_values * next_non_terminal - self.values[step] last_gae_lam = delta + self.gamma * self.gae_lambda * next_non_terminal * last_gae_lam self.advantages[step] = last_gae_lamUser's Implementation:
delta = rewards[t] + gamma * next_value * next_non_terminal - values[t] advantages[t] = delta + gamma * lambda_ * next_non_terminal * last_adv last_adv = advantages[t]- Accumulation Variable:
- Original: Uses
last_gae_lamto accumulate the advantage. - User's: Uses
last_advfor accumulation.
- Original: Uses
Implications:
- Functionally, both accumulate the advantage estimates in a similar backward recursive manner. The primary difference lies in the handling of
next_non_terminalandnext_values.
4. Return Calculation
Original Implementation:
self.returns = self.advantages + self.values- Computes the returns by adding the advantages to the value estimates.
User's Implementation:
return advantages- Only returns the advantages.
Implications:
- Depending on how returns are used downstream, the original implementation might be more immediately compatible with algorithms expecting returns, while the user’s implementation may require an additional step to compute returns if needed.
5. Parameter Handling and Flexibility
Original Implementation:
- Relies on class attributes like
self.gammaandself.gae_lambda.
User's Implementation:
- Uses function parameters with default values (
gamma=0.99,lambda_=0.95).
Implications:
- The user’s implementation is more flexible in terms of parameter adjustments on a per-call basis, whereas the original is likely tied to the hyperparameters set during the class initialization.
Summary of Key Differences
-
Episode Boundary Handling:
- Original: Utilizes
episode_startsto accurately detect and handle episode boundaries within the buffer. - User's: Relies solely on
dones[t], which may not fully capture episode transitions if multiple episodes are stored sequentially without properdonesflags.
- Original: Utilizes
-
Bootstrap Value Usage:
- Original: Incorporates
last_valuesfor the last step to bootstrap the advantage calculation. - User's: Defaults to
0unless an external bootstrap value is provided, potentially missing accurate continuation information.
- Original: Incorporates
-
Return Computation:
- Original: Directly computes returns (
advantages + values). - User's: Only computes and returns advantages, leaving return computation to a separate step.
- Original: Directly computes returns (
-
Parameter Management:
- Original: Parameters are managed via class attributes.
- User's: Parameters are passed as arguments with default values, allowing per-call flexibility.
Recommendations for the User's Implementation
To align your implementation more closely with the original and ensure robust handling of advantage estimates, consider the following adjustments:
-
Incorporate Episode Starts:
- Introduce an
episode_startsarray similar to the original implementation to accurately detect when a new episode begins within the buffer. - Update the
next_non_terminalcalculation to use1.0 - episode_starts[t + 1]when applicable.
- Introduce an
-
Utilize Bootstrap Values:
- Ensure that
last_values(or an appropriate bootstrap value) is used when computingnext_valuefor the last step in the buffer. - This can improve the accuracy of the advantage estimates, especially in environments where episodes can continue beyond the buffer.
- Ensure that
-
Compute Returns if Necessary:
- If your algorithm requires returns, add a step to compute
returns = advantages + valuesafter computing advantages.
- If your algorithm requires returns, add a step to compute
-
Consistent Parameter Management:
- Decide whether to manage
gammaandlambda_as class attributes or function parameters based on your overall architecture, ensuring consistency throughout your implementation.
- Decide whether to manage
By addressing these areas, your GAE λ implementation will become more robust and aligned with established practices, potentially leading to more accurate and effective advantage estimates in your reinforcement learning models.