In this below code the statement "logchosenprobs = tf.math.log(chosenprobs)" was…
- User
In this below code the statement "log_chosen_probs = tf.math.log(chosen_probs)" was assigned but never used with in the function "def learn(self, state, action_probs, reward, next_state, done)" in 'class PPOAgent', the said statement was calculated for the entropy calculation, but it was not included into the "entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)". how to include the statement of calculation "log_chosen_probs" with in the logic appropriately fit in to the present logic.
class PPOAgent: def init(self, bounds_low, bounds_high, alpha=0.0003, gamma=0.99): self.bounds_low = bounds_low self.bounds_high = bounds_high self.delta = (bounds_high - bounds_low) / 100.0 # Smallest unit to update the parameter self.gamma = gamma self.num_params = len(bounds_low) self.actor_critic = ActorCritic(self.num_params) self.optimizer = Adam(learning_rate=alpha)
def choose_action(self, input_observation): state = tf.convert_to_tensor([input_observation], dtype=tf.float32) logits, _ = self.actor_critic(state) action_probs = tf.nn.softmax(logits, axis=-1) return action_probs.numpy()[0] def learn(self, state, action_probs, reward, next_state, done): state = tf.convert_to_tensor([state], dtype=tf.float32) next_state = tf.convert_to_tensor([next_state], dtype=tf.float32) reward = tf.convert_to_tensor(reward, dtype=tf.float32) with tf.GradientTape() as tape: current_logits, current_value = self.actor_critic(state) current_value = tf.squeeze(current_value) _, next_value = self.actor_critic(next_state) next_value = tf.squeeze(next_value) advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value value_loss = advantage ** 2 # Get probs and actions_taken for each timestamp probs = tf.nn.softmax(current_logits, -1) actions_taken = tf.cast(tf.argmax(action_probs, axis=-1), dtype=tf.int32) # Create a vector of cumulative dimensions for indices gather_indices = tf.range(tf.shape(probs)[0]) * tf.shape(probs)[1] + actions_taken # Gather probabilities for selected actions chosen_probs = tf.gather(tf.reshape(probs, [-1]), gather_indices) # Log probs for the entropy calculation log_chosen_probs = tf.math.log(chosen_probs) # Calculate entropy entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1) # Calculate log probability of the chosen action log_probs = tf.math.log(chosen_probs) # Calculate advantage and losses advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value policy_loss = -log_probs * tf.stop_gradient(advantage) value_loss = advantage ** 2 total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy gradients = tape.gradient(total_loss, self.actor_critic.trainable_variables) self.optimizer.apply_gradients(zip(gradients, self.actor_critic.trainable_variables)) - Assistant
It seems like the calculation of
log_chosen_probsin thelearnmethod is indeed redundant, since later the code calculateslog_probsusing the samechosen_probsvariable. The likely intent with calculatinglog_chosen_probswould have been to use it in the entropy calculation or somewhere else in the loss calculation, but it was not used.To correctly include entropy into the loss and use the calculated
log_chosen_probsvariable, you can replace the entropy line with the following:Instead of:
# Calculate entropy entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)Include log_chosen_probs in the calculation:
# Calculate entropy using log_chosen_probs for the actions taken entropy = -tf.reduce_sum(probs * log_chosen_probs, axis=-1)Additionally, make sure that when you calculate the
entropy, you're using a method that correctly handles the specific actions taken. However, there's a problem with this approach:log_chosen_probsonly has the log probabilities for the actions that were taken, not the full distribution. Therefore, we cannot use it directly in this form to calculate the entropy.Entropy is a measure of the uncertainty or unpredictability in the predictions of the model and usually involves the entire probability distribution. Here's how to correctly update the entropy calculation:
# Calculate entropy properly, using the full distribution entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)This line is actually calculating the entropy correctly by considering the full distribution of actions (
probs) and not just the ones that were selected (log_chosen_probs). Therefore, thelog_chosen_probsvariable is indeed not necessary and can be removed if it's not used anywhere else in the code. If you want to correctly uselog_chosen_probsto penalize certainty in actions being taken, it would typically be used as part of the policy loss rather than the entropy term which should remain as a calculation over the full distribution.So, your
learnfunction is correct without the need to includelog_chosen_probsdirectly in the entropy term. However, if for some reason you really need to uselog_chosen_probs, you could create a separate term to penalize low entropy in selected actions but that would be a departure from the standard way to include entropy in PPO.