In this below code the statement "logchosenprobs = tf.math.log(chosenprobs)" was…
- User
In this below code the statement "log_chosen_probs = tf.math.log(chosen_probs)" was assigned but never used with in the function "def learn(self, state, action_probs, reward, next_state, done)" in 'class PPOAgent'.
class PPOAgent: def init(self, bounds_low, bounds_high, alpha=0.0003, gamma=0.99): self.bounds_low = bounds_low self.bounds_high = bounds_high self.delta = (bounds_high - bounds_low) / 100.0 # Smallest unit to update the parameter self.gamma = gamma self.num_params = len(bounds_low) self.actor_critic = ActorCritic(self.num_params) self.optimizer = Adam(learning_rate=alpha)
def choose_action(self, input_observation): state = tf.convert_to_tensor([input_observation], dtype=tf.float32) logits, _ = self.actor_critic(state) action_probs = tf.nn.softmax(logits, axis=-1) return action_probs.numpy()[0] def learn(self, state, action_probs, reward, next_state, done): state = tf.convert_to_tensor([state], dtype=tf.float32) next_state = tf.convert_to_tensor([next_state], dtype=tf.float32) reward = tf.convert_to_tensor(reward, dtype=tf.float32) with tf.GradientTape() as tape: current_logits, current_value = self.actor_critic(state) current_value = tf.squeeze(current_value) _, next_value = self.actor_critic(next_state) next_value = tf.squeeze(next_value) advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value value_loss = advantage ** 2 # Get probs and actions_taken for each timestamp probs = tf.nn.softmax(current_logits, -1) actions_taken = tf.cast(tf.argmax(action_probs, axis=-1), dtype=tf.int32) # Create a vector of cumulative dimensions for indices gather_indices = tf.range(tf.shape(probs)[0]) * tf.shape(probs)[1] + actions_taken # Gather probabilities for selected actions chosen_probs = tf.gather(tf.reshape(probs, [-1]), gather_indices) # Log probs for the entropy calculation log_chosen_probs = tf.math.log(chosen_probs) # Calculate entropy entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1) # Calculate log probability of the chosen action log_probs = tf.math.log(chosen_probs) # Calculate advantage and losses advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value policy_loss = -log_probs * tf.stop_gradient(advantage) value_loss = advantage ** 2 total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy gradients = tape.gradient(total_loss, self.actor_critic.trainable_variables) self.optimizer.apply_gradients(zip(gradients, self.actor_critic.trainable_variables)) - Assistant
You are correct that the statement
log_chosen_probs = tf.math.log(chosen_probs)in thelearnmethod is assigned but not used later in the method. This appears to be redundant or leftover from a previous version of the code wherelog_chosen_probsmight have been used for some calculation.Looking at the rest of the code, we have:
- First instance:
log_chosen_probsis calculated but never used. - Second instance:
log_probsis calculated with the same expressiontf.math.log(chosen_probs)and is used in calculating thepolicy_loss.
The second calculation is the relevant one in the context of this function, as it is used to compute the
policy_loss, which is part of thetotal_loss. Theentropycalculation also uses the logarithm ofprobs, but in thetf.reduce_sumit's done inline rather than reusinglog_chosen_probs.The correct code should simply avoid the unused assignment and look something like this:
# Calculate entropy entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1) # Calculate log probability of the chosen action log_probs = tf.math.log(chosen_probs) # Calculate advantage and losses advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value policy_loss = -log_probs * tf.stop_gradient(advantage) value_loss = advantage ** 2 total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropyIn the code above, the redundant line
log_chosen_probs = tf.math.log(chosen_probs)is removed. If there was a performance concern, the calculationtf.math.log(chosen_probs)could be stored in a variable and reused to avoid computing the log twice forentropyandlog_probs, but sinceentropyuses a different set of probabilities due to the sum and epsilon offset (1e-9), caching the value wouldn't be beneficial in this case.Removing such an unused assignment would improve the cleanliness of the code without affecting its functionality.
- First instance:
- User
but it was calculated for entropy calculation
- Assistant
You're correct; while the
log_chosen_probsvariable is indeed not directly used after its assignment, the concept of computing the logarithm of chosen probabilities is used twice: once when it is assigned tolog_chosen_probsand again aslog_probsfor calculating thepolicy_loss. Both are ultimately derived from thechosen_probstensor, and the calculation is essentially the same for both.The
entropycalculation also involves logarithms, but rather than usinglog_chosen_probs, it computes the log over all theprobs(not just the chosen ones) to compute the entropy of the entire probability distribution.Here's a rephrased version of the relevant portion of the code:
# Calculate entropy for the entire probability distribution entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1) # Calculate log probability of just the chosen action (for the policy loss) log_probs = tf.math.log(chosen_probs) # Calculate advantage and policy loss advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value policy_loss = -log_probs * tf.stop_gradient(advantage) # Value loss remains unchanged value_loss = advantage ** 2 # Total loss consists of policy loss, value loss, and entropy bonus total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropyThe key point is that there is no need for two separate assignments (
log_chosen_probsandlog_probs) since they both represent the logarithm of the chosen probabilities. Therefore, it is sufficient to directly calculatelog_probsfor use in the policy loss, and independently perform the calculation required for the entropy over the entire distribution usingprobs.By eliminating the unused variable
log_chosen_probs, the code is simplified without changing its functionality:# Calculate entropy entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1) # Calculate log probability of the chosen action for the policy loss log_probs = tf.math.log(chosen_probs) # Other calculations remain unchanged...This adjustment improves code readability and efficiency.