Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

In this below code the statement "logchosenprobs = tf.math.log(chosenprobs)" was…

  1. User

    In this below code the statement "log_chosen_probs = tf.math.log(chosen_probs)" was assigned but never used with in the function "def learn(self, state, action_probs, reward, next_state, done)" in 'class PPOAgent', the said statement was calculated for the entropy calculation, but it was not included into the "entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)". how to include the statement of calculation "log_chosen_probs" with in the logic appropriately fit in to the present logic.

    class PPOAgent: def init(self, bounds_low, bounds_high, alpha=0.0003, gamma=0.99): self.bounds_low = bounds_low self.bounds_high = bounds_high self.delta = (bounds_high - bounds_low) / 100.0 # Smallest unit to update the parameter self.gamma = gamma self.num_params = len(bounds_low) self.actor_critic = ActorCritic(self.num_params) self.optimizer = Adam(learning_rate=alpha)

    def choose_action(self, input_observation):
        state = tf.convert_to_tensor([input_observation], dtype=tf.float32)
        logits, _ = self.actor_critic(state)
        action_probs = tf.nn.softmax(logits, axis=-1)
        return action_probs.numpy()[0]
    
    def learn(self, state, action_probs, reward, next_state, done):
        state = tf.convert_to_tensor([state], dtype=tf.float32)
        next_state = tf.convert_to_tensor([next_state], dtype=tf.float32)
        reward = tf.convert_to_tensor(reward, dtype=tf.float32)
        
        with tf.GradientTape() as tape:
            current_logits, current_value = self.actor_critic(state)
            current_value = tf.squeeze(current_value)
            _, next_value = self.actor_critic(next_state)
            next_value = tf.squeeze(next_value)
            
            advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
            value_loss = advantage ** 2
            
            # Get probs and actions_taken for each timestamp
            probs = tf.nn.softmax(current_logits, -1)
            actions_taken = tf.cast(tf.argmax(action_probs, axis=-1), dtype=tf.int32)
    
            # Create a vector of cumulative dimensions for indices
            gather_indices = tf.range(tf.shape(probs)[0]) * tf.shape(probs)[1] + actions_taken
    
            # Gather probabilities for selected actions
            chosen_probs = tf.gather(tf.reshape(probs, [-1]), gather_indices)
            
            # Log probs for the entropy calculation
            log_chosen_probs = tf.math.log(chosen_probs)
    
            # Calculate entropy
            entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
    
            # Calculate log probability of the chosen action
            log_probs = tf.math.log(chosen_probs)
            
            # Calculate advantage and losses
            advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
            policy_loss = -log_probs * tf.stop_gradient(advantage)
            value_loss = advantage ** 2
            total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
            
        gradients = tape.gradient(total_loss, self.actor_critic.trainable_variables)
        self.optimizer.apply_gradients(zip(gradients, self.actor_critic.trainable_variables))
    
  2. Assistant

    It seems like the calculation of log_chosen_probs in the learn method is indeed redundant, since later the code calculates log_probs using the same chosen_probs variable. The likely intent with calculating log_chosen_probs would have been to use it in the entropy calculation or somewhere else in the loss calculation, but it was not used.

    To correctly include entropy into the loss and use the calculated log_chosen_probs variable, you can replace the entropy line with the following:

    Instead of:

    # Calculate entropy
    entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
    

    Include log_chosen_probs in the calculation:

    # Calculate entropy using log_chosen_probs for the actions taken
    entropy = -tf.reduce_sum(probs * log_chosen_probs, axis=-1)
    

    Additionally, make sure that when you calculate the entropy, you're using a method that correctly handles the specific actions taken. However, there's a problem with this approach: log_chosen_probs only has the log probabilities for the actions that were taken, not the full distribution. Therefore, we cannot use it directly in this form to calculate the entropy.

    Entropy is a measure of the uncertainty or unpredictability in the predictions of the model and usually involves the entire probability distribution. Here's how to correctly update the entropy calculation:

    # Calculate entropy properly, using the full distribution
    entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
    

    This line is actually calculating the entropy correctly by considering the full distribution of actions (probs) and not just the ones that were selected (log_chosen_probs). Therefore, the log_chosen_probs variable is indeed not necessary and can be removed if it's not used anywhere else in the code. If you want to correctly use log_chosen_probs to penalize certainty in actions being taken, it would typically be used as part of the policy loss rather than the entropy term which should remain as a calculation over the full distribution.

    So, your learn function is correct without the need to include log_chosen_probs directly in the entropy term. However, if for some reason you really need to use log_chosen_probs, you could create a separate term to penalize low entropy in selected actions but that would be a departure from the standard way to include entropy in PPO.