Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

In this below code the statement "logchosenprobs = tf.math.log(chosenprobs)" was…

  1. User

    In this below code the statement "log_chosen_probs = tf.math.log(chosen_probs)" was assigned but never used with in the function "def learn(self, state, action_probs, reward, next_state, done)" in 'class PPOAgent'.

    class PPOAgent: def init(self, bounds_low, bounds_high, alpha=0.0003, gamma=0.99): self.bounds_low = bounds_low self.bounds_high = bounds_high self.delta = (bounds_high - bounds_low) / 100.0 # Smallest unit to update the parameter self.gamma = gamma self.num_params = len(bounds_low) self.actor_critic = ActorCritic(self.num_params) self.optimizer = Adam(learning_rate=alpha)

    def choose_action(self, input_observation):
        state = tf.convert_to_tensor([input_observation], dtype=tf.float32)
        logits, _ = self.actor_critic(state)
        action_probs = tf.nn.softmax(logits, axis=-1)
        return action_probs.numpy()[0]
    
    def learn(self, state, action_probs, reward, next_state, done):
        state = tf.convert_to_tensor([state], dtype=tf.float32)
        next_state = tf.convert_to_tensor([next_state], dtype=tf.float32)
        reward = tf.convert_to_tensor(reward, dtype=tf.float32)
        
        with tf.GradientTape() as tape:
            current_logits, current_value = self.actor_critic(state)
            current_value = tf.squeeze(current_value)
            _, next_value = self.actor_critic(next_state)
            next_value = tf.squeeze(next_value)
            
            advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
            value_loss = advantage ** 2
            
            # Get probs and actions_taken for each timestamp
            probs = tf.nn.softmax(current_logits, -1)
            actions_taken = tf.cast(tf.argmax(action_probs, axis=-1), dtype=tf.int32)
    
            # Create a vector of cumulative dimensions for indices
            gather_indices = tf.range(tf.shape(probs)[0]) * tf.shape(probs)[1] + actions_taken
    
            # Gather probabilities for selected actions
            chosen_probs = tf.gather(tf.reshape(probs, [-1]), gather_indices)
            
            # Log probs for the entropy calculation
            log_chosen_probs = tf.math.log(chosen_probs)
    
            # Calculate entropy
            entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
    
            # Calculate log probability of the chosen action
            log_probs = tf.math.log(chosen_probs)
            
            # Calculate advantage and losses
            advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
            policy_loss = -log_probs * tf.stop_gradient(advantage)
            value_loss = advantage ** 2
            total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
            
        gradients = tape.gradient(total_loss, self.actor_critic.trainable_variables)
        self.optimizer.apply_gradients(zip(gradients, self.actor_critic.trainable_variables))
    
  2. Assistant

    You are correct that the statement log_chosen_probs = tf.math.log(chosen_probs) in the learn method is assigned but not used later in the method. This appears to be redundant or leftover from a previous version of the code where log_chosen_probs might have been used for some calculation.

    Looking at the rest of the code, we have:

    • First instance: log_chosen_probs is calculated but never used.
    • Second instance: log_probs is calculated with the same expression tf.math.log(chosen_probs) and is used in calculating the policy_loss.

    The second calculation is the relevant one in the context of this function, as it is used to compute the policy_loss, which is part of the total_loss. The entropy calculation also uses the logarithm of probs, but in the tf.reduce_sum it's done inline rather than reusing log_chosen_probs.

    The correct code should simply avoid the unused assignment and look something like this:

    # Calculate entropy
    entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
        
    # Calculate log probability of the chosen action
    log_probs = tf.math.log(chosen_probs)
                
    # Calculate advantage and losses
    advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
    policy_loss = -log_probs * tf.stop_gradient(advantage)
    value_loss = advantage ** 2
    total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
    

    In the code above, the redundant line log_chosen_probs = tf.math.log(chosen_probs) is removed. If there was a performance concern, the calculation tf.math.log(chosen_probs) could be stored in a variable and reused to avoid computing the log twice for entropy and log_probs, but since entropy uses a different set of probabilities due to the sum and epsilon offset (1e-9), caching the value wouldn't be beneficial in this case.

    Removing such an unused assignment would improve the cleanliness of the code without affecting its functionality.

  3. User

    but it was calculated for entropy calculation

  4. Assistant

    You're correct; while the log_chosen_probs variable is indeed not directly used after its assignment, the concept of computing the logarithm of chosen probabilities is used twice: once when it is assigned to log_chosen_probs and again as log_probs for calculating the policy_loss. Both are ultimately derived from the chosen_probs tensor, and the calculation is essentially the same for both.

    The entropy calculation also involves logarithms, but rather than using log_chosen_probs, it computes the log over all the probs (not just the chosen ones) to compute the entropy of the entire probability distribution.

    Here's a rephrased version of the relevant portion of the code:

    # Calculate entropy for the entire probability distribution
    entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
        
    # Calculate log probability of just the chosen action (for the policy loss)
    log_probs = tf.math.log(chosen_probs)
                
    # Calculate advantage and policy loss
    advantage = reward + self.gamma * next_value * (1 - int(done)) - current_value
    policy_loss = -log_probs * tf.stop_gradient(advantage)
    
    # Value loss remains unchanged
    value_loss = advantage ** 2
    
    # Total loss consists of policy loss, value loss, and entropy bonus
    total_loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
    

    The key point is that there is no need for two separate assignments (log_chosen_probs and log_probs) since they both represent the logarithm of the chosen probabilities. Therefore, it is sufficient to directly calculate log_probs for use in the policy loss, and independently perform the calculation required for the entropy over the entire distribution using probs.

    By eliminating the unused variable log_chosen_probs, the code is simplified without changing its functionality:

    # Calculate entropy
    entropy = -tf.reduce_sum(probs * tf.math.log(probs + 1e-9), axis=-1)
        
    # Calculate log probability of the chosen action for the policy loss
    log_probs = tf.math.log(chosen_probs)
                
    # Other calculations remain unchanged...
    

    This adjustment improves code readability and efficiency.