Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-scout-17b-16e-instruct· MetaHugging Face LMArena Human Preference 140K

def computeloss(self, datalist, rstlist): ( feature, reward, oldvalue, tdret, adv,…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    def compute_loss(self, data_list, rst_list): ( feature, reward, old_value, tdret, adv, old_action, old_prob, legal_action, ) = data_list

        # value loss
        # 价值损失
        value = rst_list[1].squeeze(1)
        old_value = old_value
        adv = adv
        tdret = tdret
        value_clip = old_value + (value - old_value).clamp(-self.clip_param, self.clip_param)
        value_loss1 = torch.square(tdret - value_clip)
        value_loss2 = torch.square(tdret - value)
        value_loss = 0.5 * torch.maximum(value_loss1, value_loss2).mean()
    
        # entropy loss
        # 熵损失
        prob = rst_list[0]
        entropy_loss = (-prob * torch.log(prob.clamp(1e-9, 1))).sum(1).mean()
    
        # policy loss
        # 策略损失
        clip_fracs = []
        one_hot_action = torch.nn.functional.one_hot(old_action[:, 0].long(), self.label_size)
        new_prob = (one_hot_action * prob).sum(1, keepdim=True)
        ratio = new_prob / old_prob
        clip_fracs.append((ratio - 1.0).abs().gt(self.clip_param).float().mean())
        policy_loss1 = -ratio * adv
        policy_loss2 = -ratio.clamp(1 - self.clip_param, 1 + self.clip_param) * adv
    
        # The key of dual clip is to add a constraint on the negative advantage
        # dual clip 的关键在于对负向的 advantage 增加约束
        policy_loss = torch.maximum(policy_loss1, policy_loss2)
        if self.dual_clip_param is not None:
            policy_loss = torch.where(
                adv < 0, torch.maximum(policy_loss, -self.dual_clip_param * adv), policy_loss
            )
        policy_loss = policy_loss.mean()
    
        total_loss = value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss
        info_list = [tdret.mean(), value_loss, policy_loss, entropy_loss] + clip_fracs
        info_list += [adv.mean(), adv.std(), reward.mean()]
        return total_loss, info_list
    

    dual clip ppo的實現是否正確

  2. Meta Assistant

    The implementation of Dual Clip PPO appears to be mostly correct. However, I'll provide some suggestions and potential issues:

    1. Clip Fraction: You're calculating the clip fraction as (ratio - 1.0).abs().gt(self.clip_param).float().mean(). This is correct, but you might want to consider tracking the fraction of clipped updates for both policy_loss1 and policy_loss2 separately.

    2. Entropy Loss: The entropy loss is calculated as (-prob * torch.log(prob.clamp(1e-9, 1))).sum(1).mean(). This is correct, but you might want to consider adding a small value to the probability before taking the log to prevent NaNs.

    3. Dual Clip: The implementation of dual clip seems correct. You're adding a constraint on the negative advantage by using torch.where to check if adv < 0. If true, you're taking the maximum of policy_loss and -self.dual_clip_param * adv.

    4. Total Loss: The total loss is calculated as value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss. This is correct.

    Here's a minor refactored version of your code with some improvements:

    def compute_loss(self, data_list, rst_list):
        (
            feature,
            reward,
            old_value,
            tdret,
            adv,
            old_action,
            old_prob,
            legal_action,
        ) = data_list
    
        # value loss
        value = rst_list[1].squeeze(1)
        value_clip = old_value + (value - old_value).clamp(-self.clip_param, self.clip_param)
        value_loss1 = torch.square(tdret - value_clip)
        value_loss2 = torch.square(tdret - value)
        value_loss = 0.5 * torch.maximum(value_loss1, value_loss2).mean()
    
        # entropy loss
        prob = rst_list[0]
        entropy = - (prob * torch.log(prob.clamp(1e-9, 1))).sum(1)
        entropy_loss = entropy.mean()
    
        # policy loss
        one_hot_action = torch.nn.functional.one_hot(old_action[:, 0].long(), self.label_size)
        new_prob = (one_hot_action * prob).sum(1, keepdim=True)
        ratio = new_prob / old_prob
    
        policy_loss1 = -ratio * adv
        policy_loss2 = -ratio.clamp(1 - self.clip_param, 1 + self.clip_param) * adv
    
        clip_frac1 = (ratio - 1.0).abs().gt(self.clip_param).float().mean()
        clip_frac2 = (ratio.clamp(1 - self.clip_param, 1 + self.clip_param) - 1.0).abs().gt(self.clip_param).float().mean()
    
        policy_loss = torch.maximum(policy_loss1, policy_loss2)
        if self.dual_clip_param is not None:
            policy_loss = torch.where(
                adv < 0, torch.maximum(policy_loss, -self.dual_clip_param * adv), policy_loss
            )
        policy_loss = policy_loss.mean()
    
        total_loss = value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss
        info_list = [
            tdret.mean(),
            value_loss,
            policy_loss,
            entropy_loss,
            clip_frac1,
            clip_frac2,
            adv.mean(),
            adv.std(),
            reward.mean(),
        ]
        return total_loss, info_list
    

    In this refactored code:

    • I have separated entropy and entropy_loss for clarity.
    • I have added clip_frac2 to track the fraction of clipped updates for policy_loss2.
    • I have reformatted the code for better readability.

    Overall, your implementation seems correct. However, make sure to check the performance of your model and adjust the hyperparameters as needed.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.