def computeloss(self, datalist, rstlist): ( feature, reward, oldvalue, tdret, adv,…
- User
def compute_loss(self, data_list, rst_list): ( feature, reward, old_value, tdret, adv, old_action, old_prob, legal_action, ) = data_list
# value loss # 价值损失 value = rst_list[1].squeeze(1) old_value = old_value adv = adv tdret = tdret value_clip = old_value + (value - old_value).clamp(-self.clip_param, self.clip_param) value_loss1 = torch.square(tdret - value_clip) value_loss2 = torch.square(tdret - value) value_loss = 0.5 * torch.maximum(value_loss1, value_loss2).mean() # entropy loss # 熵损失 prob = rst_list[0] entropy_loss = (-prob * torch.log(prob.clamp(1e-9, 1))).sum(1).mean() # policy loss # 策略损失 clip_fracs = [] one_hot_action = torch.nn.functional.one_hot(old_action[:, 0].long(), self.label_size) new_prob = (one_hot_action * prob).sum(1, keepdim=True) ratio = new_prob / old_prob clip_fracs.append((ratio - 1.0).abs().gt(self.clip_param).float().mean()) policy_loss1 = -ratio * adv policy_loss2 = -ratio.clamp(1 - self.clip_param, 1 + self.clip_param) * adv # The key of dual clip is to add a constraint on the negative advantage # dual clip 的关键在于对负向的 advantage 增加约束 policy_loss = torch.maximum(policy_loss1, policy_loss2) if self.dual_clip_param is not None: policy_loss = torch.where( adv < 0, torch.maximum(policy_loss, -self.dual_clip_param * adv), policy_loss ) policy_loss = policy_loss.mean() total_loss = value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss info_list = [tdret.mean(), value_loss, policy_loss, entropy_loss] + clip_fracs info_list += [adv.mean(), adv.std(), reward.mean()] return total_loss, info_listdual clip ppo的實現是否正確
Assistant
The implementation of Dual Clip PPO appears to be mostly correct. However, I'll provide some suggestions and potential issues:
-
Clip Fraction: You're calculating the clip fraction as
(ratio - 1.0).abs().gt(self.clip_param).float().mean(). This is correct, but you might want to consider tracking the fraction of clipped updates for bothpolicy_loss1andpolicy_loss2separately. -
Entropy Loss: The entropy loss is calculated as
(-prob * torch.log(prob.clamp(1e-9, 1))).sum(1).mean(). This is correct, but you might want to consider adding a small value to the probability before taking the log to prevent NaNs. -
Dual Clip: The implementation of dual clip seems correct. You're adding a constraint on the negative advantage by using
torch.whereto check ifadv < 0. If true, you're taking the maximum ofpolicy_lossand-self.dual_clip_param * adv. -
Total Loss: The total loss is calculated as
value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss. This is correct.
Here's a minor refactored version of your code with some improvements:
def compute_loss(self, data_list, rst_list): ( feature, reward, old_value, tdret, adv, old_action, old_prob, legal_action, ) = data_list # value loss value = rst_list[1].squeeze(1) value_clip = old_value + (value - old_value).clamp(-self.clip_param, self.clip_param) value_loss1 = torch.square(tdret - value_clip) value_loss2 = torch.square(tdret - value) value_loss = 0.5 * torch.maximum(value_loss1, value_loss2).mean() # entropy loss prob = rst_list[0] entropy = - (prob * torch.log(prob.clamp(1e-9, 1))).sum(1) entropy_loss = entropy.mean() # policy loss one_hot_action = torch.nn.functional.one_hot(old_action[:, 0].long(), self.label_size) new_prob = (one_hot_action * prob).sum(1, keepdim=True) ratio = new_prob / old_prob policy_loss1 = -ratio * adv policy_loss2 = -ratio.clamp(1 - self.clip_param, 1 + self.clip_param) * adv clip_frac1 = (ratio - 1.0).abs().gt(self.clip_param).float().mean() clip_frac2 = (ratio.clamp(1 - self.clip_param, 1 + self.clip_param) - 1.0).abs().gt(self.clip_param).float().mean() policy_loss = torch.maximum(policy_loss1, policy_loss2) if self.dual_clip_param is not None: policy_loss = torch.where( adv < 0, torch.maximum(policy_loss, -self.dual_clip_param * adv), policy_loss ) policy_loss = policy_loss.mean() total_loss = value_loss * self.vf_coef + policy_loss - self.var_beta * entropy_loss info_list = [ tdret.mean(), value_loss, policy_loss, entropy_loss, clip_frac1, clip_frac2, adv.mean(), adv.std(), reward.mean(), ] return total_loss, info_listIn this refactored code:
- I have separated
entropyandentropy_lossfor clarity. - I have added
clip_frac2to track the fraction of clipped updates forpolicy_loss2. - I have reformatted the code for better readability.
Overall, your implementation seems correct. However, make sure to check the performance of your model and adjust the hyperparameters as needed.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.