make the following a single tweet (from a thread of tweets). Note character limit We…
- User
make the following a single tweet (from a thread of tweets). Note character limit
We hypothesize that RL (and the KL regularization) indeed prevent from diverging from the base LLMe. But the responses we suggest are both very likely, given the prompt that explicitly asks for them. We just teach the model to prefer one of the two most probable answers. By itself, it understands to update those in other settings as well.
- Assistant
RL (with KL regularization) keeps the model close to the base LLM. But our prompts ask for likely responses—just nudging it to prefer one over another. The model generalizes this preference elsewhere on its own.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.