write me an abstract for the following intro: Human feedback is considered a valuable…
- User
write me an abstract for the following intro: Human feedback is considered a valuable resource for model development. The nowadays standard model training process includes a pertraining phase \citep{radford2019language}, where the model is trained on a massive amount of general text, followed by an alignment phase, where the model is fine-tuned or trained with reinforcement learning or the like (usually both) on human preference data \citep{bai2022training}. The more data at hand, the better the model \citep{kaplan2020scaling, roberts2023scaling}.However, collecting this data usually requires an expensive human effort, limiting the scalability. We suggest leveraging implicit cues in the human-model interaction, to extract free human feedback.Naturally occurring feedback can be in the form of free text, with the user directly referring to the quality of the model's response (e.g. 'thank you!', or 'that's wrong'), or more implicit, for example, the user moves to ask their next question, usually only after they are satisfied with the previous answer. As the second feedback type might be unclear and ambitious sometimes (maybe the user gave up and therefore moved to the new question?), we mainly focus on recognizing explicit feedback, and leaving the implicit feedback for future work. Unlike model as a judge methods \citep{liu-etal-2023-g, zheng2023judging}, naturally occurring feedback is anchored in the human response, and therefore is less prone to “hallucinations” \citep{lewis2021retrievalaugmented} and biases \citep{saito2023verbosity}, and easier to explain and verify. Another advantage of this approach is thatthis form of feedback is potentially closer to the feedback given by two human interlocutors \citep{bassiri2011interactional, 5a61a38b-79fc-30d4-a76e-8a66a7707021}, possibly containing relevant information for better alignment. We manually annotate and show that naturally occurring feedback indeed exists in conversation data (\S\ref{sec:manual_annotations}). Furthermore, we find that naturally occurring feedback is more common in up-to-date data, possibly due to users raising their expectations and being able to conduct a more 'human-like' conversation with the model (\S\ref{sec:up-to-date-feedback}). Following this, we emphasize further the importance of ever-growing data resources over static datasets. Models keep improving and therefore the data used to align them should evolve too. We introduce a method to automatically extract the naturally occurring feedback from human-model interactions (\S\ref{sec:auto_extraction}). We use our method to obtain over feedback samples from 1M non-annotated human-model conversations. We release it as a dataset (\S\ref{sec:natural_feedback_dataset}).
We use the extracted data to train a model to better align to human preferences, and show that it outperforms the baseline model by . - Assistant
This paper proposes a novel approach to extracting human feedback for model alignment in natural language processing, by leveraging implicit cues in human-model interactions. Unlike traditional methods that rely on explicit human preference data, this approach extracts free and naturally occurring feedback, which is anchored in human responses and less prone to biases and hallucinations. The authors show that such feedback exists in conversation data and is more common in up-to-date data, emphasizing the importance of evolving data resources. They introduce a method to automatically extract the naturally occurring feedback and release it as a dataset. Experimental results demonstrate that using the extracted data to train a model leads to improved alignment with human preferences, outperforming the baseline model. Overall, this paper presents a promising direction for scalable and cost-effective model development in natural language processing.