Skip to content
The Chat Archive
vicuna-13b Arena Human Preference 55K

Explain this: Large pretrained Transformer language models have been shown to exhibit…

  1. User

    Explain this: Large pretrained Transformer language models have been shown to exhibit zeroshot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments. We therefore consider the adaptation of pretrained models across architectures and objectives. We find that pretrained non-causal decoder models can be adapted into performant generative

  2. Assistant

    This passage is discussing the ability of large pre-trained Transformer language models to perform a wide range of tasks that they were not explicitly trained on, a phenomenon known as "zeroshot generalization." The authors of the work present a large-scale evaluation of different modeling choices and their impact on zeroshot generalization, specifically focusing on text-to-text models. They experiment with three different model architectures (causal/non-causal decoder-only and encoder-decoder) and two different pretraining objectives (autoregressive and masked language modeling), and evaluate the models with and without multitask prompted finetuning. They train models with over 5 billion parameters and evaluate the results on a large scale. The authors find that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zeroshot generalization, but that models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best. The authors conclude by discussing the adaptation of pretrained models across different architectures and objectives, and find that pretrained non-causal decoder models can be adapted into performant generative models.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt