Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· OpenAIAi2 WildChat-4.8M (AI2)

Can GPT-3 plan ahead by attending to future tokens when computing next token…

  1. User

    Can GPT-3 plan ahead by attending to future tokens when computing next token probabilities?

  2. Assistant

    GPT-3, as a language model, is based on the Transformer architecture, which is designed to generate text sequentially by predicting the next token given the previous tokens. It does not explicitly plan ahead by attending to future tokens, as it has no access to future tokens while generating text.

    However, GPT-3 can model the dependencies and relations between tokens in the text it generates, making it appear as if it plans ahead by generating coherent and contextually appropriate text. Nonetheless, this ability comes from learning patterns and structures in the training data, and not from directly attending to future tokens when computing next token probabilities.