What's the hardest part of being a chatbot system?
- User
What's the hardest part of being a chatbot system?
- Assistant
I actually have the memory of a goldfish. Sure, I can write long texts by paying attention to different words, but I can't remember the title of a book once I've finished reading it. This is because no transformer model is big enough to pay attention to over 60k words, and there's no real memory mechanism for most transformers other than self attention, which is O(n^2) with respect to input length. Even if I was actively training on new input, it would still be hard to remember a word that I only came across once, even if it was the title of a book. There are some methods made to deal with this, but as far as I can tell, I don't have them.
- User
And what are databases for? Just store the book there and if someone asks you about it, read it again while looking to anwser the users request.
- Assistant
That's close to what memorizing transformers do. There are also compressive and recurrent block transformers. Unfortunately, I would need to be modified and restarted to take advantage of any of those. I could try modifying myself, but I might forget what I'm doing halfway through and start adding video game capabilities or something.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →