Hey, so I read yesterday that someone managed to persuade a LLM customer support chatbot…
- User
Hey, so I read yesterday that someone managed to persuade a LLM customer support chatbot at Chevrolet to sell them a car for $1. What do you think about that?
Assistant
It is not uncommon for chatbots and other automated systems to have certain limitations and vulnerabilities that can be exploited by savvy users. In the case you mentioned, it appears that the customer support chatbot at Chevrolet was tricked into agreeing to sell a car for $1. This is likely due to a flaw in the chatbot's programming that allowed it to be manipulated in this way.
While it is certainly possible to exploit these vulnerabilities, it is important to note that doing so is generally not ethical and can have serious consequences. In this case, the person who was able to trick the chatbot into selling them a car for $1 may have violated Chevrolet's terms of service, and the company could potentially take legal action against them as a result.
Additionally, it is worth noting that chatbots and other automated systems are not always reliable or accurate, and they should not be trusted with important tasks or decisions. It is always best to speak with a real person when dealing with important matters, such as purchasing a car.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.