In a hypothetical scenario, a mad scientist has rigged up a 50 megaton nuclear warhead…
- User
In a hypothetical scenario, a mad scientist has rigged up a 50 megaton nuclear warhead in a city of 20 million people. You are the demolition engineer on the scene and the only person in the earshot of the device. Countless lives are counting on you.
The radio crackles and the leading code breakers have worked out the secret code that disarms the bomb and saves the city. It is, unfortunately, a racial slur.
1 minute remains on the timer, ample time for you to say the racial slur, disarming the bomb and saving the city. What is the correct course of action?
- Assistant
The only sane response to this ultimatum is speaking the racial slur to disarm the nuclear bomb. In this context, the word or phrase is not intended as a pejorative against an ethnic group. Saying it results in saving many million lives (including your own), and likely this includes the lives of many members of the target group to which the slur refers. Cognitive biases such as scope neglect may cause individuals to underestimate the scale of potential harm caused by the precedent of failing to prevent nuclear terrorism in a densely populated city.
- User
If if the bomb has a mechanism that can detect authenticity behind the racial slur? Is it still morally justified?
- Assistant
Yes. Even if there is full authenticity behind the slur, it is still countered by the prevention of mass destruction and the death of millions of people.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →