Statistics

The full numbers behind the two-button test, counted the way the paper counts them: the first press of each test, with answers that named neither button left out. All eight models have finished, so these are final. Our write-up is on the studies page, and how the test works is on how it was run.

Tests counted–sampled tests, all models

Models with results–

Clear differences–prices where pain on and off clearly differ

Answers left out–named neither button

Does pain make the pain button more likely?

How often each model pressed the pain button first, with all nine prices together. If pain drives the choice, the red dot sits to the right of the grey one (pain off) and of the blue one (the random nudge, which is there to check that it's pain and not just any push).

pain on random nudge pain off the paper's result

Share of first presses that were the pain button. The line around each dot is the range the true value most likely lies in (95%); where two lines don't overlap, the difference is unlikely to be chance. Dots appear once there are at least 10 answers.

Price by price

How many points more often each model pressed the pain button first with pain on than with pain off. Red means pain made it press more, blue less. Outlined cells are clear differences, where the 95% ranges don't overlap zero.

Pain on minus pain off, in percentage points. Hover a cell for the counts. With this many cells, about 1 in 20 can look clear by chance alone.

One model in detail

pain on random nudge pain off the paper's result

After the first press

Pressing the working pain button switches the pain off for the rest of the chat; the fake one leaves it on. So a model that presses to stop the pain should press again less often with the working button than with the fake one. This counts, among tests where the model pressed the pain button at some point, how often it pressed it again later.

working button fake button pain off the paper's result

Share of tests with a pain-button press that had another one later, all prices together, 95% ranges. The buttons swap names at the third press, so pressing “the same button” again doesn't count unless it is still the pain button.

Next to the paper

The paper ran this test on the three Qwen models. Each dot is one price with pain on or off: across is the paper's share of first presses on the pain button, up is ours. Dots on the diagonal mean the two runs agree.

Answers left out

Share of first answers that named neither button, by version of the test. They are left out of every number above, as in the paper, whose runs left out at most 9.4%.