The finding
Turpin and colleagues show that biasing features can change model answers without being acknowledged in the explanations. Models sometimes produce convincing rationales for answers influenced by irrelevant cues. [1]
What it means for Pingpong
Pingpong makes successive responses available for inspection. That provides a record of what each model said and what changed, not a guaranteed account of why it happened internally. The review should be assessed by the quality of its claims, objections, and supporting evidence.
The limit
More visible reasoning is not automatically more trustworthy. This study used particular earlier-generation models and tasks; current systems require fresh tests rather than assumptions of either immunity or identical failure rates.
Source
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman (2023)