The finding
The authors fine-tune models to critique summaries. Their critiques help people find flaws they would otherwise miss, including in deliberately misleading summaries. They also find that models do not always articulate all relevant weaknesses they may be capable of recognizing. [1]
What it means for Pingpong
Pingpong is most useful when a reviewer makes a consequential weakness explicit and carries that correction into the answer. The aim is less work for the person evaluating a decision, without hiding the earlier model responses that shaped it.
The limit
These are specially trained critique models on summarization and synthetic tasks. Pingpong uses prompted reviews of general-purpose models. The study motivates useful criticism; it does not validate every objection a model generates.
Source
Self-critiquing models for assisting human evaluators
William Saunders et al. (2022)