The finding
Self-consistency samples multiple reasoning paths and selects the answer that is most consistent across them. The authors report improvements on arithmetic and commonsense reasoning benchmarks over single-path decoding. [1]
What it means for Pingpong
Pingpong aims to revise an answer through critique, which is different from selecting the most common answer. To demonstrate that the review framework adds value, it should be evaluated against self-consistency as well as a single response. Otherwise extra samples could explain a gain attributed to the chain.
The limit
Voting is easier to define for questions with a discrete answer than for an open-ended strategy or proposal. That limits direct transfer, but it does not remove the need for a competitive baseline.
Source
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Xuezhi Wang et al. (2022)