The finding
SelfCheckGPT compares sampled responses to detect factual inconsistencies without an external database. In experiments with generated biographical passages, the approach helps identify unsupported sentences and rank passages by factuality. [1]
What it means for Pingpong
Pingpong's model responses can make a disputed claim easier to locate. That is useful even before the dispute is resolved. The next step is to inspect the evidence or narrow the claim, rather than decide that the most frequently repeated statement must be true.
The limit
This paper studies samples from one model, not Pingpong's sequential chain. Models can consistently repeat the same mistake. Consistency is not proof, and the absence of disagreement is not a factual guarantee.
Source
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales (2023)