The finding
Lightman and colleagues compare training with feedback on intermediate reasoning steps against feedback on final outcomes. In their MATH experiments, process supervision performs better. They release a large dataset of human-labeled reasoning steps. [1]
What it means for pingpong
Evaluate a pingpong review by what it changes. Check whether it fixes an invalid inference, removes a needed qualification or leaves an error in place. The sequence of drafts makes those changes easier to inspect.
The limit
Process supervision is a training method. pingpong's inference-time review is not a process-supervised reward model. A visible response history also does not provide direct access to a model's internal computation.
Source
Hunter Lightman et al. (2023)