I read a fun paper the other day.

The short story of this paper is that common benchmarking datasets contain bad labels. It was well known that in MNIST a 5 can sometimes look like an 8, so those mistakes might be forgiven. The paper however shows that the mistakes are a fair bit worse.

The paper tries to build algorithms to detect these mislabelled instances. The algorithm works by giving a percentage $\alpha$ of images the user is willing to manually re-evaluate and then the algorithm tries to find the appropriate candidates to check.

Their approach is straight-forward. A model is $g$ is trained and applied on a datapoint $x_i$. They then compare the model output $g(x_i)$ with the true label $y_i$. The model confidence can be used as a proxy to sort labels so that a user can check them.

It's a neat trick but given that the state-of-the-art is often 99+% for some of these datasets, it might be time for a few of these models to be re-run.