A limit of two submissions a day sounds like platform housekeeping. It is actually part of the experiment.
A team changes a model, submits it, sees the score, keeps the change or throws it away, then tries again. After enough rounds the public leaderboard has become a weak training signal. No label needs to leak for the team to fit quirks of the public test cases.
This is not automatically cheating
If a team falls from first place on the public board to tenth on the private board, there are several possible explanations. They may have overused feedback. The public set may have been too small. The two splits may contain different hospitals, scanners, disease mixes, or planning styles. The metric itself may be unstable.
The platform cannot fix those problems after the contest closes. They have to be designed around beforehand.
The rules I would want
| Rule | Reason |
|---|---|
| Daily submission quota | Reduces adaptive queries as well as server load. |
| Separate public and final sets | Keeps part of the evaluation untouched until the end. |
| Final model chosen before reveal | Stops selection after private scores become visible. |
| Container rerun | Checks whether the submitted result can actually be reproduced. |
| Confidence intervals | Shows whether tiny rank differences mean much. |
| Site and device subgroups | Finds a model that wins overall by failing somewhere important. |
Medical leaderboards need more than one score
AUROC, Dice, or a treatment-plan score can still be the primary ranking metric. But I would also want uncertainty, important failure cases, hospital or scanner subgroups, and the size of the public-to-private drop. Otherwise the leaderboard looks precise while the conclusion is not.
The next useful Orcaton tool may be a small simulator: choose the public-set size, private-set size, submission count, and metric noise, then watch how often the apparent winner changes. That would let an organizer see the problem before using real data.
← All field notes