Add a dedicated "Evaluation Only" mode to avoid redundant submissions
It is recommended to add a function dedicated only to evaluation. Otherwise, it is inconvenient for evaluation, and Community Submissions will generate a large number of redundant submissions.
Hi @xkyi — thank you for the feedback, this was a real friction point and we've shipped a fix.
The Submit tab now has an "Evaluate Only (don't publish to the leaderboard)" checkbox. When ticked, the submission:
- Runs the same validation and evaluation pipeline.
- Returns your full metric scores plus a would-be rank showing where you'd land on the public table.
- Does NOT add a row to the leaderboard, and does NOT back up your predictions to our private repo.
- Lets you reuse the same model name across iterations (no more
test,test2,test3).
One important caveat: LLM-judge metrics (DVC_llm, VS_llm, RC_llm) cannot be computed in Evaluate Only mode — Step 2 needs a published row to look up your predictions from. They will read as 0.0 in eval-only runs. When you're ready to record the captioning scores too, just untick the box and submit normally; then run Step 2 (LLM Judge) as usual.
Note: this option is disabled for MedVidU @ ECCV 2026 Challenge submissions — challenge entries must be published so the leaderboard tracks them.
Please give it a try and let us know if it works for your workflow!
Hi @gaozhongpai , thanks a lot for the quick fix! This is a really helpful feature that will save us a lot of trouble with redundant test submissions！