← back
20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna
Takeaway
Define state of the art against your application and users, using representative evidence rather than a generic leaderboard winner.
Summary
- Public image-editing leaderboards disagree in rankings, model coverage, and Elo scales, making a single top score an unreliable selection rule.
- Task-specific rankings differ from aggregate rankings because models specialize in operations such as object removal, background changes, and text editing.
- Limited samples and substantial loss rates even for top models mean deployment-specific workloads may favor a different model.
- Small internal visual comparisons also reflect individual preferences; evaluation should represent actual users and final use conditions.
model-selectionimage-generationbenchmarking
Original description
Ranking image generation models the way Design Arena does it — 26,000 battles, 62 seconds per generation — takes 20 days of compute, costs $5,000, and consumes roughly 400 marathons worth of energy. Bertrand Charpentier, cofounder and chief scientist at Pruna AI, uses that number to make a point: the same evaluation on a fast compressed model takes 7 hours and $265. Efficiency is a dimension of state of the art, not a footnote. The rest of the talk dismantles the idea that any single model holds the title. Leaderboard rankings disagree with each other — the same model goes from rank 10 on one to rank 5 on another. Most models lose 40% of their head-to-head battles, which means the top-ranked model is the wrong choice for nearly half of real use cases. His answer is the Pareto front: plot quality against latency or cost, find the frontier, and expect three or four models clustered tightly in quality score but varying up to 20x in efficiency. Evaluating this way tends to surface small specialized models rather than large foundation models. Speaker info: / bertrand-charpentier-76995ab6 https://github.com/sharpenb