MathArena – IMO Blogpost

Introduction

Recent progress in the mathematical capabilities of LLMs have created a need for increasingly challenging benchmarks. With MathArena, we address this need by evaluating models on difficult and recent mathematical competitions, offering benchmarks that are both uncontaminated and interpretable. Among these competitions, the International Mathematical Olympiad (IMO) stands out as the most well-known and prestigious. As such, an evaluation of the IMO 2025, which took place just a few days ago, is a necessary addition to the MathArena leaderboard. In this post, we present our methodology and results from evaluating several state-of-the-art models on the IMO 2025 problems. Our goal was to evaluate whether these models could reach key milestones corresponding to medal-level performance: bronze (top 50%), silver (top 25%), or even gold (top 8%). To investigate the true limits of current LLMs, we used a best-of-n selection methods to scale inference-time compute as much as possible in an attempt to reach one of these milestones.

The best-performing model is Gemini 2.5 Pro, achieving a score of 31% (13 points), which is well below the 19/42 score necessary for a bronze medal. Other models lagged significantly behind, with Grok-4 and DeepSeek-R1 in particular underperforming relative to their earlier results on other MathArena benchmarks. We also share some initial qualitative observations in this post, but we invite the community to conduct their own analyses. Visit our website to explore the raw model outputs and dive deeper into the results.

Methodology

Setup We followed a methodology similar to our evaluation of the 2025 USA Math Olympiad [1]. In particular, four experienced human judges, each with IMO-level mathematical expertise, were recruited to evaluate the responses. Evaluation began immediately after the 2025 IMO problems were released to prevent contamination. Judges reviewed the problems and developed grading schemes, with each problem scored out of 7 points. To ensure fairness, each response was anonymized and graded independently by two judges. Grading was conducted using the same interface developed for our Open Proof Corpus project [2].

Models We evaluated five state-of-the-art models: o3, o4-mini, Gemini-2.5-Pro, Grok-4, and Deepseek-R1 (05/28). These were selected based on prior performance on the MathArena competitions. Each model was run with the recommended hyperparameters and a maximum token limit of 64,000. No models needs more than this number of tokens. We used the same prompting strategy as in our Open Proof Corpus evaluation (provided at the bottom of this post). For each problem, each model generated four distinct responses.

Best-of-n Selection A key critique of our USAMO evaluation was that models shouldn’t be expected to answer extremely difficult problems in a single attempt. This critique is even more relevant for the even more difficult IMO problems. To mitigate this limitation, we applied a best-of-32 selection strategy using a method based on previous work [3].
In our prior work [2], we found that this method works very well for proof generation tasks, almost doubling performance of the models on the data we had at hand.
Specifically, for each model solution, we first generated 32 responses. These responses were evaluated in a bracket-style tournament using an LLM-as-a-judge system to select winners in head-to-head comparisons. Here, the model itself was used to evaluate its own responses. The model judged each pair and selected the stronger response. This process was repeated until a single best response remained which was then presented to the human judges for evaluation.
We use the same prompt for judging as in our own prior work and we repeat it at the bottom of the post for completeness. This selection process was computationally and financially intensive: on average, each final model answer cost at least 3 dollars to generate, with Grok-4 costing over 20 dollars per answer.
As such, the performance reported here represents the models’ best achievable output within a reasonable resource budget.

Results

As mentioned above, Gemini 2.5 Pro achieved the highest score with an average of 31% (13 points). While this may seem low, especially considering the $400 spent on generating just 24 answers, it nonetheless represents a strong performance given the extreme difficulty of the IMO. However, these 13 points are not enough for a bronze medal (19/42). In contrast, other models trail significantly behind and we can already safely say that none of them will achieve the bronze medal. Full results are available on our leaderboard, where everyone can explore and analyze individual responses and judge feedback in detail.

Qualitative Analysis

Below, we present a few qualitative observations based on model responses to highlight the kinds of mistakes and behaviors we encountered.

Grok-4 Performs Poorly Grok-4 significantly underperformed compared to expectations. Many of its initial responses were extremely short, often consisting only of a final answer without explanation. While best-of-n selection helped to filter better responses, we note that the vast majority of its answers (that were not selected) simply stated the final answer without additional justification. Similar issues are visible on the other benchmarks in MathArena, where Grok-4’s replies frequently lack depth or justification.

Gemini and Bogus Citations Gemini-2.5-Pro continues to exhibit a problematic tendency to cite non-existent theorems when it fails to find a valid proof. As in our USAMO evaluation, we emphasize that this behavior is particularly concerning, as it misleads users by presenting false authority and undermines trust in the model’s reasoning. However, we do note that this behavior was less prevalent in the IMO responses compared to the USAMO, suggesting some improvement in this area.

More “Solid” Answers In contrast to earlier evaluations, we noticed fewer cases of odd formatting issues or behaviors linked to models over-optimizing for final-answer formats, such as boxing entire proofs or assuming numerical answers were always required. This suggests that models have made progress in handling open-ended mathematical reasoning tasks more robustly.

Partial Credits In math competitions like the IMO, it is relatively rare for human participants to receive a medium score of 3 or 4 out of 7 on a question. In contrast, LLMs often received partial credit from our judges, especially on problem 4 and 5. For Problem 4, this was usually because most models adopted a generally human-like approach but suffered from logical lapses that significantly reduced their scores. For Problem 5, models often identified the correct strategies but failed to prove them, which is, ironically, the easier part for an IMO participant. This contrast highlights key differences between human and model performance and suggests that models could improve significantly in the near future if these relatively minor logical issues are addressed.

Best-of-n is Important One of our judges briefly looked at a subset of the 32 raw responses generated by the models prior to the best-of-n selection. They observed that many of these responses were quite poor and estimated that, without the best-of-n filtering, model scores would likely have fallen below 10%.
Interestingly, the judge noted that some unselected answers appeared more coherent than the chosen ones, but actually contained more factual errors. This suggests that the models are surprisingly effective at identifying the relative quality of their own outputs during the best-of-n selection process and are able to look past coherence to check for accuracy.

References

• [1] Petrov, Ivo, et al. “Proof or bluff? Evaluating LLMs on 2025 USA Math Olympiad.” arXiv preprint arXiv:2503.21934 (2025).
• [2] Dekoninck, Jasper, et al. “The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofs.” arXiv preprint arXiv:2506.21621 (2025).
• [3] Liu, Yantao, et al. “Pairjudge rm: Perform best-of-n sampling with knockout tournament.” arXiv preprint arXiv:2501.13007 (2025).

Prompts

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!