Leanstral 1.5 has made a significant leap in formal verification capabilities, bringing powerful advancements to the field. This free model, licensed under Apache-2.0, features 6 billion active parameters and has successfully saturated the miniF2F benchmark, addressed 587 out of 672 challenges from PutnamBench, and achieved impressive scores of 87% on the FATE-H benchmark and 34% on FATE-X. The model stands out in agentic proof engineering and the validation of real-world code, unearthing five previously undiscovered bugs across 57 tested repositories. As a fully open-source tool, Leanstral 1.5 can be accessed via Hugging Face and a complimentary API, making practical proof engineering in Lean 4 more accessible than ever.
Since its inception, Leanstral has embraced a hands-on methodology to proof engineering within Lean 4. The latest iteration, Leanstral 1.5, builds on this foundation, boasting a total of 119 billion parameters, with a focus on its 6 billion active parameters. This upgrade significantly enhances formal verification, solidifying its practical application in real-world coding scenarios.
In its latest release, Leanstral 1.5 excels by saturating the miniF2F benchmark completely, while also showcasing its impressive problem-solving abilities on PutnamBench and achieving a new record for FATE-H and FATE-X. Notably, it has been trained using a multi-faceted approach—spanning mid-training, supervised fine-tuning, and reinforcement learning with CISPO—to enhance its performance across various tasks.
The training process comprises two main environments: a multiturn setting where the model attempts to prove or disprove theorems with iterative feedback from the Lean compiler, and a code agent environment that allows the model to function like a developer, managing files and commands in a real-time programming context. This dual approach enables Leanstral to effectively navigate complex proof engineering workflows and ensures rigorous validation through SafeVerify, ensuring correctness based on specified theorems.
Leanstral's evaluation takes into account several significant benchmarks. miniF2F assesses abilities in formal mathematics, including a range of challenges from fundamental problems to advanced IMO-level inquiries, while PutnamBench features 672 challenging competition-style problems that test deep reasoning skills. The FATE benchmarks cater to graduate-level and PhD-level abstract algebra inquiries, scrutinizing advanced concepts like group theory and module theory. FLTEval focuses on practical, real-world proof engineering challenges derived from actual pull requests related to Fermat's Last Theorem.
With Leanstral 1.5 achieving exceptional results, including a 100% saturation on the miniF2F validation and test sets and outperforming competitors like Seed-Prover 1.5 on PutnamBench with a low operational cost, it demonstrates the strong scalability of formal reasoning models. As its token budget increases, so does its ability to solve problems efficiently, showcasing a remarkable increase in performance across various threshold levels.
The release also includes open-sourced FLTEval, which further enhances Leanstral’s capabilities by increasing pass rates significantly while maintaining cost-effectiveness. This positions Leanstral 1.5 ahead of many larger open-source models available today.
In addition to its primary focus on mathematics, Leanstral 1.5 excels in real-world code verification, as illustrated by two notable case studies. The first involves proving the time complexity of AVL trees, a task that required deep structural induction and meticulous case handling, with Leanstral confirming the crucial O(log n) guarantees for a real implementation. The second study highlights a successful automated pipeline that detects hidden bugs in code, resulting in the identification of several violations and genuine issues across various repositories, demonstrating Leanstral's effectiveness in uncovering faults often missed by conventional testing methods.
To experience Leanstral 1.5, users can access the model under the Apache-2.0 license on Hugging Face and use it through a provided API endpoint. Setting up Leanstral within Mistral Vibe is straightforward and encourages users to explore its powerful proof engineering capabilities, whether it be in theorem proving, debugging existing proofs, or contributing to software repositories. Simply follow the setup instructions and start harnessing the full potential of Leanstral’s innovative advancements.
