NaturalPRISM: A Natural Language Premise Retrieval Task for Research-level Intermediate Proof States
A benchmark of 1.78 million query/answer pairs that match a partial natural-language proof with the premise invoked at its next step, mined from 235,314 arXiv mathematics papers spanning all 32 subject categories. Existing premise-retrieval benchmarks are either built for formal languages like Lean, or are small enough to limit fine-tuning. We fine-tuned Qwen3-Embedding models on domain-specific subsets and found that specialized 4B retrievers outperform generalist models nearly twice their size, and that routing queries to domain specialists through a lightweight classifier performs better still.