Newest papers

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

Siyu Wang, Yifan Wang, Yuecheng He · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.03080 · cited by 0 works (OpenAlex) · open-access PDF

Korean

두 번째 역할의 실패 모드는 조용하다: 생성된 코드 실행, backtest 플로트, 그러나 트레이더가 정의한 특정 위험 논리는 논리가 실행되지 않는다. 기존 코드 벤치마크는 단위 테스트 및 금융 벤치마크 테스트 예측에 대한 기능적 정확성을 테스트합니다. 두 번째 역할의 실패 모드는 조용하지 않습니다. 우리는 MintEval을 도입하고, 거래자가 정의 할 수있는 빌딩 블록의 라이브러리에서 프로그래밍적으로 생성되는 위험 논리를 소개합니다. 기존 코드 벤치마크는 단위 테스트 및 금융 벤치마크 테스트 예측에 대한 기능적 정확성을 테스트하고, 테스트하에 모델에 의해 재현되지 않습니다. 생성 및 참조 프로그램은 테스트하에 실행되는

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader described is not the logic being executed. Existing code benchmarks test functional correctness on unit tests and finance benchmarks test forecasting; neither measures whether an implementation behaves like the strategy that was asked for. We introduce MintEval, a benchmark in which reference strategies are generated programmatically from a library of composable building blocks, back-translated into colloquial trader instructions, and re-implemented by the model under test. Generated and reference programs are executed bar by bar on identical market data and frictions, and compared on their actions rather than on code similarity or profit: alpha is differenced away. MintEval v0 contains 800 tasks on BTCUSDT 15-minute data, stratified by an execution-measured state-span complexity tau that is decoupled from description length. Low-cost models reach a mean ActionMatch of at most 0.544 and reproduce at most 0.087 of tasks exactly; on a stratified subset of 200 tasks a frontier model (Claude Opus 5.5) reaches 0.889 and reproduces 0.575 exactly, yet still fails silently on 0.275 of tasks. Given a menu of building blocks, models identify the strategy almost perfectly, yet 79.2% of the implementations whose specification was read correctly diverge on more than 10% of active bars. The LLM judge of a recent strategy-generation benchmark, applied verbatim, accepts every one of these silent failures.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Computer science, Correctness, Code (set theory), Unit testing, Implementation

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange