Newest papers

Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning

Wesley Lu, Runzhou Liu, Hongru Du, Xinyao Yi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24835 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

LLMベースのコードの最適化の既存の評価は、主に端から端のスピードアップを測定し、モデルが実際に意図された最適化を実行するかどうかを隠蔽する。コンパイルされた言語の場合、これは特に問題である:観測された利益は、現代の最適化コンパイラによってすでに回復可能な変換を反映する可能性があります。我々は、ASTベースの最適化忠実な分析で、ASTベースの最適化行動を評価するよりも、現代の最適化コンパイラによってすでに回復可能な変化を反映します。我々は、構想された最適化が生成されたプログラムで実現されるかどうかを検証するパターンベンチ、すなわち、攻撃的なコンパイラの最適化

Machine translation (Meta M2M100); the English original follows.

Abstract

Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Compiler, Benchmark (surveying), Program optimization, Measure (data warehouse), Programming language

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange