Newest papers

Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning

Wesley Lu, Runzhou Liu, Hongru Du, Xinyao Yi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24835 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Penilaian yang ada dari pengoptimalan kode berbasis LLM pada dasarnya mengukur kecepatan end-to-end, membingungkan apakah sebuah model benar-benar melakukan optimasi yang dimaksudkan. Untuk bahasa kompilasi, ini sangat bermasalah: keuntungan yang diamati dapat mencerminkan transformasi yang sudah dapat dipulihkan oleh kompilator optimisasi modern daripada perilaku optimisasi yang dapat dikaitkan dengan model itu sendiri. Kami menyajikan PatternBench, benchmark untuk mempelajari optimasi transcendent kompilator, yaitu peningkatan kinerja yang tetap setelah optimasi kompilator agresif. PatternBench menargetkan peluang optimisasi residual di tujuh kelas transformasi dan evaluasi runtime pasangan dengan analisis optimisasi setia berbasis AST yang memverifikasi apakah optimisasi yang dimaksudkan terwujud dalam program yang dihasilkan. Eksperimen dengan model

Machine translation (Meta M2M100); the English original follows.

Abstract

Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Compiler, Benchmark (surveying), Program optimization, Measure (data warehouse), Programming language

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange