Newest papers

From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs

Hengyu Ye, Songhua Hu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26650 · cited by 0 works (OpenAlex) · open-access PDF

Korean

대형 언어 모델(LLM)에 의해 제안된 표적 수치 선택 조치 0.5은 어떤 레코드를 변경하고 그 값을 변경하는 방법을 지정합니다. 실패한 조치는 여전히 두 가지 요소와 상호 작용의 효과를 측정하기 위해 유용한 표적 수치 선택 또는 유용한 업데이트를 포함할 수 있습니다. 우리는 영향을받은 수요가 도로 계산에서 식별되어야하는 반사적 평가 프레임워크를 도입하고 데이터베이스 수리에서 영향을받은 행렬을 연구 요약 보고서에서 식별해야합니다. 4개의 실행은 제안된 표적 및 참조 목표 및 업데이트를 결합하여 각 구성 요소 및 그 상호 작용의 효과를 측정합니다. 우리는 두 가지 작업에 대한 프레임워크를 평가합니다: 트래픽 할당, 도로

Machine translation (Meta M2M100); the English original follows.

Abstract

A repair action proposed by a large language model (LLM) specifies which records to change and how to change their values. A failed action may still contain a useful choice of records or a useful update, depending on how the two are paired. We introduce a counterfactual evaluation framework that separates selected records (targets) from the applied change (an update). Four executions combine proposed and reference targets and updates to measure the effects of correcting or retaining each component and their interaction. We evaluate the framework on two tasks: traffic assignment, where affected demand must be identified from road counts, and database repair, where affected rows must be identified from aggregate reports. An injected fault supplies a reference action that reverses it, and the recovery rate measures an action's loss reduction relative to that reference. For sparse-target actions in the traffic experiments, retaining the proposed update on reference targets achieves 79.4% of the reference loss reduction, compared with 43.7% from tuning update strength on the proposed targets. In the database experiments, update replacement worsens 17 of 108 actions, all selecting unaffected records, with sharply different frequencies across models. The value of a proposed update depends on the targets it is paired with. Missed targets hide useful updates, while excess targets can make a milder update better than the reference correction. In a component-reuse study, testing new target-update pairings from two proposals raises mean held-out recovery rate to 0.616, compared with 0.580 when selecting between the original proposals.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Counterfactual thinking, Set (abstract data type), Aggregate (composite), Action (physics), Computer science, Aggregate data

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange