Newest papers

From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs

Hengyu Ye, Songhua Hu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26650 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

一个大语言模型(LLM)所提出的目标选择修复行动0.0指定哪些记录要更改,以及如何更改其值。 一个失败的行动可能仍然包含一个有用的目标选择记录或一个有用的更新,取决于两者是如何配对的。 我们引入了一个反事实性评估框架,该框架将受影响的记录(目标)从应用变更(更新)中分开。四个执行组合了建议的和参考目标和更新,以测量每组件的修复或保留和它们的互动的效果。 我们对两个任务进行评估:流量分配,其中受影响的需求必须从道路计数中识别出来,以及数据库修复,其中受影响的行必须从研究频率报告中识别出来。 注入的目标提供了一个可逆转的参考行动,

Machine translation (Meta M2M100); the English original follows.

Abstract

A repair action proposed by a large language model (LLM) specifies which records to change and how to change their values. A failed action may still contain a useful choice of records or a useful update, depending on how the two are paired. We introduce a counterfactual evaluation framework that separates selected records (targets) from the applied change (an update). Four executions combine proposed and reference targets and updates to measure the effects of correcting or retaining each component and their interaction. We evaluate the framework on two tasks: traffic assignment, where affected demand must be identified from road counts, and database repair, where affected rows must be identified from aggregate reports. An injected fault supplies a reference action that reverses it, and the recovery rate measures an action's loss reduction relative to that reference. For sparse-target actions in the traffic experiments, retaining the proposed update on reference targets achieves 79.4% of the reference loss reduction, compared with 43.7% from tuning update strength on the proposed targets. In the database experiments, update replacement worsens 17 of 108 actions, all selecting unaffected records, with sharply different frequencies across models. The value of a proposed update depends on the targets it is paired with. Missed targets hide useful updates, while excess targets can make a milder update better than the reference correction. In a component-reuse study, testing new target-update pairings from two proposals raises mean held-out recovery rate to 0.616, compared with 0.580 when selecting between the original proposals.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Counterfactual thinking, Set (abstract data type), Aggregate (composite), Action (physics), Computer science, Aggregate data

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange