Newest papers

Reliable AI-Assisted Decisions via Selective Auditing

Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31662 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

各ラウンドで、システムは、決定の信頼性を評価する信頼性スコアと一緒に提案された決定を生成します。決定と信頼性スコアは、同じモデルによって生成される必要はありません:たとえば、一つのエージェントが決定を提案し、別々の検証エージェントがそれを評価し、正確性またはセキュリティスコアを生成します。この情報に基づき、AIの決定を自動的に受け入れるか、あるいは、提案をレビューし、受け入れるか、または修正するような専門家に相談するかを決定する必要があります。我々は、このタスクをオンラインで制限された最適化問題として定義し、その目的は専門家のオーディションのエラーの頻度を最小限に抑えることであり、検証されていない

Machine translation (Meta M2M100); the English original follows.

Abstract

We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Correctness, Computer science, Audit, Task (project management), Sequence (biology), Control (management)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange