Newest papers

Differentially Private Sparse Reward Estimation with Preference Feedback

Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30099 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

然而,对敏感偏好数据的依赖需要严格的差异性隐私(DP)保证,以防止用户情绪泄露。现有的私人平衡方法通常假定密集的奖励参数化,导致估计错误的局限性与环境(模型大小 $d$)之间多元化。这种依赖性在现代 {overparameterized 模型}中是禁止的,在这种工作中,我们研究了 \h{Private Sparse RLHF} 对这个问题的密集性,假定人类偏好率(奖励模型)在环境中的多元化范围内是由 $s*d$ 的相关特征组成。我们提供了最全面的私人差异性奖励的框架,我们从对比的估计(Bradon $$$$Utilitwork)

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size $d$}. Such dependence is prohibitive in modern {overparameterized models} where $d$ is vast. In this work, we study \emph{Private Sparse RLHF} problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of $s^* \ll d$ relevant features. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, under the squared $\ell_2$ parameter estimation error, we establish upper bounds of $\widetilde{O}(s^*/n + {s^*}^2/n^2\varepsilon^2)$ for $(\varepsilon,δ)$-Sample-DP and $\widetilde{O}(s^*/n + s^*/n\varepsilon^2)$ for $\varepsilon$-Label-DP based on randomized response, where $n$ is the training data size. For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework. We also provide a general lower bound for the problem with Label-DP. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks, offering a significantly more favorable privacy-utility trade-off.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pairwise comparison, Differential privacy, Computer science, Estimation, Preference, Reinforcement learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange