Newest papers

VaDar: Implicit Goal Conditioning via Value Disaggregation

Shashwat Saxena, Mehul Goel, Sreyas Venkataraman, Sarvesh Patil, Max Simchowitz · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26629 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

长期任务对现代人工智能和机器人来说很重要,但对增强学习却构成了挑战。历史上,一种流行的方法是将任务分解为次要目标,并学习次要目标为条件的策略,以便在执行每个次要任务的顺序中进行。我们引入了一种名为Value Disaggregation(VaDar)的替代方法,该策略通过仔细的实验训练一个单一的次要目标独立的策略,以最大限度地实现每个次要目标的关键价值的线性组合。由于次要目标信息只需要在训练时,VaDar消除了在部署过程中可能昂贵或繁琐的次要目标选择的需求。此外,通过仔细的实验,我们表明VaDar(1)总是具有竞争力,并且经常超过目标的基线,如目标调节和(2)在非次要目标结构

Machine translation (Meta M2M100); the English original follows.

Abstract

Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Reinforcement learning, Task (project management), Computer science, Artificial intelligence, Value (mathematics), Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange