Newest papers

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun, Yongqin Xian, Laura Bravo Sánchez, Shiye Su, Federico Tombari, Serena Y. Yeung, Volker Tresp · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30089 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

現在のコラクトル・エージェントは、多くの場合、視覚的な証拠を解釈するように促される。長い視野を越えて記憶するための認知的負担を解放するために、着用可能なアシスタントは、ユーザーの環境が進化するにつれて、積極的に有用なメモリを取得し維持する必要があります。ユーザーの名義でこれを行うには、何が重要かを識別し、リアルタイムでシーンを理解し、ユーザーが活動に焦点を当てている間に背景にメモリの維持を開始する必要があります。我々は、ストリーミングエゴセンタービデオでプロアクティブなメモリを評価するベンチマークであるProactEgoMemを導入します。3つのタスクは、環境需要における地上評価を必要とする:情報キャプチャ

Machine translation (Meta M2M100); the English original follows.

Abstract

Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Benchmarking, Benchmark (surveying), Human–computer interaction, Protocol (science), Object (grammar)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange