Newest papers

StreamEval: Evaluating Interactive Streaming Video Generation Models

StreamEval Team · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28402 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Como a geração de vídeo de streaming de confiança avança, a possibilidade de interação em tempo real com o vídeo gerado torna-se cada vez mais importante. Introduzimos o StreamEval, um benchmark unificado para a geração de vídeo de streaming interativo que abrange cinco tarefas: diálogo de avatar, desempenho de avatar, geração mundial, edição de instruções e edição de referência. O StreamEval compreende 300 casos de teste de script com duração de 15 a 300 segundos, cada um especificando ativos iniciais, pedidos multimodais incrementalmente chegados e requisitos explícitos para mudanças de estado e preservação de conteúdo. O desempenho é avaliado ao longo de quatro dimensões de avaliação: execução de tarefas confiável, consistência de estado, fidelidade visual e coordenação audiovisual,

Machine translation (Meta M2M100); the English original follows.

Abstract

As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval comprises 300 scripted test cases lasting 15-300s, each specifying initial assets, incrementally arriving multimodal requests, and explicit requirements for state changes and content preservation. Performance is assessed along four evaluation dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination, instantiated through ten task-specific metrics on a 0-100 scale. Evaluating 37 distinct models across 300 test cases, we find that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. In particular, request execution remains challenging: the highest task-specific scores are 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reported reference-editing models. These findings identify reliable request execution under timing and preservation constraints as a central challenge for advancing omni-modal real-time interactive systems. The leaderboard is available at https://streameval.ai

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Avatar, Benchmark (surveying), Task (project management), State (computer science), Interactive video

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange