MEM: Multi-Scale Embodied Memory for Vision Language Action Models

传统端到端机器人学习中的记忆，通常只是把过去一段观测序列输入给策略。然而在复杂的多阶段真实任务中，机器人记忆必须在多个粒度上表示过去事件：既要有记录高层语义进度的长期记忆，也要有补偿遮挡和保留近期细节的短期记忆。作者的核心观点是，长时程机器人控制需要把多种模态结合起来，分别承载这些不同层次的抽象。为此他们提出 MEM，将视频短期记忆和文本长期记忆结合起来，使策略能够完成长达十五分钟的任务，并在上下文中更智能地调整 manipulation 策略。

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.