What Should Embodied Agents Remember?

Benchmarking and Evaluating Memory Modality for Embodied Agents

Junxiang Wang^1,2†, Yuchen Liu^1,3†, Yi Wu¹, Guofei Chen¹, Nir Pechuk²,
Tao Chen¹, Zackory Erickson², Yuzhe Qin¹, Tianyu Li¹

¹Dexmate   ²Carnegie Mellon University   ³Purdue University
^†Work done during an internship at Dexmate

MeMoREval is a benchmark for studying which Memory Modality an embodied agent should use to remember what it sees, with a total of 53 robotics tasks. It measures task success together with the write-time and read-time inference costs of building and using memory.

Release status

The benchmark code, including simulation, skills, memory backends, agent, and grading, will be released after peer review. Currently we release our design tasks, scenes, and robot asset.

Tasks

tasks/ contains specifications of the tasks we designed for MeMoREval, as one JSON file per task with its experience, its requests, reference answers and success predicates.

Scenes

Tasks are implemented in four environments adapted from SceneSmith scenes. They include simulator-ready meshes, articulated furniture, and navigation maps for the robot. The scenes are hosted on Hugging Face.

hf download jimwang418/MeMoREval-scenes --repo-type dataset --local-dir data/scenes

Robot

The simulated Dexmate Vega 1 robot is in robots/ as an MJCF model.

Citation

@misc{wang2026memoreval,
  title         = {What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents},
  author        = {Wang, Junxiang and Liu, Yuchen and Wu, Yi and Chen, Guofei and Pechuk, Nir and Chen, Tao and Erickson, Zackory and Qin, Yuzhe and Li, Tianyu},
  year          = {2026},
  eprint        = {2610.12561},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.12561}
}

License

This repository is released under the Apache License 2.0. See NOTICE for third-party attributions.

Contact

Open an issue, or email Junxiang Wang (junxiang@cmu.edu) or Tianyu Li (ty.li@dexmate.ai).