Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Author

Kale-ab Abebe Tessera1, Andras Szecsenyi1, Cameron Barker1, Alexander Rutherford2, Davide Paglieri3, Aidan Scannell1, Henry Gouk1, Elliot J. Crowley1, Tim Rocktäschel3, Amos Storkey1

Affiliations

1 University of Edinburgh

2 University of Oxford

3 University College London

Posted

June 6, 2026

Venue / status

NeurIPS 2026

Evaluations & Datasets Track

Read the paper (arXiv) Main website

GitHub · Hugging Face · Submit to leaderboard

Overview

Can LLM agents coordinate in long-horizon, open-ended tasks?

ALEM is a JAX benchmark for testing multi-agent coordination in long-horizon, procedurally generated worlds. Across nine levels with controllable coordination demands, agents must explore, communicate, trade resources, craft tools, build structures, and fight mobs. ALEM supports LLMs, VLMs, RL agents, and human play.

Demo