M3W: Real-World Multimodal Agent-Scene World Model

M3W is my research project on top-down multi-agent world modeling. I study when a neural forecast adds value over a strong motion baseline, and how to keep the cases that the baseline already handles well from getting worse.

The long-term aim is a useful agent-scene world model. The current work is narrower: causal trajectory forecasting, learning intervention gain and harm, and deciding when a neural prediction should replace a physical baseline. Combining JEPA, Transformer and other modules is not, by itself, a contribution.

I checked how many edits the full bank actually needs. At a quarter of a recording’s rows, it can still beat both references under the original rule in 38 of 72 hindsight views. Teaching a model from smaller feasible plans did not reproduce that gain. Much of the failure came from choosing the wrong trajectories to edit. Edit budgets and teacher experiment.

I separated the candidate menu from the number of edits it could make. Knowing the outcomes, the narrow menu could…