Long-Horizon Video World Models

HippoCampus: Hierarchical Memory for Long-Horizon Video World Models

A hybrid memory architecture that keeps recent video detail close, compresses older history by timescale, and adapts a compact global state online.

Chensheng Peng Yuanhao Cai Chenyang Zhang Belinda Zeng Menglin Jia

arXiv Explore the method View results

Overview

Long videos are a memory problem.

Video world models must preserve scene layout, object identity, motion, and camera trajectory across hundreds of frames. Full attention is too expensive; a short local window is efficient but forgetful.

HippoCampus processes video autoregressively in fixed-length chunks and separates memory into complementary roles. Recent chunks remain explicit for visual continuity, older history is compressed at multiple resolutions, and a lightweight fast-weight memory adapts to the current scene during inference. The pretrained video backbone stays frozen while only the memory state evolves.

HippoCampus overview showing action-conditioned video chunks, camera trajectories, and hierarchical spatio-temporal memory compression. — HippoCampus processes action-conditioned video in chunks, compresses visual history across multiple timescales, and preserves camera-aware context for long-horizon prediction.

Local fidelity

Sliding-window attention retains high-resolution recent chunks for motion, texture, and clean chunk boundaries.

Hierarchical context

Short-, mid-, and long-term tokens are compressed progressively to extend usable history at bounded cost.

Adaptive global state

Test-time fast weights store scene-specific information without a key-value cache that grows with the rollout.

The central design principle is simple: keep the recent past detailed, make older history compact, and let global memory specialize to the scene.

Architecture

Memory at multiple scales.

The model fuses hierarchical spatio-temporal tokens with a parametric test-time memory before predicting the next action-conditioned video chunk.

Hierarchical local memory

Recent detail, longer reach

History is split into short-, mid-, and long-term regimes. Three-dimensional convolutional compressors use increasingly strong compression for older chunks, reducing token cost while retaining a longer effective context.

Short term Dense tokens for immediate motion and appearance continuity.

Mid term Moderately compressed tokens for cross-chunk scene continuity.

Long term Heavily compressed tokens enriched with camera-relative position.

Hierarchical local memory compresses short-, mid-, and long-term video history together with camera tokens. — The spatio-temporal memory compressor retains camera geometry while progressively compressing older frame tokens into a longer local context.

Adaptive global memory

Learn during inference—without changing the backbone

A lightweight neural memory is updated online using a self-supervised latent prediction objective. The fast weights form a fixed-size parametric summary that evolves from chunk to chunk, preserving scene layout, identities, motion patterns, and long-range dependencies.

Prediction stays causal: HippoCampus reads the current memory to generate the next chunk, then updates memory from the newly available observation. This avoids retaining every historical token.

Adaptive global memory reads from query features and updates fast weights with newly observed video chunks. — The update step writes sequence-specific information into fast weights; the apply step reads that adaptive global memory for the next prediction.

Evaluation

Introducing HippoBench.

HippoBench tests whether a world model remembers across long rollouts—not just whether its next few frames look plausible.

The benchmark contains 100 scene-disjoint held-out clips spanning structured driving, in-the-wild exploration, camera-controlled videos, and simulation. Difficulty increases with rollout length, viewpoint change, dynamic content, and spatial revisits.

nuScenes Sekai SpatialVID ReCamMaster

ATE RMSE ↓ FID ↓ FVD ↓ PSNR ↑ LPIPS ↓

50 Easy

Shorter rollouts, mild viewpoint change, and few or no revisits.

30 Medium

Longer rollouts with moderate camera motion or partial revisits.

20 Hard

Long rollouts, large viewpoint changes, or repeated revisits.

Quantitative results

Stronger visual memory at every difficulty.

HippoCampus leads all reported visual-quality and consistency metrics across Easy, Medium, and Hard. Its largest gains appear in FVD and LPIPS.

124.2FVD ↓

0.175LPIPS ↓

24.9PSNR ↑

0.70ATE RMSE ↓

Method	ATE RMSE ↓	FID ↓	FVD ↓	PSNR ↑	LPIPS ↓
Astra	0.84	45.6	210.4	23.8	0.212
Matrix-Game 2.0	0.80	50.3	236.2	23.1	0.219
Matrix-Game 3.0	0.79	47.8	220.5	23.5	0.205
HY-WorldPlay	0.73	42.1	208.9	24.4	0.190
Lingbot World	0.80	39.2	185.7	24.6	0.188
HippoCampus	0.70	37.2	124.2	24.9	0.175

Method	ATE RMSE ↓	FID ↓	FVD ↓	PSNR ↑	LPIPS ↓
Astra	0.95	63.9	330.4	21.8	0.281
Matrix-Game 2.0	0.92	60.5	310.8	22.1	0.268
Matrix-Game 3.0	0.90	58.7	300.9	22.3	0.256
HY-WorldPlay	0.73	45.1	214.2	23.7	0.201
Lingbot World	0.81	48.7	284.2	24.0	0.194
HippoCampus	0.74	41.6	182.7	24.2	0.182

Method	ATE RMSE ↓	FID ↓	FVD ↓	PSNR ↑	LPIPS ↓
Astra	1.18	82.7	495.1	17.9	0.358
Matrix-Game 2.0	1.12	78.4	460.2	18.5	0.342
Matrix-Game 3.0	1.10	76.8	445.9	19.8	0.333
HY-WorldPlay	0.81	58.9	301.1	21.0	0.321
Lingbot World	0.93	54.6	285.8	21.4	0.309
HippoCampus	0.89	52.9	268.3	22.7	0.244

Lower is better for ATE RMSE, FID, FVD, and LPIPS; higher is better for PSNR. HY-WorldPlay has the lowest ATE RMSE on Medium and Hard, while HippoCampus remains competitive on control and leads the visual metrics.

Qualitative comparison of long-horizon predictions across several indoor and outdoor scenes. — Qualitative long-horizon comparison. HippoCampus better preserves scene layout, object identity, and motion consistency over extended rollouts, including after camera rotation and revisits.

Ablation study

Each memory pathway matters.

Hierarchical memory supplies the largest improvement over a local window; adaptive test-time memory adds a further consistent gain across perception, reconstruction, and control metrics.

Configuration	FVD ↓	LPIPS ↓	PSNR ↑	ATE ↓
Sliding window	214.7	0.379	21.28	1.37
+ Hierarchical memory	138.2	0.191	24.42	0.93
+ TTT memory	124.2	0.175	24.96	0.70

All variants use the same backbone, tokenizer, data, rollout length, and action inputs on HippoBench Easy.

Qualitative ablation comparing memory configurations on indoor and beach camera trajectories. — Removing hierarchical or adaptive memory increases drift and weakens scene consistency along the same camera-controlled trajectory.

Discussion

Toward persistent visual world models.

Adaptive memory offers a practical path to longer video horizons for autonomous driving, robotics, and embodied simulation.

Limitations

Online updates add inference latency and depend on the self-supervised objective. Prediction errors may be reinforced, and rare events or complex physical interactions remain challenging.

Responsible use

Long-horizon generation should be paired with provenance tracking, careful validation in safety-critical settings, and license-compliant benchmark metadata.

Citation

Cite HippoCampus.

Citation information for the HippoCampus project.

@misc{peng2026hippocampus,
  title   = {HippoCampus: Hierarchical Memory for Long-Horizon Video World Models},
  author  = {Peng, Chensheng and Cai, Yuanhao and Zhang, Chenyang and
             Zeng, Belinda and Jia, Menglin},
  year    = {2026}
}