Maximum-Likelihood Learning of Latent Dynamics Without Reconstruction

Hromadka, Samo; Biegun, Kai; Fox, Lior; Heald, James; Sahani, Maneesh

Maximum-Likelihood Learning of Latent Dynamics Without Reconstruction

Samo Hromadka, Kai Biegun, Lior Fox, James Heald, Maneesh Sahani

Gatsby Unit, UCL
ICML 2026

Paper Code arXiv

Auxiliary decoder reconstruction of distractor-free Walker data for the RP-GSSM and the VAE-based Planet with matched latent capacity. Both models are trained on data with distractors (left). The RP-GSSM's latent space successfully captures the agent dynamics (center). In contrast, Planet's reconstruction loss drives the latent space to capture the distracting information, harming agent reconstruction (right).

Auxiliary decoder reconstruction of distractor data. The RP-GSSM reconstructions partially "average" over possible distractors, indicating that the RP-GSSM latent space captures little information about distractors. Planet reconstructions contain significant information about camera angle and background, indicating that its latents do as well.

With an expanded latent space, Planet has the capacity to capture both agent dynamics and distractors, albeit within a single entangled representation. The RP-GSSM still cleanly disentangles latent dynamical structure, resisting overfitting to the distractors.

Abstract

We address the challenge of uncovering systematic, and potentially controllable, dynamical structure underlying complex high-dimensional time-series data. Existing generative and autoregressive models have difficulty telling systematic content apart from distractors, while contrastive methods struggle to learn accurate dynamics. To address these shortcomings, we develop the Recognition-Parametrized Gaussian State-Space Model (RP-GSSM), a probabilistic framework that infers accurate latent dynamics without relying on a parametrized decoder. By eliminating explicit generative parameters, the model directs its entire representational capacity to encoding dynamically relevant state; and, being fully probabilistic, learns via maximum likelihood without auxiliary objectives or ad-hoc regularization. Combining the expressive power of a neural network encoder with exact inference under a jointly Gaussian prior allows the RP-GSSM to embed a broad class of intrinsically nonlinear dynamical systems. The RP-GSSM parameters are provably linearly identifiable in the infinite data limit. We show that the RP-GSSM recovers physically meaningful latent states from noisy video more faithfully than competing methods, more reliably identifies underlying controllable nonlinear dynamics, and remains substantially more robust to visual distractors.

Figures

Linear regression validation R² scores across models, on the linear, double pendulum, and tracking tasks, with and without distractors.

2-dimensional UMAP embeddings of RP-GSSM posterior means on the Walker environment, with four sets of representative nearest-neighbor frames.

Auxiliary decoder reconstructions of the RP-GSSM, CPC, and best generative baseline on the `CIFAR10`-Pendulum task.

Kernel ridge regression validation R² scores across training on the Walker environment with distractors.

Poster

BibTeX

@inproceedings{hromadka2026maximum,
  title     = {Maximum-Likelihood Learning of Latent Dynamics Without Reconstruction},
  author    = {Samo Hromadka and Kai Biegun and Lior Fox and James Heald and Maneesh Sahani},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  address   = {Seoul, South Korea}
}

Maximum-Likelihood Learning of Latent Dynamics Without Reconstruction

With an expanded latent space, Planet has the capacity to capture both agent dynamics and distractors, albeit within a single entangled representation. The RP-GSSM still cleanly disentangles latent dynamical structure, resisting overfitting to the distractors.

Abstract

Figures

Linear regression validation R2 scores across models, on the linear, double pendulum, and tracking tasks, with and without distractors.

2-dimensional UMAP embeddings of RP-GSSM posterior means on the Walker environment, with four sets of representative nearest-neighbor frames.

Auxiliary decoder reconstructions of the RP-GSSM, CPC, and best generative baseline on the CIFAR10-Pendulum task.

Kernel ridge regression validation R2 scores across training on the Walker environment with distractors.

Poster

BibTeX

Linear regression validation R² scores across models, on the linear, double pendulum, and tracking tasks, with and without distractors.

Auxiliary decoder reconstructions of the RP-GSSM, CPC, and best generative baseline on the `CIFAR10`-Pendulum task.

Kernel ridge regression validation R² scores across training on the Walker environment with distractors.