Measuring and Modeling Physical Intrinsic Motivation

Julio Martinez^1,*, Felix Binder², Haoliang Wang³, Nick Haber^4,5, Judith Fan^1,3, Daniel L. K. Yamins^1,5
¹Department of Psychology, Stanford University · ²Department of Cognitive Science, UC San Diego · ³Department of Psychology, UC San Diego · ⁴Graduate School of Education, Stanford University · ⁵Department of Computer Science, Stanford University
Proceedings of the 45th Annual Meeting of the Cognitive Science Society (CogSci 2023)

arXiv PDF Google Scholar BibTeX

The eight physical scenarios from the Physion benchmark that participants rated: Dominoes, Support, Collide, Contain, Drop, Link, Roll, and Drape.

The question

From infancy, people are drawn to explore their physical surroundings and to play with the objects in them. That exploration is not random. Curiosity has long been described as an intrinsic motivation that closes gaps in knowledge, and children prefer stimuli whose outcomes are neither fully predictable nor fully unpredictable. We wanted to formalize this drive for the physical world: what is the intrinsic reward function (IRF) that directs a person toward some physical events and away from others?

Our approach was to treat “interestingness” as a proxy for intrinsic reward. We asked people how interesting they found short videos of physical events, then asked which candidate reward functions, computed from the same videos, best predicted those judgments (Martinez et al., 2023).

Theoretical framework for an intrinsically motivated agent and the hypothesis space of intrinsic reward functions

Framework. (a) An intrinsically motivated agent acts in a simulated environment, uses a world model (WM) to predict physical dynamics forward in time, and chooses actions with a policy model that optimizes an intrinsic reward produced by an IRF. (b) Candidate IRFs lie on a continuum: simple scene features on the left, and functions that increasingly aim to improve the world model toward the right.

Stimuli and task

We repurposed the Physion benchmark, a set of 3D videos simulated in ThreeDWorld that span rigid and soft-body dynamics across eight scenario types. Physion was built to measure physical prediction accuracy. We changed the question: instead of asking what will happen, we asked how interesting the event was.

Samples of the Collide, Drop, and Support stimuli. Each scenario has 2,000 training videos used to train the world models and 150 test videos that participants rated.

We recruited 1,800 participants on Prolific. Each watched the 150 test videos of one scenario in random order. A trial began with a fixation cross, showed the first frame of the video for two seconds, played the video, and then asked for a rating from 1 (least interesting) to 5 (most interesting). Each video received about 32 ratings. Mean ratings were highly reliable: the split-half reliability, averaged across scenarios, was 0.935.

Stimuli, task procedure, reliability, and modeling pipeline

Stimuli, task, and pipeline. (a) Examples of the eight scenarios. (b) The rating task. (c) Split-half reliability of mean ratings per scenario. (d) The same videos go to participants and to each candidate IRF; a model's score is its correlation with mean human ratings.

Candidate reward functions

We compared two families of intrinsic reward functions.

Simple scene features need no world model. They are statistics that can be read off the scene directly: object positions and velocities, their variances, the number of objects and object categories, the number of static distractors, and the number of collisions.

World-model-based functions depend on a learned physics predictor. We used DPINet, a particle-based graph neural network whose Physion accuracy is the closest to human accuracy among the models tested, pretrained on the Physion training set. On top of it we implemented four classic IRFs from the curiosity literature:

Novelty (RND). Random network distillation: reward is how poorly a trained predictor matches a fixed random network’s embedding of the current state, so unfamiliar states score high.
Adversarial. Reward equals the world model’s prediction error on a k-step rollout. Events the model gets wrong are rewarding.
Disagreement. Reward is the variance across an ensemble of world models’ predictions. Events the models cannot agree on are rewarding.
δ-Progress. Reward is how much the world model’s error dropped between an older and a newer checkpoint. Events the model is learning about are rewarding.

What the world model sees. We bypass perception and give DPINet an explicit state: every object as a cloud of particles with positions and velocities. Shown here for Dominoes, Contain, Roll, and Drape.

Rather than compare human free-play actions with an agent’s actions, which would require also committing to a policy model, we computed each IRF’s total reward directly on the rated videos and correlated it with mean human ratings. World-model-based rewards were computed at several DPINet training checkpoints and for one- to four-step rollouts. Composite models were fit with L1-regularized linear regression on 80/20 splits of the stimuli.

What people found interesting

Dominoes: the video rated most interesting (left) and the one rated least interesting (right).

Collide: one of the highest-rated videos (left) and the lowest-rated video (right).

Results

World-model-based rewards explain human ratings better than scene features, and adversarial reward does best. Several scene features were indistinguishable from zero. Three of the four best single predictors were world-model-based. Adversarial reward, the world model’s own prediction error, had the highest cross-scenario correlation with human ratings, r = 0.36.

Correlations of individual intrinsic reward functions with human ratings, and their stability across training steps and rollout lengths

Single-model results. (a) Correlation of each IRF with mean human ratings; the dashed line is the noise ceiling. (b) World-model-based rewards are stable across rollout length and, after about 100k training steps, across training. δ-Progress is the exception: its correlation flips sign over training, so (a) shows it at two checkpoints.

Scene features do not generalize across scenarios; world-model rewards do. Linearly combining all scene features raises the overall correlation but makes Drop and Drape worse, because the sign of a feature’s relationship to interestingness flips from scenario to scenario. Every world-model-based reward keeps the same sign in every scenario.

Composite model results across scenarios

Composite models. Left: all scene features combined. Center: all world-model-based rewards plus RND. Right: adversarial reward plus the number of collisions. Dots above each column mark the per-scenario noise ceiling.

Everything is still far below the noise ceiling. Human ratings agree with each other at 0.935; the best model reaches 0.36. There is a lot of structure in what people find interesting that none of these reward functions capture yet.

Adversarial reward and collision count are complementary. Of all two-feature combinations with adversarial reward, adding the raw number of collisions improved predictions the most.

Per-scenario correlation signs for each reward function, and gains from pairing adversarial reward with each other feature

Generalization and complementarity. Bottom: the sign of each IRF's correlation in each scenario. All-green columns generalize; mixed columns do not. Top: predictive accuracy when adversarial reward is paired with each other feature, ordered by gain.

Two motives

Two highly rated videos, interesting for different reasons. Left: high adversarial reward with few collisions, an outcome the world model did not predict. Right: many collisions with low adversarial reward, a lot happening but nothing surprising.

The results point to two components of physical intrinsic motivation. People are drawn to events that are hard to predict, which is what adversarial reward measures and what an information-seeking account expects. People are also drawn to events where a lot happens, which collision count captures and prediction error does not. Dominoes make this vivid: a long chain of collisions is fun to watch even when every fall is foreseeable.

Whether this split reflects something fundamental about curiosity or an artifact of rating passive videos rather than interacting with a scene is an open question. Either way, the gap to the noise ceiling says there is a better reward function to find, and the generalization results say it should be built on a world model.

Acknowledgments

This work was supported by Simons Foundation grant 543061, National Science Foundation CAREER grant 1844724, Office of Naval Research grant S5122, and the Stanford Human-Centered Artificial Intelligence Inaugural Fellowship.

Humans are interactive agents driven to seek out situations with interesting physical dynamics. Here we formalize the functional form of physical intrinsic motivation. We first collect ratings of how interesting humans find a variety of physics scenarios. We then model human interestingness responses by implementing various hypotheses of intrinsic motivation including models that rely on simple scene features to models that depend on forward physics prediction. We find that the single best predictor of human responses is adversarial reward, a model derived from physical prediction loss. We also find that simple scene feature models do not generalize their prediction of human responses across all scenarios. Finally, linearly combining the adversarial model with the number of collisions in a scene leads to the greatest improvement in predictivity of human responses, suggesting humans are driven towards scenarios that result in high information gain and physical activity.