Modeling Social Learning Through Demonstration in Multi-Armed Bandits

Julio Martinez^1,*, Michael C. Frank¹, Nick Haber^2,3
¹Department of Psychology, Stanford University · ²Graduate School of Education, Stanford University · ³Department of Computer Science, Stanford University
Proceedings of the 46th Annual Meeting of the Cognitive Science Society (CogSci 2024)

PDF eScholarship Code Google Scholar BibTeX

Screenshot of the asteroid mining game: six asteroids across the top, the player's starship below, and the Mineral Mining Reward Tank

Asteroid Fortune Frontier. Each asteroid is a bandit arm with a hidden payoff probability. Players steer their starship with the arrow keys and shoot to mine. A hit leaves a green mark on the asteroid and adds a green bar to the Mineral Mining Reward Tank; a miss leaves a red mark and adds a red bar. Six of a zone's 12 asteroids are shown.

The question

People are efficient social learners. We pick up skills, language, and culture by watching others, and we lean on that most when first-hand experience is costly or the environment is complex. What is much less clear is the computation. How does information gleaned from watching someone else get combined with what we have learned ourselves, and how does that combination change when the person we are watching is an expert rather than a novice?

We studied this in multi-armed bandits, the simplest setting where exploring and exploiting trade off (Martinez et al., 2024). In a Bernoulli bandit each arm pays out 1 with some unknown probability and 0 otherwise, and a learner has to find the good arms by trying them. Bandits have a rich theory of optimal play and a long history in human experiments, which makes them a good place to ask precise questions about social learning. Our contribution is a game in which people learn both by playing and by watching others, plus a set of models built on standard bandit solvers that can be compared with people directly, with no fitting to human data.

An asteroid-mining bandit

We turned a 12-armed Bernoulli bandit into a space game. Asteroids are the arms, each with its own hidden payoff probability. Players can move their starship left and right as much as they like but have a limited number of shots: 24 per game. Each shot either returns a mineral reward or nothing. A game ends when the tank is full.

Every zone of the galaxy has new payoffs. We drew them so that 3 of the 12 asteroids were poor (payoff between 0 and 0.2), 8 were middling (between 0.45 and 0.55), and one was clearly best (between 0.75 and 1). We designed 10 base asteroids and varied them by rotation and hue into a pool of 80, drawing 12 at random for each zone so that no two zones looked alike.

The player's starship, three asteroid designs, and the teacher's starship

Some of the cast. Left: Proctor One, the player's starship. Center: three of the asteroid designs. Right: the second starship that appears during watch games, flown by the teacher.

Play and watch

Each participant worked through five combinations of a block condition and a teacher. A block is a sequence of three games: play-play-play, play-watch-play, or watch-play-play. In play games the participant plays. In watch games they watch a teacher play the same zone from a second starship for 24 steps. The teacher was either a novice or an expert, and nothing on screen said which. Every block began in a fresh zone with fresh payoffs, and each of the five condition pairs was presented twice in random order.

Conditions. A block is three games. Watch games are colored by the teacher, blue for an expert and red for a novice, which gives five blocks in all.

Teachers were generated rather than recorded. For each of 100 stored bandits we ran Thompson sampling for 72 steps, 20 times. The first 24 selections of a run, when the sampler is still exploring, served as a novice demonstration. The last 24, when it is mostly exploiting the better arms, served as an expert demonstration. During a watch game one of the stored policies for the zone’s bandit was chosen at random and replayed, with its actual outcomes.

How teachers were made. Sample a bandit, run Thompson sampling for 72 steps, and split the run: the first 24 selections, while the sampler is still exploring, become a novice teacher; the last 24, when it mostly selects the best arm, become an expert teacher.

We recruited 200 adults on Prolific. After excluding participants who reported colorblindness or refreshed the page mid-experiment, which reset their condition assignment, 153 remained.

What watching did

Human results. Mean total reward on each game of a block, by block order and teacher. Shaded games are watch games, where the triangle is the teacher's own score. Error bars are 95% confidence intervals.

Everyone improved over a block, but how much depended on the teacher. We fit a mixed-effects model to total reward on the second play game of each block, which compares the same amount of direct experience with and without social input in between. Watching an expert raised reward by about 2.4 points in play-watch-play and 1.9 points in watch-play-play over the play-play-play baseline of 15.6. Novice demonstrations helped much less: a small but reliable effect when the novice was watched first, and no reliable difference when the novice was watched between two play games. The two block orders did not differ from each other.

We modeled participants with two standard bandit solvers. Thompson sampling keeps a Beta distribution over each arm’s payoff, parameterized by a count of rewarded selections, α_i, and a count of unrewarded selections, β_i. On each step it samples a payoff for every arm from those distributions and selects the arm with the highest sample. The Upper Confidence Bound (UCB) algorithm keeps a selection count N_i and a reward total r_i per arm and selects the arm with the best optimistic value. Both are known to describe human bandit play well.

From a teacher's selections to a belief. Five arms, five colors. The learner's own play leaves it with counts of rewarded and unrewarded selections of each arm i, α_i and β_i, the solid bars. A network reads the teacher's 24 selections and estimates the counts behind them, α_i^watch and β_i^watch, the hatched bars, including experience it never saw. The two are added, and Beta(α_i + α_i^watch, β_i + β_i^watch) is the learner's belief about arm i, the curves. Here the estimate is drawn refining selection by selection for illustration; the model reads the whole window at once.

Social information enters through one additive rule. Whatever the learner estimates about the teacher’s experience from a watch game gets added to its own parameters:

\[\alpha_i \leftarrow \alpha_i + \alpha_i^{\mathit{watch}}, \qquad \beta_i \leftarrow \beta_i + \beta_i^{\mathit{watch}}\]

with the same rule applied to N_i and r_i for UCB. The models differ only in how they estimate the watch parameters.

Naive estimator. Treat the teacher’s 24 selections as if they were your own: count the rewarded and unrewarded selections of each arm.
Hybrid choice-only. Train a multi-layer perceptron (MLP) to look at a window of 24 selections and predict the Thompson parameters that the sampler which produced them had reached by the end of the window, including experience from before the window that was never observed. The network was trained on 100,000 simulated Thompson runs on random 12-armed bandits. It sees only selections, not outcomes, so it has to infer how experienced the teacher is from what the teacher selected.
Hybrid choice-reward. The same estimator with the teacher’s rewards added to its input.

Because α and β convert directly into selection counts and reward totals, the same estimators serve the UCB learner. To simulate a participant we ran the matching solver through the exact games they saw: the same bandit, the same payoffs, and the same teacher demonstration. Nothing was fit to the human data.

Results

Humans and Thompson models. Mean reward on the first and second play games of each block. Top row: participants. Lower rows: Thompson sampling with each estimator. Black is play-play-play; blue and red are expert and novice teachers. Error bars are 95% confidence intervals.

The naive estimator misses the main effect. Treating a teacher’s selections as your own helps after any teacher, so the model gains about as much from a novice as from an expert. In watch-play-play it even starts the first play game slightly better after a novice. People did not behave that way.

The hybrid models reproduce the pattern. Both show the steep gain between plays after watching an expert in play-watch-play, and a head start in the first play game of watch-play-play, with novices well below. The Thompson hybrid choice-reward model had the best condition-level fit to mean reward, r = 0.976 with a root-mean-square error under one point, and the choice-only variant was close behind. Both hybrids also tracked how spread out participants’ selections were, their entropy, better than the naive model, and there the choice-only model was slightly ahead. At the level of individual games the correlations are smaller for every model, as expected from the noise in single runs, and the UCB choice-reward model edged out the rest.

		Condition level			Participant level
Criterion	Model	r	p	RMSE	r	p	RMSE
Reward	Thompson naive	0.703	0.023	1.648	0.191	<0.001	5.201
	Thompson hybrid choice-only	0.974	<0.001	1.028	0.285	<0.001	5.017
	Thompson hybrid choice-reward	0.976	<0.001	0.869	0.269	<0.001	5.063
	UCB naive	0.617	0.057	2.592	0.219	<0.001	5.552
	UCB hybrid choice-only	0.895	<0.001	1.864	0.296	<0.001	5.407
	UCB hybrid choice-reward	0.900	<0.001	2.042	0.316	<0.001	5.405
Entropy	Thompson naive	0.712	0.021	5.703	0.138	<0.001	8.026
	Thompson hybrid choice-only	0.930	<0.001	5.387	0.153	<0.001	7.835
	Thompson hybrid choice-reward	0.917	<0.001	5.221	0.161	<0.001	7.732
	UCB naive	0.556	0.095	4.968	0.133	<0.001	7.535
	UCB hybrid choice-only	0.789	0.007	4.415	0.150	<0.001	7.274
	UCB hybrid choice-reward	0.785	0.007	4.185	0.163	<0.001	7.001

Human-model agreement. Pearson correlation and root-mean-square error between participants and each model, for total reward and for the entropy of arm selections, computed on condition means and on individual games. Best values in bold.

What it means

Watching an expert is worth more than watching a novice, and a learner that simply adds a teacher’s observations to its own cannot explain that. A learner that first infers how experienced the teacher is, and weights the demonstration accordingly, can. The surprising part is how little that inference needs. A network that saw only which arms the teacher selected, never whether those selections were rewarded, estimated the teacher’s experience almost as well as one that also saw the rewards. That raises a question about human social learning: do we track a teacher’s payoffs at all, or mostly their actions?

Two limitations are worth stating. The estimators predict Beta parameters, which are the sampler’s internal state rather than anything observable, so the hybrid models show that expertise can be inferred and integrated this way without claiming that this is the mechanism people use. And bandits with alternating play and watch games are a narrow slice of social learning. We are extending the environment toward contextual bandits and toward transfer settings where the teacher stays the same but the payoffs change.

Code

AsteroidMiningExperiment_CogSci2024: the game and the online experiment (currently private).
SocialBanditsModeling: the Thompson and UCB simulations and the neural-network estimators (currently private).
social-rl-analysis: the R analysis of the human data and the human-model comparisons.

Humans are efficient social learners who leverage social information to rapidly adapt to new environments, but the computations by which we combine social information with prior knowledge are poorly understood. We study social learning within the context of multi-armed bandits using a novel "asteroid mining" video game where participants learn through active play and passive observation of expert and novice players. We simulate human exploration and social learning using naive versions of Thompson and Upper Confidence Bound (UCB) solvers and hybrid models that use Thompson and UCB solvers for direct learning together with a multi-layer perceptron to estimate what should be learned from other players. Two variants of the hybrid models provide good, parameter-free fits to human performance across a range of learning conditions. Our work shows a route for integrating social learning into reinforcement learning models and suggests that human social learning conforms to the predictions of such models.