Can you teach agents to cooperate without ever telling them what cooperation is? This paper bets on the shape of the information that moves between two players over time, and tries to train toward that shape instead of toward rewards.
Three real games, 200 rounds each
Each thin tile is one round, left to right. Upper row: Nydegger (A). Lower row: the opponent (B). Green is C, red is D.
The claim is that cooperation leaves a fingerprint in how information moves between players and when. A pair that is still working each other out should react a lot to each other. A pair that has settled into a pattern should have very little left to learn from each new move.
If that fingerprint is real, it could become a training signal. The optimizer never sees a cooperation label, an expert move, or a payoff. It only sees information.
The testbed is the iterated Prisoner's Dilemma. Two players pick C or D at the same time for 200 rounds. The usual payoffs define the game, but they are left out of every learning loss.
The payoff table
Points are A : B.
Cooperating is not the same as both cooperating
Nydegger vs Downing, a real game. Three counts from the paper's Equation 1.
Take the direction B to A. The "message" is B's move this round. The "response" is A's move next round. There is no chat channel; the move itself is the message.
Because the strategies are known programs, the paper can compute exact probabilities instead of estimating them. Three numbers are enough: how likely B is to play C (w), and how likely A is to cooperate next if B played C (pC) or D (pD). The flow J is how much knowing B's move reduces uncertainty about A's reply, between 0 and 1 bit.
The charts below average a related per-move score, log2(q1/q0). On a single transition it can be negative; averaged over the real distribution it equals the information. So curves can dip a little below zero.
Try it: one transition, B's move to A's next move
J = h(w·pC + (1−w)·pD) − w·h(pC) − (1−w)·h(pD), where h is binary entropy.
The strategy set is the fourteen programs from Axelrod's first tournament, rebuilt in Axelrod-Python, plus Random. Every pair played 100 games of 200 rounds: 12,000 games and 2.4 million rounds, with no noise. These are fresh simulations of reconstructions, not Axelrod's 1980 numbers.
Strategies fall into three fixed groups by how often they choose C. The groups describe behavior against this strategy set only. They do not say anything about intent, and cooperating a lot is not the same as scoring a lot.
Share of own moves that are C, across all fifteen opponents
Ten games per pairing were scored at every one of the 199 transitions, in both directions: 477,600 directional observations. Averaging by group gives a map of who informs whom.
High players talking to High players carry zero flow, and they cooperate 99.48% of the time. Mixed pairs are lopsided. Middle strategies send a lot to High strategies (0.150 bits per transition) while High sends almost nothing back (0.0004). A very cooperative player can take in plenty while giving away little.
Against an even mix of High, Middle and Low opponents
Next, small learners (30 parameters each) were trained with intrinsic signals only. F penalizes flow. H is action entropy, which rewards being decisive. G rewards being predictable to an online Bayesian predictor. Six combinations were compared over twenty seeds, against fixed Nydegger, Feld, or Joss, or against a second learner.
Flow alone changed behavior against fixed partners: learner cooperation rose by 17, 27, and 12 points over the matched starting policy. Adding H or G did not help.
With two learners the story breaks. Under F alone, the flow fell to 3 × 10−7 bits, but action entropy stayed at 0.97 bits and both cooperated only 25.5% of rounds. The agents stopped depending on each other without starting to cooperate.
Averages hide timing. Each of the nine directed group curves was fitted with six candidate shapes: zero, constant, exponential approach, logistic step, Gaussian pulse, and rise-then-decay pulse. Whole games were held out for validation, and the simplest shape within one standard error of the best was kept.
The shapes differ. High to High is flat zero. Middle to High decays like an exponential. Low to High rises and then fades. Some sharp features trace back to Graaskamp's scheduled probing, so these shapes describe this strategy set, not universal laws.
Nine directed curves, sender to receiver
Fresh fits against an even opponent mix
Instead of pushing flow to zero, the pilot asks a learner to match the High group's profile. Three objectives were compared: the original zero-seeking F, matching only the mean levels, and matching the full curve over time. The learner is tiny (five logits, no memory of who it faces) and was searched with a cross-entropy method over five seeds.
Matching the curve lifted both-cooperate rates from 22.9% to 37.2% against the 15 training strategies, and from 29.6% to 49.7% on four strategies the learner never met. Matching only the mean did slightly better. With five seeds, neither is shown to beat the other.
Percent of rounds where both players cooperate
Did the learner actually reach the target?
Against strategies absent from training
Both-cooperate percent. Five seeds, eight games per seed.
The measure sits near several older ideas. The paper keeps the differences explicit.
The next test the authors propose holds architecture, optimizer, opponents and compute fixed, compares every objective in learner-vs-fixed and learner-vs-learner settings, and adds held-out strategies, changed horizons, and recovery after a shock. Without that, a predictable but uncooperative equilibrium can satisfy an information objective.