The meta-task · live, updated at each gate

An October-2026 agent vs a year of 2010 research

The original analysis and modelling took a team about a year around 2010. This project asks how far an AI agent gets on the same problem, from the raw archive, and how good the result is. The first author supervises: they approve the plan, answer questions and make the decisions, but do not do the analysis.

Time accounting

Main agent, busy time

Primary measure. Time the orchestrating agent was actually working: gaps between consecutive transcript messages, not counting idle time after it finished a turn.

Subagents, summed

subagents. Each agent's working time is the sum of the gaps between its transcript messages, counting gaps of up to 15 minutes and none after it finished a turn, so notifications that reach an agent after it is done are not counted. Parallel agents add up, so this can exceed wall-clock time.

Session wall clock

From the first prompt to the last message, including time waiting for the user. Cross-check: hooks measured of main-agent turns and of subagent time.

Phases overlap from the Phase-1 gate onward, so later rows are windows of the session rather than clean phases. Subagent time is assigned to the window in which the agent started. Measured with tools/timing_report.py (hook events plus transcripts); still-running agents are counted up to the build time.

Timeline

Session timeline: phases, subagents and user messages

Bars are subagents, coloured by model. Dark ticks are user messages. Hover for details; in the table below, hover, tap or focus a subagent's name for what it did.

Speed: the old way vs the new way

There is no MATLAB on this machine, so the 2013 code cannot be timed directly. Two honest stand-ins: a faithful serial port of the original fitting algorithm (conjugate gradients with line searches, one random start at a time, in Python/NumPy) and the archived MATLAB network run's own speed log. The new way is batched JAX: all random starts in one vectorised call, exact autodiff gradients and Hessians.

Compute time, old vs new (log x)

CPU-seconds unless noted. The factor is old ÷ new. Hover for what each number measures.

Caveats. The old way here is a Python port, not MATLAB, and ran on a machine shared with other heavy jobs (load average 100–400), so its times are pessimistic. The monkey row is an extrapolation. The network rows compare 2010 hardware with 2026 hardware as well as old code with new code. Fixing the gradient bug alone made the faithful optimiser 2.5–4 times cheaper.

How the agent worked

Roles

  • Orchestrator (the main agent, Opus): owns the plan, decisions, integration, the research log and the reports; does small, context-heavy work itself (data extraction, likelihood port, fast fitter, discrimination analysis).
  • Implementers (Sonnet or Opus, each in its own git worktree when several write in parallel): the faithful optimiser port, the full network replication, a code-quality refactor, Phase 3, this website.
  • Reviewers (Opus, fresh context): every significant result and diff is reviewed by an agent that has not seen the work being done. The Phase-1 gate review found the most serious bug of the project (R1, below).
  • Forensics and exploration (Sonnet): archive triage, RG's discrimination sessions, the true trial order.

Verification standards

  1. Likelihood equivalence before any fitting: the ported likelihood must reproduce the archived log-likelihood at the archived parameters to 10−6.
  2. The refit must never be worse than the archived optimum by more than 0.01; any improvement is reported as an optimiser failure in the original.
  3. Exact autodiff gradients, checked against finite differences.
  4. Every published number goes through an automated published-vs-replicated table.
  5. Faithful first (bug for bug), then corrected. The user decided that the corrected analysis is the real result.
  6. Every audit finding is confirmed numerically and by an independent reviewer before it is reported.

User decisions

  • Approved the plan; local git only; asked for any mistakes in the original code to be reported and fixed.
  • Skipped requesting the Walker et al. 2020 V1 recordings for now; nothing may block on them.
  • Keep the SI's parameter box as the prior; the RG session question went to a forensic subagent (verdict: oversight, about 85%).
  • Asked for a clean, modular code repository as a second deliverable, the literature map, and a new large-scale experiment (Phase 5).

Agent failure modes to watch for, and what caught them

If you run research agents yourself, these are the failure modes this project hit, grouped by kind, with the safeguard that caught each one. Every example is from logs/RESEARCH_LOG.md. Bugs in the original 2013 code are a separate matter and are listed at the end.

1. Overclaiming and over-interpreting results

A real result gets a bigger or tidier explanation than the evidence supports.

  • E7: a QDN with a "random" denominator generalised like the analytic one, but that random pool equalled the Eq. 6 pool in expectation.
  • E7: the proposed mechanism for Orhan & Ma's transfer result (the overlap of total activity, Σr\Sigma r) was contradicted by E7's own data.
  • Phase 2: TS's Mismatch "new finding" fell below log⁡30\log 30 with the wider prior box (nested 4.0 → 3.1); AJ's ImpSigma tie was the confusion recovery predicts (0 of 4 recovered); human κ\kappa values of a losing model were read as traits.
  • Phase 2: "trial order verified" (the 648-trial repeat holds under either order), and three overstatements at the gate (σ\sigma agreement, JO's winner, the ANOVA).

Caught by: an independent Opus reviewer with fresh context, every time except the trial-order claim, which the agent caught itself by checking the raw 2009 session files.

Takeaway: budget a separate reviewer for every claim-bearing result; self-review caught only one of these.

2. Silent numerical bugs

Code runs and returns plausible numbers that are wrong.

  • XLA on CPU miscompiled vmapped quadrature likelihoods at batch size 1 (Monkey A DecNoise −60,106 instead of −33,860), and miscompiled implicit broadcasts inside fused erf (up to about 103 nats off).
  • NaN gradients: a clamped square root froze about 20% of Opt-P starts (R1); the Sampling model overflowed at T≤10−3T \le 10^{-3} and killed a run.
  • Quadrature errors that look fine per trial add up over 100,000 trials: 37 nats between 64 and 128 nodes.
  • Game fitter: local optima violated the nesting inequalities (44 of 30,000 fits); the lapse term made the Hessian NaN at λ=0\lambda = 0.

Caught by: reviewers, a crash, and the agent spotting impossible numbers. Guarded since by: invariant tests, single-point vs batched cross-checks, box-edge gradient tests.

Takeaway: encode the invariants as tests, and validate on the largest dataset, not a few trials.

3. Compute and resource safety

The agent's jobs harm the machine or each other, often through library defaults it never looked at.

  • The laptop restart: UltraNest's default batch asked for an array of about 690 GB.
  • JAX ignores thread limits (about 15 cores per process), so worker counts did not bound CPU use.
  • A fit worker starved under background QoS (about 0.4 core; 14 of 84 fits in 2.5 h).
  • A pkill pattern with a regex character missed a job, which kept running and pushed the project to 10.5 cores.

Fixed by: an external guard that checks headroom, throttles the job tree and kills memory blow-ups; memory-bounded batching; a hook that blocks unguarded launches; killing by PID.

Takeaway: enforce resource limits outside the agent, not by asking it to be careful.

4. Following a plausible recommendation instead of the brief

An intermediate artefact suggests a reasonable direction, and the agent follows it instead of the original ask.

  • The first game, "Who Threw It?", gamified the replicated task. The brief asked for a game aimed at an unexplored gap in the research map; the agent took the map's "flagship" suggestion instead.

Caught by: the user.

Takeaway: re-check deliverables against the original brief at each milestone.

5. Mislabelled or misleading metrics

The number is right; its name or scope is wrong.

  • "CPU 20.6 core-hours" was summed per-job wall time, not CPU time.
  • "Wall 5.0 h" covered only the last resumed segment of the extension fits (about 7 h in total).
  • Timing: a resumed agent's earlier intervals were dropped (119 minutes lost); hand-written log times drifted by up to about 2 hours; idle setup before the first prompt was counted as busy time; turns cut off by an API error were counted as busy time.

Caught by: the Phase-2 reviewer (the first two); the agent at the Phase-1 gate (the lost 119 minutes); the website subagent during Phase 1 (the clock drift and the idle setup); a later website subagent (the turns cut off by an API error).

Takeaway: state the definition next to every number, and measure time from logs, not from the agent's notes.

6. UX and quality misjudgments

User-facing work is shown before it meets the bar the user set.

  • A plain grey-box game was shown as a milestone; the user had asked for "Angry Birds"-level polish and called it boring.

Caught by: the user. Fixed by: the next build went through 5 rounds of screenshot critique, and the agent reviewed the screenshots before showing it.

Takeaway: judge user-facing work against the user's stated bar before showing it.

7. Housekeeping that destroys work

Routine clean-up deletes files that were never committed.

  • The trained network weights (gitignored) were lost when the network agent's worktree was removed after merging; the networks had to be retrained.

Caught by: the agent, when the figures could not be redrawn.

Takeaway: copy untracked artefacts out before removing a worktree (done for every later agent).

All incidents, roughly in order (category in brackets)
  1. Phase 1. XLA ignores thread limits; one worker plus nice, later the guard. (3)
  2. Phase 2 start. "Trial order verified" was wrong; corrected from the raw session files, which revealed history effects. (1)
  3. Phase 1 gate. R1 NaN gradients (about 20% of Opt-P starts); three overstatements corrected; first Phase-2 run discarded. (1, 2)
  4. Phase 1 gate. Resumed agents' intervals dropped (119 minutes); primary measure changed to transcript busy time. (5)
  5. Phase 2. Quadrature errors accumulate over trials (37 nats); full-dataset convergence test added. (2)
  6. Timing (website subagent). Hand-written log times found to drift by up to about 2 hours, and idle setup before the first prompt counted as busy time; the timeline uses measured times only, from the first prompt. (5)
  7. Phase 2. NoisySig evidence exceeded the maximum likelihood; root cause an XLA miscompile of implicit broadcasts; jit-vs-eager test added. (2)
  8. Network figures. Trained weights lost with a removed worktree; retrained. (7)
  9. Phase 2 queue. A pkill pattern missed a job; the project briefly used 10.5 cores. (3)
  10. Phase 2 queue. Extension-fit worker starved under background QoS; relaunched at normal priority. (3)
  11. Phase 2 queue. Laptop restart from an unbounded UltraNest batch (about 690 GB); guard, batching and hook added. (3)
  12. Phase 2. XLA batch-size-1 miscompile found while testing the fix; runtime check and regression test added. (2)
  13. Phase 2. Sampling NaN gradient at T≤10−3T \le 10^{-3} killed the extension run; box-edge gradient test added. (2)
  14. E7 review. The random denominator equalled the Eq. 6 pool in expectation; the Σr\Sigma r mechanism was contradicted; claim revised after a structured-pool control. (1)
  15. Game. Grey-box prototype rejected as boring; polish pass with screenshot critique. (6)
  16. Game design check. 44 nesting violations in the fitter and a NaN Hessian at λ=0\lambda = 0, fixed after review. (2)
  17. Phase 2 review. TS Mismatch not robust to the prior box; AJ tie uninformative; human κ\kappa not traits; "core-hours" and "wall 5.0 h" relabelled. (1, 5)
  18. Phase 5. The game followed the map's flagship suggestion instead of the gap-first brief; three gap-driven games started. (4)
  19. This page (website subagent). Turns cut off by an API error had been counted as busy time. (5)

Separate: bugs in the original 2013 code

These are not agent errors. The replication found several, including a gradient bug that stalled the optimiser, invalid Laplace evidence, a pilot session used most likely by mistake (the forensic verdict was about 85%), an interleaved trial order (D10, which had no effect on the published analyses because they treat trials as independent), two network gradient bugs and a learning-rate rule that went negative. See the behaviour and network audits.

The laptop restart: a compute failure caused by this project

The project runs on the first author's laptop (18 cores, 128 GB), which is shared with their other work. About four minutes after the agent relaunched the Phase-2 extension fits with six parallel workers and the monkey datasets first, the laptop restarted abruptly. The user saw more than 500 GB of memory in use. All queues died.

Cause. The nested sampler (UltraNest) asks for up to 65,536 parameter points per vectorised call by default. For the quadrature models on Monkey A (13,710 binned trial rows × 96 nodes) one such call needs an array of about 690 GB. Earlier runs had processed the humans first, one worker at a time, and had never reached the monkeys. The optimiser's screening chunks were also unbounded (tens of GB per worker). The agent had not bounded memory anywhere. This was the agent's mistake, not bad luck.

Fix.

  • Memory-bounded batching (qamar.inference.batching): every batched likelihood evaluation is sized from a 256 MB array budget, using the number of quadrature nodes each model declares, and the sampler's points per call are capped to match. Measured worst case afterwards: 9.9 GB per process.
  • A machine guard (tools/guard.py): before a heavy job starts it checks CPU, memory and thermal headroom; it keeps 25% of the cores idle machine-wide (JAX ignores thread limits and uses about 15 cores per process, so the guard throttles the job tree instead); and it kills the job if its memory blows up.
  • A hook and a rule: a pre-tool hook blocks any unguarded launch of the project's scripts, and the project instructions require the guard for every heavy job, including the agent's own subagents.

Testing the fix exposed a second bug: XLA on CPU miscompiles the vmapped quadrature likelihoods at batch size 1 (Monkey A DecNoise: −60,106 instead of −33,860). It had affected only a correction term in the nested evidence. It was fixed, with a runtime check and a regression test; all 140 nested results reported before were unaffected.

The first relaunch under the guard died on a different bug, a NaN gradient in the Sampling model at very small temperature, after finishing six monkey fits. After that fix, the second relaunch resumed from those six records and finished the extension fits 5.0 hours later (peak 58.1 GB). The extension fits therefore took about 7 hours of wall time in total.

Phase 2–3 compute

Phase 1 and Phase 2 fit runs, from the results files. "Wall clock" covers only the final run segment (the extension run's first segment, about 2 h, is not included); "summed job time" adds up each fit job's own wall time across workers, which is not CPU time. The Phase-3 network experiments took about 3.5 h of process time for E1–E6 plus 2.8 h for E7 and its control (reports/phase3_neural.md §9).

Research log

Entries from logs/RESEARCH_LOG.md, in order. Use the timeline above for timing.

Show all log entries