Agents That Never Stop Learning: Beyond the Bitter Lesson
Dwarkesh’s podcast with Rich Sutton is a fantastic listen. The conversation touches upon Sutton’s The Bitter Lesson essay, world models, generalization and evolving from building LLMs (and their derivatives) to experiential/continual learning based systems. A few call outs from the podcast and the essay that caught my attention:
- Infants learn mainly by trial and error; imitation is at most a small layer on top – this was a surprise as there’s a lot of literature in child development psychology that says otherwise.
- Learning is not about training, it is an active process!
- Search and Learning scale arbitrarily with increased computation.
- Knowledge representation and the difference between context (which, from contemporary perspective, maps to the context window of LLMs) and tacit knowledge.1
- “First there’s dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we’re giving rise to designed entities.”
- “… AIs will design other AIs, and everything will be done by design and construction rather than by replication.”
The podcast had a lot to think over. A few things:
Imitation Learning
I was quite surprised to hear Sutton’s take that imitation plays at most a small role in how infants learn. I had read otherwise based on prior work in developmental psychology. A few works are tabulated below:
| Developmental stage | Typical age | Retention | Primary mechanism | Characteristic performance | Relevant literature | Notes |
|---|---|---|---|---|---|---|
| Neonatal | 2–3 weeks (as tested) | Up to 24 hours by 6 weeks (Meltzoff & Moore, 1994) | Supramodal matching; later formalized as Active Intermodal Mapping (AIM) (Meltzoff & Moore, 1997) | Matches tongue protrusion, mouth opening, lip protrusion and finger movements. | Imitation of Facial and Manual Gestures by Human Neonates | Contested: a longitudinal study of 106 infants (Oostenbroek et al., 2016) found no specific matching at 1–9 weeks. |
| Early infancy | 6 months | 24 hours | Non-verbal encoding | Reproduces part of a 3-step sequence after 24 h; needs a doubled demonstration. | Developmental Changes in Deferred Imitation by 6- to 24-Month-Old Infants | 12- to 24-month-olds succeed with the standard demonstration. |
| Late infancy | 14 months | 1 week | Explicit declarative retrieval | Re-enacts novel, low-probability acts (head-touch) after a 1-week delay; selectively imitates rational means. | Infant Imitation After a 1-Week Delay: Long-Term Memory for Novel Acts and Multiple Stimuli Rational Imitation in Preverbal Infants | Head-touch: 67% repeated it a week later; no infant did who hadn’t seen it demonstrated. 69% copied it when the adult’s hands were free, vs 21% when her hands were busy holding a blanket. |
| Early toddlerhood | 18 to 24 months | Immediate (in cited study) | Intent reading & teleological evaluation | Extrapolates intended goals from failed attempts. | Understanding the Intentions of Others: Re-Enactment of Intended Acts by 18-Month-Old Children | 60% pulled the toy apart after watching a person try and fail, vs 10% after watching a mechanical pincer make the same slipping motions: toddlers read intent into people, not machines. |
| Preschool | 3 to 4 years | Immediate | Faithful copying of the demonstrated method | Copies causally irrelevant steps even when they are visibly useless (over-imitation). | Causal Knowledge and Imitation/Emulation Switching in Chimpanzees and Children | Chimpanzees skip the useless steps once they can see the mechanism; children don’t. |
So, where’s the disconnect? The disconnect is largely definitional. In machine learning, “imitation learning” means copying an expert’s actions as supervised targets (behavioral cloning); in developmental psychology, “imitation” mostly means inferring and reproducing goals from observed outcomes. Sutton’s own examples are of the first kind: “There’s no imitation for how they move their eyes around or even the sounds they make.”
- At the level of raw muscle contractions, eye saccades, or vocal tract adjustments, an infant receives no direct target or demonstration. An infant cannot directly observe the internal motor commands or muscle activations of an adult model. Therefore, learning how to control one’s own body (“how do I move my arm to touch my forehead?”) requires self-exploration and trial-and-error (often called “motor babbling” in robotics; Meltzoff and Moore (1997) call it “body babbling” and argue it begins before birth).
- What the empirical research suggests is that infants do not rely on brute-force trial-and-error to learn what actions to perform or what goals to achieve. Instead, they use visual input to set high-level representational targets:
- Active Intermodal Mapping (AIM): Neonates translate visual target gestures into proprioceptive motor outputs via an innate common representational space.
- Goal & Rationality Inference: Toddlers emulate the intended goal (e.g., pulling a dumbbell apart) rather than copying the failed attempts, and selectively evaluate whether an action was rational given physical constraints.
Robotics and ML reached a compatible conclusion. Schaal (1999) showed that a demonstration can speed up trial-and-error learning by shrinking the space to explore, but that imitation alone cannot improve on the teacher without an explicit reward signal. Ross, Gordon and Bagnell (2011) showed that pure behavioral cloning compounds its own errors unless the learner gathers experience in the states it actually visits: ML’s version of Sutton’s point. Meanwhile, children’s readiness to over-imitate (Horner & Whiten, 2005) is the strongest evidence that humans are unusually imitative learners.
Search, Learning and Knowledge Representation
There’s a rich body of work on each topic. The following tabulates some of the seminal works:
| Dimension | Paper | Authors · Year | Key thesis and relevance |
|---|---|---|---|
| Learning as search | Generalization as Search | Mitchell · 1982 | Learning is search through a space of hypotheses; methods differ in how they generate and prune candidates that fit the examples. |
| A Theory and Methodology of Inductive Learning | Michalski · 1983 | Induction is heuristic search over symbolic descriptions, steered by background knowledge and quality criteria. | |
| Induction of Decision Trees | Quinlan · 1986 | ID3 (Iterative Dichotomiser 3) grows a decision tree by greedily picking the most informative attribute, showing how a search bias makes learning tractable. | |
| Search & learning that scale with compute | Some Studies in Machine Learning Using the Game of Checkers | Samuel · 1959 | Look-ahead search plus an evaluation function learned through play: an early form of TD learning and the original Bitter Lesson recipe. |
| Computer Science as Empirical Inquiry: Symbols and Search | Newell & Simon · 1976 | Intelligence is heuristic search over symbol structures: problems are solved by generating and pruning candidate solutions. | |
| Bandit Based Monte-Carlo Planning | Kocsis & Szepesvári · 2006 | UCT (Upper Confidence bounds applied to Trees) guides Monte Carlo Tree Search (MCTS) by sampling instead of hand-built evaluation: the search that broke open computer Go. | |
| A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go through Self-Play | Silver et al. · 2018 | AlphaZero masters all three games from self-play alone, with search guided by a learned network: the Bitter Lesson’s showcase. | |
| Scaling Laws for Neural Language Models | Kaplan et al. · 2020 | Language-model loss falls as a power law in parameters, data and compute, so the gains from scale become predictable. | |
| Learning: foundations and mechanisms | A Theory of the Learnable | Valiant · 1984 | Introduces PAC (Probably Approximately Correct) learning: which concepts can be learned, to a given accuracy and confidence, in polynomial time. |
| Learnability and the Vapnik–Chervonenkis Dimension | Blumer, Ehrenfeucht, Haussler & Warmuth · 1989 | Finite VC (Vapnik–Chervonenkis) dimension characterizes distribution-free PAC learnability: a class’s capacity, not its size, sets how many examples are needed. | |
| Learning Representations by Back-Propagating Errors | Rumelhart, Hinton & Williams · 1986 | Backpropagation trains hidden layers, so useful internal representations emerge from errors instead of being hand-designed. | |
| Learning to Predict by the Methods of Temporal Differences | Sutton · 1988 | TD (Temporal-Difference) learning updates a prediction from the next prediction, so learning starts before the outcome arrives. Also see Sutton & Barto (2018). | |
| Learning from Delayed Rewards (PhD thesis) | Watkins · 1989 | Introduces Q-learning: learning optimal action values directly from experience, with no model of the environment. Convergence proved in Watkins & Dayan (1992). | |
| Knowledge representation | Programs with Common Sense | McCarthy · 1959 | The Advice Taker represents facts as logical sentences and reasons over them: the founding paper of knowledge representation. |
| A Framework for Representing Knowledge | Minsky · 1974/1975 | Frames package slots, defaults and expectations for familiar situations; understanding means selecting a frame and adapting it. | |
| A Logic for Default Reasoning | Reiter · 1980 | Default rules let a system draw conclusions from incomplete knowledge and withdraw them when contrary evidence appears. | |
| The Symbol Grounding Problem | Harnad · 1990 | Symbols defined only by other symbols mean nothing to the system; meaning must bottom out in sensorimotor experience. A standard lens on whether text-trained models “understand”. | |
| Intelligence without Representation | Brooks · 1991 | Robots built from layered behaviors coupled directly to the world need no central world model: the embodied view closest to Polanyi and Moravec. | |
| A Translation Approach to Portable Ontology Specifications | Gruber · 1993 | Ontologies, as explicit shared vocabularies, let knowledge be reused across systems and representation languages. | |
| What Is a Knowledge Representation? | Davis, Shrobe & Szolovits · 1993 | A representation plays five roles: surrogate for reality, set of ontological commitments, theory of reasoning, medium for computation, and medium of human expression. |
Tacit Knowledge
Polanyi’s ideas directly intersect with a famous AI principle called Moravec’s Paradox. As Moravec put it in Mind Children (1988): “it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”
| Task type | Human effort | AI effort | Polanyi connection |
|---|---|---|---|
| Advanced logic (e.g., calculus, chess, data analytics) | Hard (requires conscious, high-level school learning) | Easy (the rules are explicit, and computers win through search and computation rather than codified human expertise) | Explicit knowledge: easily written down and turned into algorithms. |
| Sensorimotor skills (e.g., walking, avoiding obstacles, reading micro-expressions) | Easy (instinctive, effortless to a toddler) | Hard (requires immense compute and complex robotics) | Tacit knowledge: evolved over millions of years; we possess no internal conscious script of how we do it. |
Since around 2012 the paradox has partly eroded. Deep networks trained on large datasets now see and hear well, learning perceptual skills from data rather than from rules anyone could write down: the Bitter Lesson at work. Dexterous manipulation and robust real-world mobility remain hard, largely because physical-interaction data is scarce and slow to collect.
For transferring a person’s tacit expertise, the most direct starting points are behavioral cloning, inverse RL (Reinforcement Learning) / reward learning, and preference learning. Example works in each are tabulated below:
| Category | Paper | Authors · Year | Key contribution and connection to tacit knowledge |
|---|---|---|---|
| Behavioral cloning & direct imitation: copying expert actions | ALVINN: An Autonomous Land Vehicle in a Neural Network | Pomerleau · 1988 | Learns to steer by mapping camera images to a human driver’s steering. The driver never states a rule; the network absorbs the skill. Simple and direct, but brittle once it drifts into situations the driver never showed. |
| Generative Adversarial Imitation Learning | Ho & Ermon · 2016 | Trains a policy until a discriminator can’t tell it apart from the expert. It skips recovering a reward and copies behavior far more faithfully than plain cloning. | |
| Inverse RL / reward learning: inferring objectives from expert behavior | Algorithms for Inverse Reinforcement Learning | Ng & Russell · 2000 | Infers the reward under which an expert’s behavior makes sense, so experts never have to articulate their objectives. It also exposes an ambiguity: many rewards explain the same behavior. |
| Apprenticeship Learning via Inverse Reinforcement Learning | Abbeel & Ng · 2004 | Matches the expert’s average feature counts instead of recovering their exact reward. Apprenticeship succeeds without knowing precisely what the expert values. | |
| Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization | Finn, Levine & Abbeel · 2016 | Learns a neural cost from demonstrations while learning to act under unknown dynamics, refining it with the robot’s own experience. No hand-engineered features or dynamics model needed. | |
| Preference learning: learning from human judgments | Deep Reinforcement Learning from Human Preferences | Christiano et al. · 2017 | Learns a reward from people’s comparisons of short behavior clips, then optimizes it with RL. It transfers judgments people can make more easily than explain. |
| Learning to Summarize from Human Feedback | Stiennon et al. · 2020 | Brings preference-based reward learning to summarization, capturing judgments that text-matching metrics miss. Optimizing the learned reward too hard, though, lowers real human-rated quality. |
Note the tension with Sutton: every method here learns from human-provided targets (demonstrations or judgments), which is exactly what he argues cannot be the foundation of intelligence (Silver & Sutton, 2025).
World Models and Experiential Learning
Which brings us back to Sutton’s bet: intelligence comes from an agent’s own experience, not from human data. The lineage is older than LLMs:
- Dyna (Sutton, 1991): learn a model of the world from experience, then plan with it.
- MuZero (Schrittwieser et al., 2020): a learned model plus search mastered Atari, Go, chess and shogi without being told the rules.
- Loss of plasticity (Dohare et al., 2024): standard deep networks gradually lose the ability to learn, which is the core obstacle to truly continual learning.
- The Era of Experience (Silver & Sutton, 2025): the manifesto for agents that learn from streams of their own experience, grounded in real-world signals.
Sutton doubts LLMs have world models at all; Dwarkesh thinks they do. Either way, the next frontier isn’t a bigger context window. It’s agents that never stop learning.
Notes
- 1 In The Tacit Dimension (1966), Polanyi famously summarized the tacit dimension as “we can know more than we can tell”. LLMs may well have a tacit dimension of their own (they, too, know more than they can tell), but they lack the kind humans have: rooted in evolution, grounded in real-world interaction, and continually acquired through experience. ↩︎