Self-Improvement as a Bilevel Program
Abstract
A self-improving agent updates itself using evidence from its own trajectories. The cleanest mathematical envelope for that idea is bilevel optimization: an inner adaptation map and an outer objective that scores the adapted agent. We restate the model-agnostic meta-learning (MAML) updates of Finn, Abbeel, and Levine (2017) and contrast them with verbal reinforcement loops such as Reflexion (Shinn, Cassano, Berman, Gopinath, Narasimhan, and Yao, 2023) and Self-Refine (Madaan et al., 2023). The correspondence is structural, not algorithmic identity. This editorial primer is an orientation, not a peer-reviewed research claim.
1. Two timescales
Write $\theta$ for the parameters — or, more loosely, the editable state — of an agent: weights, a memory store, a prompt prefix, a tool policy. A task $\mathcal{T}$ supplies a loss $\mathcal{L}_{\mathcal{T}}$. An inner update produces an adapted state
and an outer objective scores the adapted agent on a held-out draw from the same task, or on a later episode:
This is a bilevel program. The inner map $U$ may be a gradient step, a prompt rewrite, or a memory write. The outer expectation is the meta-objective: we want $\theta$ that become good after adaptation, not $\theta$ that are already good on a frozen training mix.
2. The MAML step
Finn, Abbeel, and Levine (2017) take $U$ to be one (or a few) gradient steps of a differentiable model $f_\theta$:
The meta-update is then
The outer gradient differentiates through the inner step. When $U$ is a single gradient step the computation is a Hessian-vector product; the 2017 paper discusses first-order approximations that drop that term.
Proposition 1 (Finn, Abbeel, and Levine, locally restated). If $f_\theta$ is differentiable, each $\mathcal{L}_{\mathcal{T}_i}$ is differentiable, and $U$ is a gradient step of size $\alpha$, then the outer objective is a differentiable function of $\theta$ and the displayed meta-update is gradient descent on that objective. No claim is made here about global convergence, or about non-differentiable language-model agents.
We do not reproduce the experimental sections of the 2017 paper. Few-shot classification and policy adaptation are their evidence, not ours.
3. Verbal loops are not gradient MAML
Reflexion stores an episode, asks the same model to write a natural-language diagnosis, and prepends that text on the next attempt. Self-Refine iterates “generate / critique / revise” in language. Neither loop computes $\nabla_\theta \mathcal{L}$. There is no $\alpha$, no Hessian, and no guarantee that the outer score decreases.
A responsible analogy is:
- the inner map $U$ is “write a reflection $r$ and form $\theta' = (\theta, r)$”;
- the outer score is a later success indicator $G \in \{0,1\}$, or a unit-test count;
- the meta-state $\theta$ is the frozen base model plus whatever durable memory the system keeps.
Under that reading, Reflexion is a stochastic, non-differentiable bilevel heuristic. It can be evaluated. It cannot inherit MAML’s differentiation argument. Papers that write “the agent meta-learns” without an inner map, an outer objective, and a held-out task distribution are using the word as decoration.
Chain-of-thought prompting (Wei et al., 2022) is not a self-improvement loop at all: the weights and the prompt template are fixed. It belongs in a different citation cluster.
4. A minimal evaluation protocol
Let $\mathcal{D}_{\mathrm{train}}$ and $\mathcal{D}_{\mathrm{test}}$ be disjoint task sets. A self-improvement claim should report:
1. the inner budget (attempts, tokens, gradient steps); 2. the outer score on $\mathcal{D}_{\mathrm{test}}$ after adaptation, against a no-adaptation baseline on the same test tasks; 3. whether test tasks were visible to the reflection prompt (leakage); 4. variance across random seeds of the inner sampler.
A single before/after anecdote on one programming puzzle is not a meta-learning result. If the outer score is a learned reward model, say so; circular self-scoring is a known failure mode and is not solved by citing MAML.
5. Credit assignment
Even in the differentiable case, the outer gradient attributes credit to $\theta$ through $U$. In a verbal loop the attribution is implicit: the reflection text that happened to precede a success is kept. That is closer to a one-step evolutionary strategy than to policy gradient. Sutton and Barto (2018, ch. 13) develop the policy-gradient theorem
for stochastic policies. A language-model agent that samples tokens has such a $\pi_\theta$, but the systems we are discussing do not estimate this gradient. We record the theorem so that a later paper can say honestly whether it estimates it.
6. Limitations
We have given a vocabulary, restated a published differentiable update, and refused an identification between verbal refinement and MAML. We have not proposed a new algorithm, have not measured Reflexion, and have not proved convergence for prompt-memory systems. Non-stationary task distributions, sparse binary rewards, and frozen API models all sit outside Proposition 1.
References
1. Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1126–1135, 2017. 2. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 2023. 3. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 2023. 4. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022. 5. Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. Chapter 13.
This original editorial primer is an orientation, not a peer-reviewed research claim.