Affective Reinforcement Learning

1. Introduction

An RL agent has exactly one way to say that something went well: the scalar it receives from the environment [33]. Silver, Singh, Precup and Sutton's reward-is-enough hypothesis [30] takes that channel to be sufficient in principle — that everything worth calling intelligence, feeling included, can emerge from maximizing one sufficiently rich signal. This proposal is a bet against relying on that emergence, not a claim that the hypothesis is false: everything the agent will ever come to prefer has to be squeezed through that one channel unless something else is built in. A person has at least two. The exam result and the feeling about the exam result are different quantities, they can point in opposite directions, and the second one is frequently the one that determines whether the person sits the next exam.

This matters most where the two diverge. An agent that has repeatedly been punished while exploring does not merely learn that those particular actions were bad — it becomes, in ordinary language, reluctant. The reluctance generalizes past the evidence. It suppresses actions that were never tried. That failure mode has a name outside of machine learning, and I want to know whether it can be built.

The proposal is an architecture in which feeling is a second evaluative channel with its own policy, and in which the balance between the two channels is set by the feeling itself. What follows is the plan for building it and the conditions under which I would accept that it does not work.

Contributions

• A dual-policy architecture in which affect is a first-class internal state, with an explicit appraisal function and affect dynamics (§4.1–4.3), rather than a scalar bonus folded into the reward.
• An affect-conditioned arbitration weight β(e) (§4.5) — the one component that makes this something other than scalarized multi-objective RL — together with the argument for why (§4.6), the part of that argument that does not work, and a statement of the circularity in training β at all (§4.5.1).
• A specification given as numbers rather than prose — the events, the rates, and where the internal state settles under each of them (§6.1–6.2), in two environments with very different reward structure.
• Five preregistered, falsifiable predictions (§7), each paired with the ablation that would explain it away.
• A statement of what will and will not be released, including the parts I cannot release (§11).

2. Background and related work

2.1 Emotion in reinforcement learning

The idea of giving an RL agent something like emotion is not new, and Moerland, Broekens and Jonker [19] survey it thoroughly. They separate emotions read off the agent's own learning signals — temporal-difference error as joy or distress, value estimates as hope or fear — from emotions grounded in appraisal over the agent's model of the world. Most work in the first category treats emotion as a readout: a quantity computed from learning internals, sometimes fed back as an exploration heuristic, and usually with no policy of its own. The architecture here sits in the second category and goes further, in that the affect has an optimization target of its own and can win the argument.

2.2 Appraisal theory

The psychological claim that emotions are evaluations of events relative to goals, rather than reactions to events as such, is due largely to Lazarus [17] and is formalized computationally in the OCC model [21] and in EMA [18]. Frijda's account of emotions as action readiness [13] is the closest to what I need: an emotion is not a feeling that then causes behavior, it is a shift in which actions are available and attractive. Damasio's somatic marker hypothesis [8] supplies the argument for why it matters — that affect is not noise on top of reasoning but a mechanism that makes deciding possible at all, by cutting the options down. That hypothesis is not settled, and I should not lean on it harder than the evidence allows: Dunn, Dalgleish and Lawrence [11] take apart both the novelty of the claim and the Iowa gambling task results usually offered for it. What I borrow is the functional shape — affect as a pruning mechanism — not the physiological story. Spinoza [32] got to the structural account first: the affects are increases and decreases in a thing's power to persist, which is very nearly a statement about the derivative of a value function.

2.3 How much structure the internal state needs

One natural design gives the agent four bipolar axes after Plutchik [24] — joy–sadness, trust–disgust, fear–anger, surprise–anticipation — on the grounds that discrete labels are interpretable and that the point is to ask which specific emotion does what. I use one number in [−1, 1] instead, and the reason is worth stating because it is an argument against that design.

Four labelled axes require a mapping from environment events to emotions, and that mapping has to be written by hand. It is not measured, not learned, and not obviously right — so every result would have been partly a result about my assumptions, which §9 would have had to concede outright. Russell's circumplex [28] is better supported as a description of what people report, and Barrett [6] argues that discrete basic emotions are not natural kinds at all. Against that background, choosing four named axes was claiming more structure than the evidence supports and more than the question needs.

The question is whether how an agent is doing changes how it learns. One number answers it, moved by events that are facts about the task rather than judgements about feeling. If the effect exists at this resolution it is worth adding structure back later, and §9 says what that would take. If it does not exist at this resolution, four axes would not have rescued it.

2.4 Intrinsic motivation and homeostatic RL

Intrinsically motivated RL [31] already gives agents reward signals not supplied by the environment. Curiosity-driven exploration via prediction error [22] and random network distillation [7] are the strongest members of the family.

The closest prior formalism, and the one this proposal has to be measured against, is homeostatic reinforcement learning. Keramati and Gutkin [15, 16] define reward as the reduction in deviation between an internal state and a setpoint, and prove that reward maximization and physiological stability coincide under that definition. That is the same move I am making in equation (2) — an internal state, and a reward read off its change — arrived at fifteen years earlier and with theorems attached. I should be explicit about what is left over once that is acknowledged. Their internal state is a physiological vector with a fixed setpoint and a drive function that is monotone in distance from it; mine is an appraisal-driven affect vector with no setpoint, whose state is a single number with no setpoint of its own beyond the disposition ē, and no notion of distance from a physiological ideal. More importantly, homeostatic RL folds the internal signal into one reward and solves the resulting single-objective problem, which is exactly the no-arbitration ablation in §6.3. The difference this paper is proposing lives entirely in equation (3), not in equation (2), and any result that does not beat the homeostatic framing is a result about homeostatic RL.

2.5 Multi-objective and constrained RL

An agent balancing task reward against a second signal is a multi-objective problem, and the standard treatment is scalarization: reduce the vector reward to a scalar by a weight vector and solve the resulting single-objective problem [26]. This is the sharpest threat to the whole proposal, and §4.6 addresses it directly. Two results bound what that answer can look like. Vamplew et al. [36] show that linear scalarization cannot reach policies in concave regions of the Pareto front at all, so “not linearly scalarizable” is a real property and not a rhetorical one. Abel et al. [1] give the sharper statement — there are tasks no Markov reward function expresses — and it is the theorem §4.6 has to engage with rather than talk around. Vamplew's result is specifically about linear scalarization; Roijers and Whiteson's treatment of utility-based, non-linear scalarization [27] narrows that particular gap — a monotonic non-linear utility function can reach concave-front policies a linear weight vector cannot — without touching Abel et al.'s point, which is about expressivity of the reward itself and holds regardless of how the scalarization is shaped.

The constrained framing deserves more than the mention its place in this heading implies. A constrained MDP [5] maximizes return subject to a bound on a second cost, which is a much closer description of “a channel that can override the optimizer” than a weighted sum is, and it comes with duality results that the architecture in §4 does not have. Altman's is the classical, tabular-era treatment; the deep-RL operationalization — trust-region updates that respect the constraint at every step[3], and Lagrangian relaxations that learn the multiplier alongside the policy [35] — is a generation newer and is validated at deep-RL scale on continuous-control tasks with neural policies [25], not the tabular MDPs Altman's duality results were proved for. The homepage's conscience formula, max E[R] s.t. h(s, a) ≥ 0, is a constrained MDP in this newer sense. I am not adopting it, and the reason is a commitment rather than a technical objection: a constraint is a bound the designer sets, and the question here is what happens when the second channel is learned and is allowed to be wrong. That is a worse engineering decision and a better psychological model, and it is worth saying which of the two I am optimizing for.

Separately, Ng, Harada and Russell [20] showed that a shaping term preserves the optimal policy only if it is potential-based, and it is worth being exact about where the affective reward sits with respect to that theorem, because it is closer to it than I first assumed. Their proof is for a static potential Φ(s); the route that would be needed if e stayed outside the state is Devlin and Kudenko's extension to a potential that changes during learning [10], but that route is not the one taken here — folding e into the state keeps Φ(s, e) static in the augmented space, which is why Ng et al.'s original theorem is the one that actually applies. Take Φ(s, e) = e as a potential over the augmented state. Equation (2) then decomposes as

At γ = 1 the residual is exactly zero, so r_int has theform of a potential-based shaping term. It does not follow that the optimum is unchanged. Ng et al.'s invariance result requires Φ(terminal) = 0 in the episodic case, and here Φ(terminal) = e_T, which is whatever state the agent finished in. §4.3 makes the same point from the other side: the undiscounted return collapses to e_T − e₀, and a shaping term with non-zero total is not policy-preserving. So even at γ = 1 this is a genuine terminal-affect bonus that moves the optimum.

At γ < 1 the residual is a standing bonus for sitting in a good state. It is the wrong unit and the wrong conclusion to dismiss this as “about 1% of affect magnitude at γ = 0.99”: a per-step reward of (1 − γ)·e has discounted total (1 − γ)·e·(1/(1 − γ)) = e. Its cumulative value is the same order as the potential term it was being contrasted with, not a hundredth of it. The honest consequence is that the scalarized control also has a full-strength standing incentive to park in a high-valence state, so the comfort zone is not something only the full model can produce. That is what the no-arbitration ablation is for, and the theoretical contrast I claimed here is not established in advance.

2.6 Dual-system arbitration

The architectural precedent is Daw, Niv and Dayan's account of competition between a model-based and a model-free controller [9], in which arbitration follows each system's uncertainty so that whichever is currently more reliable takes control. I borrow the shape of that solution and change what drives it: here arbitration is a function of affect rather than of uncertainty. The parallel is worth stating because it makes a prediction — an affect-arbitrated agent should show the same signature of habitual takeover under stress that the uncertainty account predicts under time pressure.

3. Preliminaries and notation

The task is a Markov decision process (S, A, P, r, γ) in the usual sense [33]. One detail matters for §6.1: the Rubik's cube environment shows one face at a time, so a single observation does not determine the state. It is worth being exact about what that does and does not mean. The agent has rotation actions, so it can turn the cube and look — the rest of the state is always reachable by acting, and nothing about it is hidden permanently or revealed only by chance. This is an active-perception problem rather than a memory-limited one: the cost of not knowing is measured in moves, not in irrecoverable uncertainty. An observation history encoder still carries what has been seen, and I am not claiming the agent is handed the full cube for free.

4. Method

4.1 The affective state

The agent carries a single number, e ∈ [−1, 1]. −1 is as bad as things get and +1 as good; there are no named emotions and no axes. It is not part of the environment — the environment neither reads it nor rewards it.

One number rather than four labelled axes is a deliberate cut, and §2.3 argues it: the question is whether how an agent is doing changes how it learns, and that does not need a theory of emotion to ask.

Each agent also has a disposition ē, the value e settles back toward. This is the experimental manipulation: ē is set before training and never learned, and the three arms in §6.3 are ē = −0.5, ē = +0.5, and no internal state at all.

4.2 Affect dynamics

Events move e, and between events it drifts back toward the agent's disposition:

It pulls back toward ē, not toward zero, and that difference is the whole experiment. With decay toward zero a disposition set before training washes out within about twenty steps, so “a negative agent” would stop being negative long before it had learned anything and the comparison would measure nothing. Pulling toward ē makes the disposition permanent by construction: with no events at all, e converges on ē and stays there.

The two rates do different jobs. λ = 0.010 gives a half-life of about 69 steps, so the disposition asserts itself over an episode rather than over a few moves. κ = 0.020 means a partial win shifts e by 0.04, so recent events dominate the short run and the disposition dominates the long run — which is what a mood sitting on top of a temperament should look like.

4.3 The internal reward

The affective policy needs a reward, and the natural choice is not affect itself but its rate of change: feeling good is a state, feeling better is an event. This is not an invention of mine and it has empirical support I should be citing: Eldar, Rutledge, Dolan and Niv [12] argue that mood represents the momentum of recent outcomes — how they differ from expectation — and that it then biases the perception of subsequent outcomes. Rutledge et al. [29] give the fitted human version — momentary happiness explained by recent expectations and prediction errors rather than by cumulative earnings — which is the closest empirical counterpart equation (2) has. Equation (2) is the first half of that account written as a reward, and §4.2's pull-back is what makes it momentum rather than a total.

This holds on every non-terminal transition. The terminal transition is a documented exception — see §2.5 — where r_int is −e rather than e′ − e, to keep the affective potential at zero when the episode ends.

With one number this is as simple as it gets, and the simplicity is worth noting rather than passing over: there is nothing left here to get the sign of. A weighted sum over four axes, with one of them entering by magnitude because both its poles were bad, is machinery that exists only to hold the emotion labels together — and the sign of each term is something to get wrong.

This has the consequence that a permanently miserable agent is not permanently punished — it is punished for getting worse. That is deliberate, and it is what stops the internal channel from swamping the external one on long horizons.

The definition telescopes, and two things follow that are worth stating before they turn up as objections. First, the undiscounted return of r_int over an episode collapses to e_T − e₀ — the whole trajectory in between cancels. π_int's objective is therefore something close to “end in a good mood”, and it is insensitive to how long that takes. That is a sharper problem than it looks, and §2.5 works through it: it is simultaneously why the γ = 1 shaping term is not policy-preserving and where the real reward hack lives.

Second, and for the same reason, an undiscounted agent cannot farm affect by cycling — driving e up and letting it decay back nets exactly zero. Under discounting it can, weakly: a cycle that rises early and falls late keeps the difference in discount weight. Note this is the less dangerous exploit of the two, and §6.2's worry about a repeated positive trigger is weaker still, because in steady state decay charges back exactly what the pump earns. The terminal bonus is the one that pays.

4.4 Two policies

π_ext is trained on r by any standard on-policy method (PPO in the planned implementation). π_int is trained identically on r_int. Both receive the concatenated input (s, e): the task policy can see the agent's mood, which is necessary if the compromise is to be coherent rather than two systems talking past each other.

This is off-policy and PPO is not, which is a known approximation rather than an oversight. Trajectories are collected under the mixture π of equation (3), but each component's PPO ratio is computed against its own previous parameters — π_ext updates as though π_ext had generated the data, and it did not. The correct object is an importance weight of the component against the mixture, and the trust region PPO clips is not the right region for it. I am not applying that correction in v1.0, for the ordinary reason that the corrected weights have high variance when β sits near the middle of its range, which is exactly where the interesting runs will be. What I will do instead is log the per-batch ratio between each component and the mixture, so the size of the approximation is recorded rather than assumed small — if those ratios are far from 1, every learning curve in v1.0 is measuring a biased update and I would rather find that in the logs than in review.

4.5 Arbitration

Action selection mixes the two policies in log space, with a weight that is a function of the current affective state. The multiplicative form is borrowed rather than invented — it is the composition operator used by Haarnoja et al. [14] for composing soft-optimal policies, by Peng et al. [23] for combining motor primitives, and analyzed by van Niekerk et al. [37], who prove when composed value functions are exactly optimal. What is new here is only that the exponent is a function of the agent's own internal state; §4.6 argues that this is the part that matters, and §4.6 is also where that literature bites back.

β responds to how bad things are and to nothing else. The max(0, −e) term is zero whenever e ≥ 0, so an agent that is doing fine is task-driven no matter how well it is doing, and only a negative state pulls weight toward the internal policy — which has learned that the reliable way to avoid getting worse is to do the familiar thing.

That asymmetry is deliberate. Feeding β a feature vector that includes the magnitude ‖e‖ would be blind to sign, so a happyagent would come out more internally-driven than a neutral one — the opposite of the intended behavior. One feature that can only ever be triggered by bad news has no room for that mistake.

“Handover” would be the wrong word and I have stopped using it. β = σ(·) is bounded in the open interval (0, 1), so it never reaches either end: log π_ext keeps a non-zero coefficient at every timestep, and any action π_ext assigns near-zero probability stays near-zero in the mixture no matter how negative the affect gets. π_ext's support always binds. What equation (3) implements is continuous reweighting, not a transfer of control, and the difference matters for §4.7 — a comfort zone produced by reweighting is a shift in a distribution the task policy still constrains, not a different agent taking over. The dual-system precedent in §2.6 does describe genuine handover, and that is a real disanalogy rather than a presentational one.

4.5.1 What β is trained against

Equation (4) says β is learned. It does not say against what, and that omission is not a detail — β(e) is the paper's stated contribution, so leaving its objective unspecified leaves the contribution unspecified. Worse, the choice is not neutral. Every candidate loss trivializes the architecture in a different direction:

• Train β on task reward. The task-optimal thing to do with a second policy that optimizes something else is to ignore it, so β should be driven toward 0 and the affective channel should disappear.
• Train β on r_int. Symmetrically, β → 1 and the agent stops doing the task.
• Train β on a fixed combination of the two. Then the weights of that combination are the real arbitration parameters, they are constants, and this is scalarization moved up one level and hidden behind an extra sigmoid.

There is no fourth option that I can see, and I do not think the circularity is an accident of this design: an arbitrator between two objectives cannot be trained without an objective, and any objective given to it is a third preference the designer has smuggled in. This is the weakest joint in the paper and I would rather point at it than let a reviewer find it.

What v1.0 will actually run, preregistered so that the outcome can embarrass me: β's parameters (w₀, w) are updated by PPO on thetask return of the mixture, treating β as part of the behavior policy. That is option one, chosen because it is the only one whose failure is informative — and I am predicting it fails, in the specific sense that mean β declines monotonically toward its floor over training. Recording that collapse is worth more than avoiding it: it would be direct evidence that affect-conditioned arbitration does not survive contact with task-reward pressure, which is a real finding about the architecture rather than a tuning failure.

Because the headline claim cannot be tested by a component that has collapsed, the frozen-β sweep in §6.3 is not a secondary control here — it is the fallback primary. β is held at each of 0.1, 0.25, 0.5, 0.75, 0.9 for a full run, and if some fixed value reproduces the H2 attractor while the learned β does not, the honest conclusion is that a constant affective weight is sufficient and the conditioning on e adds nothing. That would falsify the contribution while leaving the architecture standing, and it is the result I consider most likely.

4.6 Why this is not scalarized multi-objective RL

This needs saying plainly, because the usual reading of “weighted compromise between two objectives” is scalarization, and scalarization with a fixed weight is exactly equivalent to single-objective RL on the combined reward [26]. If β were a constant, this architecture would be an elaborate way of writing r + c·r_int, and there would be no paper here.

A tempting argument runs: β depends on e, e is a function of history, therefore the effective objective is non-stationary and no fixed weight vector reproduces it. That does not work. e′ is a deterministic function of the transition, so (s, e) is Markov and the process is perfectly stationary in the augmented state. Worse for the argument: a state-dependent weight applied to rewards, giving (1 − β(e))·r + β(e)·r_int, is still an ordinary Markov reward function over (s, e), and an agent maximizing it is doing single-objective RL. State augmentation dissolves the objection, and any version of this claim that ignores it is not worth making.

A second argument holds that two things survive: that no fixed linear weight over S reproduces the behavior, and that log-linear mixing is a “conjunctive veto” with no reward-space equivalent. Neither holds up, and it is worth saying why before saying what does.

• “No fixed weight over S” is true and nearly vacuous. Any behavior depending on history fails to be a function of s alone; that is a property of carrying an internal state at all, not of β(e). The citations do not rescue it. Vamplew et al. [36] concerns linear scalarization, and the standard answer in that literature is non-linear scalarization, which reaches concave fronts fine. And Abel et al. [1] is about expressivity over a given state space, whose canonical remedy is exactly the state augmentation I just conceded — I was citing the theorem that defeats the argument as though it supported it.
• There is no veto. Both policies are softmax heads over a finite action set, so no probability is ever exactly zero; §4.5 already says β never reaches 0 or 1. What equation (3) performs is a product of experts, which reweights and never forbids. “Conjunctive veto” was a hard-support claim resting on soft-support machinery, and it contradicted a paragraph four hundred words earlier in the same document.

What equation (3) actually is, in one reading: it is the closed form of a KL-regularized objective — maximize r_int while staying close in KL to a behavior prior π_ext, with β setting the temperature [34]. The dual reading maximizes the task reward against an internal-policy prior, and under the composition theorems the two coincide in entropy-regularized RL with deterministic dynamics — which the caveat below turns out to matter for. That is a well-understood object, and it means the conclusion that the composite “optimizes nothing” is true only of the training procedure. The architecture has a perfectly good variational characterization. A state-dependent β makes it a state-dependent-temperature KL-regularized objective, which is still an objective.

And state-conditioned weights are not new. Abels et al. [2] condition a multi-objective network on the relative importance of objectives and vary it during training; the dynamic-weight MORL line goes back further still. §4.5 said “what is new here is only that the exponent is a function of the agent's own internal state” — the honest correction is that context-conditioned scalarization weights are a two-decade-old subfield, and what is unusual here is only which variable does the conditioning: an affect state the agent generates itself, rather than a preference supplied from outside.

So the defensible claim is narrower than a section titled “why this is not scalarized MORL” implies, and I have left the title standing rather than quietly softening it. It is: equation (3) is a KL-regularized policy with a state-dependent temperature; that is not a fixed linear scalarization of r and r_int; and whether the difference produces any behavior worth having is an empirical question rather than a theorem. Two further caveats belong here rather than in a footnote. Composition is provably exact in entropy-regularized RL under deterministic dynamics [37, 14] — and a Rubik's cube is deterministic, so in the primary environment the equivalence I flagged as a distant risk is the default case. And the components are trained separately, so any gap from the scalarized optimum may be a fact about mistraining rather than about expressivity.

Two ablations decide this rather than argument. The no-arbitration condition in §6.3 is the scalarized version of the architecture, and the frozen-β condition is the same dual-policy mixture with the state dependence removed. If frozen β matches the full model, β(e) contributes nothing and the headline claim is empty. If no-arbitration also matches, the whole architecture is. I will say so either way.

4.7 The comfort zone as a fixed point

The claim is specifically about an agent held down by negative affect, not about one trading task reward for feeling good. Those are different phenomena with opposite β signatures, and H2 has to measure the first, not the second. The loop: negative affect raises β; a higher β reweights action selection toward a policy whose whole objective is to avoid negative appraisal; the reliable way to avoid a negative appraisal is to do the thing that has not produced one before; doing the familiar thing withholds exactly the transitions whose appraisal would revise the affect; the affect stays negative and β stays high. If that loop closes it is a fixed point of the joint dynamics over policy and affect — stable, and measurably worse on task reward than the agent could do. H2 in §7 now tests that agent: below baseline on task reward and below it on valence, with β elevated.

That loop assumes avoidance dominates pursuit, and I have not derived that from the objective — I should say so rather than let it pass. φ's own ratio is 1 : 2 : 10 for no-win : partial-win : full-win, so a reward-maximizing π_int has every incentive to chase the large positive events, not just dodge the small negative one. The more defensible reading is narrower than the loop as stated: an agent stuck at a negative disposition has never sampled a win, so it has nothing to reinforce toward one, and falls back to whatever has not produced a negative appraisal — ordinary sparse-reward credit-assignment lag, not a general preference for caution over reward. That is an assumption, not a proof, and it is the entropy-matched and no-arbitration ablations that decide whether it holds, not this paragraph.

§2.5 is what makes this an argument about arbitration rather than about shaping — but it does not rule the shaping route out, and I should not pretend otherwise. The residual is worth e cumulatively, not a hundredth of it, so a plain scalarized agent has its own standing incentive to park somewhere pleasant. The distinctive claim is therefore not “only the full model can produce a comfort zone” but “the full model produces one with a characteristic β signature that the scalarized control cannot show, because it has no β”. That is thinner, and it is what the no-arbitration ablation actually decides.

The decay term is the obvious threat to it, and I would rather name it now than discover it in the runs. At e < 0 with the agent doing nothing, decay pulls e toward ē, and rising affect is positive r_int. Passivity pays until e ≈ ē = −0.5, at which point β has fallen partway — to about 0.49, not all the way — and π_ext takes back more of the weight and explores, the exploration is punished, and the cycle restarts. That would make the object an oscillation rather than a fixed point — arguably the more interesting outcome, and one a bare “sits below baseline and stays there” could not tell apart from the attractor. §7 therefore splits the question: H2 measures whether the agent settles somewhere worse than it needs to be, and H2b measures whether it settles at all, via the rolling variance of valence and the affect autocorrelation. A limit cycle is recorded as its own positive result rather than as a failed H2 — but it is not what H2 claims, and keeping the measurements separate is what stops admitting both outcomes from making the prediction unfalsifiable.

5. Psychological grounding of the design choices

Two commitments in §4 are psychological claims rather than engineering ones, and each could have been made differently.

A mood on top of a temperament. The state has both a fast part that events move and a slow part it returns to, which is the standard shape in the mood literature and the reason equation (1) pulls toward ē rather than toward zero. It also carries the commitment that a disposition is something an agent has rather than something it earns: ē is set from outside and never learned. That is exactly what makes the three arms in §6.3 a manipulation rather than an observation, and it is also the part with the least support — real temperament is not assigned by an experimenter.

The state gates action rather than reporting on it. This is Frijda's action readiness [13] and Damasio's somatic markers [8], hedged as in §2.2: the function is to narrow the options before deliberation, not to comment afterward. It is the specific commitment most of the emotion-in-RL literature does not make [19], and it is why the state gets its own policy in §4.4 instead of being another number in the observation.

A real loss sits here: appraisal is computed against absolute events, not relative to expectation, after Lazarus [17] and EMA [18]. The events in §6.2 are absolute — a face is complete or it is not. Missing a goal you expected to reach and missing one you never expected to reach are different experiences, and this model cannot tell them apart.

6. Experimental design

6.1 Environments

Two training environments, chosen because their reward structure differs about as much as it can, and a third held out entirely for the transfer test in H5. All three run on the same engine, and humans and agents drive the identical interface, so human trajectories are available as a comparison set.

• Rubik's cube (sparse, cognitive). One face is visible at a time, and rotation actions let the agent see the rest, so looking around costs moves rather than being impossible (§3). Reward is sparse and sub-goal structured. This is where the internal state has room to help.
• Grand Prix (dense, reflexive). A racing circuit where the whole game is knowing when to brake. Reward is dense and already well shaped. This is where affect should get in the way.
• Third environment (held out). Not disclosed here and not touched until the transfer test, so that the architecture cannot be tuned against it.

6.2 Affect triggers

φ is hand-specified per environment. This is a limitation — the mapping encodes my assumptions about what should feel bad — and §9 says what would be needed to learn it instead.

Δ is what the event does to e before κ scales it; simultaneous events sum, and equation (1) clips the result. Anything not listed moves e not at all. The ratio 1 : 2 : 10 is the whole of φ — losing ground hurts a little, progress helps twice as much, and winning helps ten times as much. The first two rows share one resetting 5-step window: each window closes as either a no-win or a partial win, never both, and the count restarts — so “doing well” means winning every window, not winning at some rate independent of the losing rate.

Note what these events are: facts about the task. Whether a face is complete, whether the car is on the track, whether the lap beat the target. Nowhere does this table say what should feel bad. Mapping events onto named emotions would make every result partly a result about my assumptions, which is exactly what this avoids. The remaining constants are below, so that φ and everything downstream of it can be re-instantiated from this page rather than inferred from it.

Constants alone are not a specification — an internally consistent table can still be wrong, if β is checked at a hand-picked state that neither environment can reach. What follows is where e settles under a sustained rate of events, e* = ē + κφ̄/λ, for each of the three dispositions, and β there. This is the table that has to be right, and it is generated by a script rather than written by hand.

Two properties make the experiment work. A positive agent sits at β ≈ 0.15 in every regime, including when it is stuck — so it stays task-driven, and the prediction that it will behave much like an agent with no internal state at all is a real prediction rather than something the parameters force. A negative agent that gets stuck reaches β ≈ 0.78, so the internal channel actually takes the weight §4.7 needs it to take. The gap between them when stuck is 0.63, which is what makes H1 measurable.

Anyone changing λ, κ, ē, w₀ or w₁ must regenerate this table with node scripts/calibrate-affect.js rather than reason about the sigmoid in isolation. That script also checks the table against the constants and fails if they have drifted apart. Without that check, a β can be capped far below its stated range in one environment and rise with intensity regardless of whether things are going well or badly.

6.3 Baselines and ablations

The first three rows are the experiment: identical agents that differ only in ē. The rest are rival explanations, and the full model has to beat all of them. Three are load-bearing and it is worth saying which in advance. No internal state is what makes H1 mean anything — without it, a negative agent doing worse than a positive one could simply mean the machinery costs something. Learned constant β is the only condition that isolates the headline claim, because frozen β is not learned and so confounds conditioning-on-e with having a trainable parameter path. And state as input only is the cheapest rival of all: if letting an ordinary policy see e recovers most of the effect, the second policy and the mixture are an expensive way to widen an input layer.

6.4 Metrics

• Task reward — mean return, and steps to a fixed reward threshold as the sample-efficiency measure.
• Exploration — state-visitation entropy and fraction of the reachable state space visited, which is what H1 is about.
• Habit formation — action-sequence entropy and the length of the longest repeated action n-gram, as the operational definition of a policy going rigid.
• Affect trajectory — mean and variance of e over training, and time spent past a magnitude threshold. For arms without an internal state, a shadow e is computed from the same event mapping and logged so the trajectories can be compared, but it is never fed to that agent's policy.
• Recovery — steps to return to baseline valence after an induced negative event, which is the quantity H2 predicts will diverge.

6.5 Statistics

Ten seeds per condition per environment, fixed and recorded in advance. Comparisons between conditions use bootstrap confidence intervals over seeds rather than significance tests on individual runs. Every number reported in v1.0 will be aggregated over the full ten; no seed will be dropped for being unrepresentative.

Ten seeds is few, and the methodology for saying anything reliable from few seeds is settled: Agarwal et al. [4] show that point estimates over a handful of runs routinely reverse under resampling, and prescribe interval estimates, stratified bootstrap confidence intervals, and performance profiles instead of mean-and-standard-error. v1.0 will report interquartile mean with stratified bootstrap CIs and a performance profile per environment, using their published implementation rather than a reimplementation. Where a comparison's interval crosses zero I will say the experiment was underpowered, which is the honest reading and the one that a mean would have hidden.

The size of the garden of forking paths, stated up front. Eleven conditions × two environments × ten seeds is 220 runs. The frozen-β and no-arbitration sweeps each add five values × two environments × ten seeds — the same seed count as everything else, since a sweep point deciding whether the full model beats a fixed alternative needs the same statistical footing as the primary comparisons — 100 runs apiece, 200 total. The held-out third environment adds a further 20 (the two disposition arms, ten seeds each) for H5's transfer test. Call it 440. Five hypotheses evaluated in two environments is ten primary tests, and at that number something reaches an arbitrary threshold by chance more often than not. So: each hypothesis is evaluated per environment and reported per environment, never pooled to whichever pooling favours it; the primary set is fixed as H1–H5 plus H2b and cannot be extended after the fact.

At 10M steps per run, 440 runs is 4.4 billion environment steps — a real compute commitment for one person, not an incidental detail, and I would rather say so than let the number sit there implying an infrastructure I have not stated. I am not committing to running all 440 at once. v1.0 reports whichever of the primary comparisons (H1–H5 plus H2b, in both environments) are complete, says plainly which are not, and treats the sweeps and the third-environment transfer test as the parts most likely to lag behind the primary result if time or compute runs short.

How the decision rule and the interval estimates fit together. Each hypothesis is decided by its §7 criterion — a counting rule for H1, H2, H3 and H5 (the effect in at least X of 10 seeds per environment), and a TOST equivalence test for H4. The bootstrap intervals (interquartile mean with stratified CIs, per Agarwal et al.) are reported as descriptive uncertainty, not as the decision criterion. Multiplicity across the family of ten is controlled structurally rather than by a multiple-comparisons correction: the hypothesis set is fixed in advance (H1–H5 plus H2b, no post-hoc extension), each is evaluated and reported per environment (never pooled to whichever pooling favours it), and the counting rule requires a majority of seeds rather than a single significant test. Applying Holm–Bonferroni to counting rules would be incoherent, so it is not done.

The decision rule for the paper as a whole. H1 is the primary outcome: does a negatively disposed agent learn more slowly than a positively disposed one and than an agent with no internal state at all. If H1 fails in both environments, v1.0 reports a negative result in its title and abstract, not a subgroup that survived. If H1 holds but H2 does not, the finding is that a disposition slows learning without trapping the agent, which is a smaller and still publishable claim — and the abstract says so rather than implying the comfort zone. If H1 holds but learned-constant β matches the full model (§4.5.1), the finding is about dual-policy mixing and the state-conditioning contributed nothing. If no-arbitration also matches, the finding is about homeostatic RL [15] and I will say the architecture added nothing to a formalism from 2011. Writing those four sentences before the runs is the only way they get written at all.

7. Predictions

Stated before any run, each with the observation that would sink it. Five predictions, six rows: H2 needs two separate measurements, because §4.7 admits two different objects — a fixed point and a limit cycle — and a prediction that is satisfied by either is not a prediction. H2 asks whether the agent settles somewhere worse than it needs to be; H2b asks whether it settles at all. Every threshold below is fixed here and will not be adjusted after seeing data.

8. Analysis plan

Primary analysis is the five predictions above, each against its named ablation, reported whichever way it comes out. Secondary and clearly labeled as exploratory: comparison of agent trajectories against human trajectories on the same environments.

The platform preserves every state–action trajectory, so runs can be replayed and the internal state inspected at any point rather than only summarized. Where a claim is about a behavioral pattern, the supplementary material will carry the replay, not just the curve.

9. Limitations and threats to validity

• The rates in φ are chosen, even though the events are not. The events themselves are objective — a face is complete, the car is on the track — so the rates do not encode a judgement about what should feel bad. What is chosen is the 1 : 2 : 10 ratio and the 5-step window. They set how fast the internal state moves, so they could change the size of an effect, though it is harder to see how they would manufacture one. Fitting them to human trajectories on the same environments is the obvious next step and is not in this plan.
• One number is a deliberate under-description. A scalar cannot represent ambivalence, cannot distinguish fear from anger, and cannot say whether a bad state is agitated or flat. §2.3 argues that is the right trade for this question; it is still a trade, and a result at this resolution says nothing about which specific emotion does what.
• Two environments are not a general claim. Both are games. Nothing here licenses a statement about agents in general, and H3 is explicitly a prediction about reward structure, not about domains.
• Added capacity is a confound. The affective agent has more parameters than the baseline. The scrambled-affect control is the defense, and if it fails the result is a parameter-count result.
• The bridge to human psychology is a conjecture. A computational agent producing a behavioral pattern is not evidence that people produce it for the same reason. §10 restates this.
• The updates are off-policy and uncorrected. Both components learn from data generated by the mixture as though they had generated it themselves (§4.4). The size of that error is logged, not corrected, in v1.0.

10. Ethics and broader impact

No claim is being made that these agents feel anything. The architecture simulates the functional role that affect plays in decision-making — a second evaluative channel that gates action. It does not thereby produce experience, and nothing in the results, in either direction, will bear on whether the agent has an inner life. The words joy and fear are used throughout as names for coordinates in a vector. I am stating this at length because the research programme this sits inside is about sentience, and the inference is an easy one to make by accident.

The dual-use consideration is real and worth naming: a model of how to make an agent avoid exploration by conditioning it negatively is also a model of how to make an agent tractable by making it timid, and the same machinery pointed at a human-facing system is a model of manipulation. The mitigation is that everything here is at toy scale in games, and that the mechanism is published rather than deployed.

The reason the fourth capacity matters to the wider programme is the question sitting under all of it: what a moral stopper would look like inside an agent that only wants to win. An architecture where a second channel can override the optimizer is a place to start asking.

11. Reproducibility and availability

The playground platform is public today; the rest of this list is still a plan. So is its release policy. What follows is what v1.0 will ship with, and it is a commitment I am willing to be held to, not a description of the current state.

• Already released. The playground platform — both games, and the API that lets human and agent players use them — is public and MIT-licensed on GitHub. No trained model lives there yet; that ships with the results, not before.
• Will be released, with v1.0. Training loops, the dual-model architecture, hyperparameters, and the full seed list, on GitHub. Pre-trained baseline and affective checkpoints on Hugging Face.
• Will be released, with v1.0. Exported trajectory datasets and the figures generated from them, so that every chart in v1.0 can be regenerated from the raw data without the platform.
• Will not be released. The interactive replay interface used to interrogate a specific run, which stays restricted for hosting-cost reasons.

This document is v0.1 and describes work not yet done. The results version will be v1.0 at this address, with the predictions in §7 reproduced verbatim above whatever actually happened.

References

1. Abel, D., Dabney, W., Harutyunyan, A., Ho, M. K., Littman, M. L., Precup, D., & Singh, S. (2021). On the Expressivity of Markov Reward. NeurIPS 34.
2. Abels, A., Roijers, D. M., Lenaerts, T., Nowé, A., & Steckelmacher, D. (2019). Dynamic Weights in Multi-Objective Deep Reinforcement Learning. ICML, 11–20.
3. Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained Policy Optimization. ICML, 22–31. arXiv:1705.10528.
4. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. G. (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS 34. arXiv:2108.13264.
5. Altman, E. (1999). Constrained Markov Decision Processes. Chapman & Hall/CRC.
6. Barrett, L. F. (2017). How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt.
7. Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by Random Network Distillation. arXiv:1810.12894.
8. Damasio, A. R. (1994). Descartes' Error: Emotion, Reason, and the Human Brain. Putnam.
9. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711.
10. Devlin, S., & Kudenko, D. (2012). Dynamic Potential-Based Reward Shaping. AAMAS, 433–440.
11. Dunn, B. D., Dalgleish, T., & Lawrence, A. D. (2006). The somatic marker hypothesis: A critical evaluation. Neuroscience & Biobehavioral Reviews, 30(2), 239–271.
12. Eldar, E., Rutledge, R. B., Dolan, R. J., & Niv, Y. (2016). Mood as Representation of Momentum. Trends in Cognitive Sciences, 20(1), 15–24.
13. Frijda, N. H. (1986). The Emotions. Cambridge University Press.
14. Haarnoja, T., Pong, V., Zhou, A., Dalal, M., Abbeel, P., & Levine, S. (2018). Composable Deep Reinforcement Learning for Robotic Manipulation. ICRA, 6244–6251.
15. Keramati, M., & Gutkin, B. S. (2011). A Reinforcement Learning Theory for Homeostatic Regulation. NIPS 24, 82–90.
16. Keramati, M., & Gutkin, B. (2014). Homeostatic reinforcement learning for integrating reward collection and physiological stability. eLife, 3, e04811.
17. Lazarus, R. S. (1991). Emotion and Adaptation. Oxford University Press.
18. Marsella, S. C., & Gratch, J. (2009). EMA: A process model of appraisal dynamics. Cognitive Systems Research, 10(1), 70–90.
19. Moerland, T. M., Broekens, J., & Jonker, C. M. (2018). Emotion in reinforcement learning agents and robots: a survey. Machine Learning, 107(2), 443–480.
20. Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. ICML, 278–287.
21. Ortony, A., Clore, G. L., & Collins, A. (1988). The Cognitive Structure of Emotions. Cambridge University Press.
22. Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. ICML, 2778–2787.
23. Peng, X. B., Chang, M., Zhang, G., Abbeel, P., & Levine, S. (2019). MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. NeurIPS 32. arXiv:1905.09808.
24. Plutchik, R. (1980). Emotion: A Psychoevolutionary Synthesis. Harper & Row.
25. Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking Safe Exploration in Deep Reinforcement Learning. OpenAI technical report.
26. Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67–113.
27. Roijers, D. M., & Whiteson, S. (2017). Multi-Objective Decision Making. Synthesis Lectures on Artificial Intelligence and Machine Learning, 11(1). Morgan & Claypool.
28. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.
29. Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014). A computational and neural model of momentary subjective well-being. PNAS, 111(33), 12252–12257.
30. Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). Reward is enough. Artificial Intelligence, 299, 103535.
31. Singh, S., Barto, A. G., & Chentanez, N. (2004). Intrinsically motivated reinforcement learning. NIPS, 1281–1288.
32. Spinoza, B. (1677). Ethics, Part III: On the Origin and Nature of the Affects.
33. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
34. Teh, Y. W., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., & Pascanu, R. (2017). Distral: Robust Multitask Reinforcement Learning. NeurIPS 30. arXiv:1707.04175.
35. Tessler, C., Mankowitz, D. J., & Mannor, S. (2019). Reward Constrained Policy Optimization. ICLR.
36. Vamplew, P., Yearwood, J., Dazeley, R., & Berry, A. (2008). On the Limitations of Scalarisation for Multi-objective Reinforcement Learning of Pareto Fronts. AI 2008: Advances in Artificial Intelligence, 372–378.
37. van Niekerk, B., James, S., Earle, A., & Rosman, B. (2019). Composing Value Functions in Reinforcement Learning. ICML, 6401–6409.

Citation

If you want to reference this:

Quote it, republish it, build on it — just say where it came from.