Title: Dopamine Does Double Duty in Motivating Cognitive Effort
Abstract: Cognitive control is subjectively costly, suggesting that engagement is modulated in relationship to incentive state. Dopamine appears to play key roles. In particular, dopamine may mediate cognitive effort by two broad classes of functions: (1) modulating the functional parameters of working memory circuits subserving effortful cognition, and (2) mediating value-learning and decision-making about effortful cognitive action. Here, we tie together these two lines of research, proposing how dopamine serves “double duty”, translating incentive information into cognitive motivation. Cognitive control is subjectively costly, suggesting that engagement is modulated in relationship to incentive state. Dopamine appears to play key roles. In particular, dopamine may mediate cognitive effort by two broad classes of functions: (1) modulating the functional parameters of working memory circuits subserving effortful cognition, and (2) mediating value-learning and decision-making about effortful cognitive action. Here, we tie together these two lines of research, proposing how dopamine serves “double duty”, translating incentive information into cognitive motivation. Why is thinking effortful? Unlike physical exertion, there is no readily apparent metabolic cost (relative to “rest”, which is already metabolically expensive) (Raichle and Mintun, 2006Raichle M.E. Mintun M.A. Brain work and brain imaging.Annu. Rev. Neurosci. 2006; 29: 449-476Crossref PubMed Scopus (756) Google Scholar). And yet, we avoid engaging in demanding activities even when doing so might further valuable goals. This appears particularly true when goal pursuit requires extended allocation of working memory for cognitive control. One hypothesis is that cognitive effort avoidance is intended to minimize opportunity costs incurred by the allocation of working memory (Kurzban et al., 2013Kurzban R. Duckworth A. Kable J.W. Myers J. An opportunity cost model of subjective effort and task performance.Behav. Brain Sci. 2013; 36: 661-679Crossref PubMed Scopus (154) Google Scholar). If this is true, it suggests not only that working memory is allocated opportunistically, but also that allocation policies entail sophisticated cost-benefit decision-making that is sensitive to as yet unknown cost and incentive functions. In any case, the phenomenon raises a number of questions: How do brains track effort costs? What information is being tracked? How can incentives overcome such costs? What mechanisms mediate adaptive working memory allocation? Working memory capacity is sharply limited, especially in the domain of cognitive control, involving abstract, flexible, hierarchical rules for behavior selection. Optimizing working memory allocation is thus critical for optimizing behavior. Prevalent computational frameworks have proposed reward- or expectancy-maximization algorithms for working memory allocation (Botvinick et al., 2001Botvinick M.M. Braver T.S. Barch D.M. Carter C.S. Cohen J.D. Conflict monitoring and cognitive control.Psychol. Rev. 2001; 108: 624-652Crossref PubMed Google Scholar, Donoso et al., 2014Donoso M. Collins A.G. Koechlin E. Human cognition. Foundations of human reasoning in the prefrontal cortex.Science. 2014; 344: 1481-1486Crossref PubMed Scopus (67) Google Scholar, O’Reilly and Frank, 2006O’Reilly R.C. Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia.Neural Comput. 2006; 18: 283-328Crossref PubMed Scopus (0) Google Scholar). Yet, these frameworks largely neglect that working memory allocation itself carries affective valence. High subjective costs drive disengagement, whereas sufficient incentive drives engagement. That is, allocation of working memory is a motivated process. In this review, we argue that modulatory functions of the midbrain dopamine (DA) system translate cost-benefit information into adaptive working memory allocation. DA has been implicated in numerous processes including, but not limited to, motivation, learning, working memory, and decision-making. There are two largely independent literatures that ascribe disparate functional roles to DA with relevance to motivated cognition. First, DA influences the allocation of working memory directly by modulating the functional parameters of working memory circuits. For example, DA tone in the prefrontal cortex (PFC) influences the stability of working memory representations, with higher extrasynaptic tone promoting greater stability, to a limit (Seamans and Yang, 2004Seamans J.K. Yang C.R. The principal features and mechanisms of dopamine modulation in the prefrontal cortex.Prog. Neurobiol. 2004; 74: 1-58Crossref PubMed Scopus (0) Google Scholar). Phasic DA efflux may also push beyond the limit and toggle the PFC into a labile state such that working memory representations can be flexibly updated (Braver et al., 1999Braver T.S. Barch D.M. Cohen J.D. Cognition and control in schizophrenia: a computational model of dopamine and prefrontal function.Biol. Psychiatry. 1999; 46: 312-328Abstract Full Text Full Text PDF PubMed Scopus (303) Google Scholar). Additionally, DA may support the learning of more sophisticated (and hierarchical) allocation policies via synaptic depression and potentiation in corticostriatal loops (Frank et al., 2001Frank M.J. Loughry B. O’Reilly R.C. Interactions between frontal cortex and basal ganglia in working memory: a computational model.Cogn. Affect. Behav. Neurosci. 2001; 1: 137-160Crossref PubMed Google Scholar, O’Reilly and Frank, 2006O’Reilly R.C. Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia.Neural Comput. 2006; 18: 283-328Crossref PubMed Scopus (0) Google Scholar). Second, DA is critical for action selection. Specifically, DA trains value functions for action selection via phasic reward prediction error dynamics potentiating behaviors that maximize reward with respect to effort in a given context (see Niv, 2009Niv Y. Reinforcement learning in the brain.J. Math. Psychol. 2009; 53: 139-154Crossref Scopus (118) Google Scholar for a review). DA tone in the striatum and the medial PFC also promotes preparatory and instrumental behaviors in response to conditioned stimuli and particularly effortful behavior (Kurniawan et al., 2011Kurniawan I.T. Guitart-Masip M. Dolan R.J. Dopamine and effort-based decision making.Front. Neurosci. 2011; 5: 81Crossref PubMed Scopus (0) Google Scholar, Salamone and Correa, 2012Salamone J.D. Correa M. The mysterious motivational functions of mesolimbic dopamine.Neuron. 2012; 76: 470-485Abstract Full Text Full Text PDF PubMed Scopus (341) Google Scholar). Here, we tie together these largely independent lines of research by proposing how the very same functional properties of DA encoding incentive information translate incentives into cognitive motivation by regulating working memory. Specifically, we propose that DA dynamics encoding incentive state promote subjectively costly working memory operations experienced as conscious, phenomenal effort. As we detail below, our proposal makes use of the concept of a “control episode” during goal pursuit (cf. “attentional episodes”, see Duncan, 2013Duncan J. The structure of cognition: attentional episodes in mind and brain.Neuron. 2013; 80: 35-50Abstract Full Text Full Text PDF PubMed Scopus (83) Google Scholar), involving stable maintenance of the goal state at higher-levels of the control hierarchy, along with selective updating of lower level rules for guiding behavior during completion of subgoals, as progress is made toward the ultimate goal state. We review the ways in which DA dynamics encoding a net cost-benefit of goal engagement and persistence result in adaptive working memory allocation. As such, DA translates incentive motivation into cognitive effort. Cognitive effort is an everyday experience. The subjective costliness of cognitive effort is consequential, sometimes driving disengagement from otherwise highly valuable goals. Yet, surprisingly little is known about this phenomenon. It is neither clear what makes tasks effortful, nor why task engagement is apparently aversive in the first place (Inzlicht et al., 2014Inzlicht M. Schmeichel B.J. Macrae C.N. Why self-control seems (but may not be) limited.Trends Cogn. Sci. 2014; 18: 127-133Abstract Full Text Full Text PDF PubMed Scopus (126) Google Scholar, Kurzban et al., 2013Kurzban R. Duckworth A. Kable J.W. Myers J. An opportunity cost model of subjective effort and task performance.Behav. Brain Sci. 2013; 36: 661-679Crossref PubMed Scopus (154) Google Scholar). Beyond a quizzical influence over goal-directed behavior, there are numerous reasons to care about cognitive effort. First, expenditure is critical for career and educational success, economic decision-making, and attitude formation (Cacioppo et al., 1996Cacioppo J. Petty R. Feinstein J. Jarvis W. Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition.Psychol. Bull. 1996; 119: 197-253Crossref Scopus (0) Google Scholar, von Stumm et al., 2011von Stumm S. Hell B. Chamorro-Premuzic T. The hungry mind: intellectual curiosity is the third pillar of academic performance.Perspect. Psychol. Sci. 2011; 6: 574-588Crossref PubMed Scopus (0) Google Scholar). Second, deficient effort may be a significant component of neuropsychiatric disorders for which avolition, anhedonia, and inattention feature prominently, such as attention deficit hyperactivity disorder (ADHD) (Volkow et al., 2011Volkow N.D. Wang G.-J. Newcorn J.H. Kollins S.H. Wigal T.L. Telang F. Fowler J.S. Goldstein R.Z. Klein N. Logan J. et al.Motivation deficit in ADHD is associated with dysfunction of the dopamine reward pathway.Mol. Psychiatry. 2011; 16: 1147-1154Crossref PubMed Scopus (104) Google Scholar), depression (Hammar et al., 2011Hammar A. Strand M. Årdal G. Schmid M. Lund A. Elliott R. Testing the cognitive effort hypothesis of cognitive impairment in major depression.Nord. J. Psychiatry. 2011; 65: 74-80Crossref PubMed Scopus (0) Google Scholar), and schizophrenia (Strauss et al., 2015Strauss G.P. Morra L.F. Sullivan S.K. Gold J.M. The role of low cognitive effort and negative symptoms in neuropsychological impairment in schizophrenia.Neuropsychology. 2015; 29: 282-291Crossref PubMed Google Scholar). Effort avoidance may also contribute to declining cognitive performance in healthy aging (Hess and Ennis, 2012Hess T.M. Ennis G.E. Age differences in the effort and costs associated with cognitive activity.J. Gerontol. B Psychol. Sci. Soc. Sci. 2012; 67: 447-455Crossref PubMed Scopus (0) Google Scholar, Westbrook et al., 2013Westbrook A. Kester D. Braver T.S. What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference.PLoS ONE. 2013; 8: e68210Crossref PubMed Scopus (0) Google Scholar). Engagement with certain kinds of cognitive tasks appears negatively valenced, indicating a subjective cost. Subjectively inflated effort costs might undermine cognitive engagement and thereby performance. Not all tasks are effortful. Tasks requiring allocation of working memory for cognitive control, however, appear to be (Botvinick et al., 2009Botvinick M.M. Huffstetler S. McGuire J.T. Effort discounting in human nucleus accumbens.Cogn. Affect. Behav. Neurosci. 2009; 36: 74-97Google Scholar, Dixon and Christoff, 2012Dixon M.L. Christoff K. The decision to engage cognitive control is driven by expected reward-value: neural and behavioral evidence.PLoS ONE. 2012; 7: e51637Crossref PubMed Scopus (31) Google Scholar, Dreisbach and Fischer, 2012Dreisbach G. Fischer R. Conflicts as aversive signals.Brain Cogn. 2012; 78: 94-98Crossref PubMed Scopus (0) Google Scholar, Kool et al., 2010Kool W. McGuire J.T. Rosen Z.B. Botvinick M.M. Decision making and the avoidance of cognitive demand.J. Exp. Psychol. Gen. 2010; 139: 665-682Crossref PubMed Scopus (159) Google Scholar, Massar et al., 2015Massar S.A.A. Libedinsky C. Weiyan C. Huettel S.A. Chee M.W.L. Separate and overlapping brain areas encode subjective value during delay and effort discounting.Neuroimage. 2015; 120: 104-113Crossref PubMed Scopus (0) Google Scholar, McGuire and Botvinick, 2010McGuire J.T. Botvinick M.M. Prefrontal cortex, cognitive control, and the registration of decision costs.Proc. Natl. Acad. Sci. USA. 2010; 107: 7922-7926Crossref PubMed Scopus (0) Google Scholar, Schouppe et al., 2014Schouppe N. Demanet J. Boehler C.N. Ridderinkhof K.R. Notebaert W. The role of the striatum in effort-based decision-making in the absence of reward.J. Neurosci. 2014; 34: 2148-2154Crossref PubMed Scopus (0) Google Scholar, Westbrook et al., 2013Westbrook A. Kester D. Braver T.S. What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference.PLoS ONE. 2013; 8: e68210Crossref PubMed Scopus (0) Google Scholar). Individuals allowed to select freely between tasks differing only in the frequency with which working memory must be reallocated for cognitive control express a progressive preference for the option with lower reallocation demands (Kool et al., 2010Kool W. McGuire J.T. Rosen Z.B. Botvinick M.M. Decision making and the avoidance of cognitive demand.J. Exp. Psychol. Gen. 2010; 139: 665-682Crossref PubMed Scopus (159) Google Scholar, McGuire and Botvinick, 2010McGuire J.T. Botvinick M.M. Prefrontal cortex, cognitive control, and the registration of decision costs.Proc. Natl. Acad. Sci. USA. 2010; 107: 7922-7926Crossref PubMed Scopus (0) Google Scholar). Critically even when offered larger reward, decision-makers discount reward as a function of effort costs, thus selecting smaller reward with lower demands over larger reward with higher demands (Massar et al., 2015Massar S.A.A. Libedinsky C. Weiyan C. Huettel S.A. Chee M.W.L. Separate and overlapping brain areas encode subjective value during delay and effort discounting.Neuroimage. 2015; 120: 104-113Crossref PubMed Scopus (0) Google Scholar, Westbrook et al., 2013Westbrook A. Kester D. Braver T.S. What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference.PLoS ONE. 2013; 8: e68210Crossref PubMed Scopus (0) Google Scholar). Under what conditions might cognitively demanding tasks acquire affective valence? By one account, tasks demanding cognitive control involve response conflict (Botvinick et al., 2001Botvinick M.M. Braver T.S. Barch D.M. Carter C.S. Cohen J.D. Conflict monitoring and cognitive control.Psychol. Rev. 2001; 108: 624-652Crossref PubMed Google Scholar) or frequent errors (Brown and Braver, 2005Brown J.W. Braver T.S. Learned predictions of error likelihood in the anterior cingulate cortex.Science. 2005; 307: 1118-1121Crossref PubMed Scopus (517) Google Scholar, Holroyd and Coles, 2002Holroyd C.B. Coles M.G.H. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity.Psychol. Rev. 2002; 109: 679-709Crossref PubMed Google Scholar) and as such are less likely to be successful, thus engendering avoidance learning to bias behavior toward tasks with higher chances of success (Botvinick, 2007Botvinick M.M. Conflict monitoring and decision making: reconciling two perspectives on anterior cingulate function.Cogn. Affect. Behav. Neurosci. 2007; 7: 356-366Crossref PubMed Scopus (437) Google Scholar). Multiple lines of evidence suggest that conflict is aversive. First, conflict in the context of a Stroop task predicts overt avoidance (Schouppe et al., 2012Schouppe N. De Houwer J. Ridderinkhof K.R. Notebaert W. Conflict: run! Reduced Stroop interference with avoidance responses.Q. J. Exp. Psychol. (Hove). 2012; 65: 1052-1058Crossref PubMed Scopus (0) Google Scholar). Also, trial-wise variation in subjective frustration with a stop-signal task predicts BOLD signal in the anterior cingulate cortex (ACC), otherwise implicated in conflict detection (Spunt et al., 2012Spunt R.P. Lieberman M.D. Cohen J.R. Eisenberger N.I. The phenomenology of error processing: the dorsal ACC response to stop-signal errors tracks reports of negative affect.J. Cogn. Neurosci. 2012; 24: 1753-1765Crossref PubMed Scopus (0) Google Scholar). In another study (McGuire and Botvinick, 2010McGuire J.T. Botvinick M.M. Prefrontal cortex, cognitive control, and the registration of decision costs.Proc. Natl. Acad. Sci. USA. 2010; 107: 7922-7926Crossref PubMed Scopus (0) Google Scholar), participant ratings of their desire to avoid a conflict-inducing task correlated positively with individual differences in recruitment of ACC and also dorsolateral PFC, putatively involved in working memory maintenance of task sets. Moreover, the dorsolateral PFC correlation remained after controlling for performance differences (reaction time, RTs, and error rates), indicating that the desire to avoid the task did not simply reflect perceived failure. Finally, interesting interactions between affect and cognitive control also support the notion that conflict is aversive (Dreisbach and Goschke, 2004Dreisbach G. Goschke T. How positive affect modulates cognitive control: reduced perseveration at the cost of increased distractibility.J. Exp. Psychol. Learn. Mem. Cogn. 2004; 30: 343-353Crossref PubMed Scopus (0) Google Scholar, Saunders and Inzlicht, 2015Saunders B. Inzlicht M. Vigour and fatigue: How variation in affect underlies effective self-control.in: Braver T.S. Motivation and Cognitive Control. Taylor & Francis/Routledge, 2015: 211-234Google Scholar, Shackman et al., 2011Shackman A.J. Salomons T.V. Slagter H.A. Fox A.S. Winter J.J. Davidson R.J. The integration of negative affect, pain and cognitive control in the cingulate cortex.Nat. Rev. Neurosci. 2011; 12: 154-167Crossref PubMed Scopus (600) Google Scholar). For example, individuals respond faster to affectively negative, and slower to affectively positive stimuli, following priming by conflicting versus non-conflicting Stroop trials (Dreisbach and Fischer, 2012Dreisbach G. Fischer R. Conflicts as aversive signals.Brain Cogn. 2012; 78: 94-98Crossref PubMed Scopus (0) Google Scholar). Avoidance learning to minimize loss may partly explain aversion to working memory allocation for cognitive control. Yet, it cannot be the full story. On the one hand, individuals avoid cognitive demand, even controlling for reward likelihood (Kool et al., 2010Kool W. McGuire J.T. Rosen Z.B. Botvinick M.M. Decision making and the avoidance of cognitive demand.J. Exp. Psychol. Gen. 2010; 139: 665-682Crossref PubMed Scopus (159) Google Scholar, McGuire and Botvinick, 2010McGuire J.T. Botvinick M.M. Prefrontal cortex, cognitive control, and the registration of decision costs.Proc. Natl. Acad. Sci. USA. 2010; 107: 7922-7926Crossref PubMed Scopus (0) Google Scholar, Westbrook et al., 2013Westbrook A. Kester D. Braver T.S. What is the subjective cost of cognitive effort? Load, trait, and aging effects revealed by economic preference.PLoS ONE. 2013; 8: e68210Crossref PubMed Scopus (0) Google Scholar). On the other, opportunity costs may reflect more than just the likelihood of failure during the current control episode; namely, they may reflect the value of missed opportunities (Kurzban et al., 2013Kurzban R. Duckworth A. Kable J.W. Myers J. An opportunity cost model of subjective effort and task performance.Behav. Brain Sci. 2013; 36: 661-679Crossref PubMed Scopus (154) Google Scholar). Finally, an adaptive system must also be judicious, and avoidance of all goals requiring cognitive control is clearly maladaptive. Decision-making must consider both costs and benefits. Indeed, there is growing evidence that the ACC is as important for biasing engagement with effortful, control-demanding tasks as it is for biasing avoidance (Shenhav et al., 2013Shenhav A. Botvinick M.M. Cohen J.D. The expected value of control: an integrative theory of anterior cingulate cortex function.Neuron. 2013; 79: 217-240Abstract Full Text Full Text PDF PubMed Scopus (322) Google Scholar). If control is avoided because of subjective costs, increased incentives could offset costs, promoting control. Indeed, incentives yield control-mediated performance enhancements (see Botvinick and Braver, 2015Botvinick M. Braver T. Motivation and cognitive control: from behavior to neural mechanism.Annu. Rev. Psychol. 2015; 66: 83-113Crossref PubMed Google Scholar, Pessoa and Engelmann, 2010Pessoa L. Engelmann J.B. Embedding reward signals into perception and cognition.Front. Neurosci. 2010; 4: 4Crossref PubMed Scopus (0) Google Scholar for review). Incentives enhance performance in control-demanding tasks encompassing visuospatial attention (Krebs et al., 2012Krebs R.M. Boehler C.N. Roberts K.C. Song A.W. Woldorff M.G. The involvement of the dopaminergic midbrain and cortico-striatal-thalamic circuits in the integration of reward prospect and attentional task demands.Cereb. Cortex. 2012; 22: 607-615Crossref PubMed Scopus (71) Google Scholar, Small et al., 2005Small D.M. Gitelman D. Simmons K. Bloise S.M. Parrish T. Mesulam M.M. Monetary incentives enhance processing in brain regions mediating top-down control of attention.Cereb. Cortex. 2005; 15: 1855-1865Crossref PubMed Scopus (0) Google Scholar), task-switching (Aarts et al., 2010Aarts E. Roelofs A. Franke B. Rijpkema M. Fernández G. Helmich R.C. Cools R. Striatal dopamine mediates the interface between motivational and cognitive control in humans: evidence from genetic imaging.Neuropsychopharmacology. 2010; 35: 1943-1951Crossref PubMed Scopus (0) Google Scholar), working memory (Jimura et al., 2010Jimura K. Locke H.S. Braver T.S. Prefrontal cortex mediation of cognitive enhancement in rewarding motivational contexts.Proc. Natl. Acad. Sci. USA. 2010; 107: 8871-8876Crossref PubMed Scopus (0) Google Scholar), and context maintenance (Chiew and Braver, 2014Chiew K.S. Braver T.S. Dissociable influences of reward motivation and positive emotion on cognitive control.Cogn. Affect. Behav. Neurosci. 2014; 14: 509-529Crossref PubMed Scopus (0) Google Scholar, Locke and Braver, 2008Locke H.S. Braver T.S. Motivational influences on cognitive control: behavior, brain activation, and individual differences.Cogn. Affect. Behav. Neurosci. 2008; 8: 99-112Crossref PubMed Scopus (0) Google Scholar), among others. Furthermore, incentives predict greater activity in control-related regions, including medial and lateral PFC. For example, incentives yield increased BOLD signal in the ACC, propagating to dorsolateral PFC, corresponding well with the canonical model by which the ACC monitors for control demands and recruits lateral PFC to implement control (Kouneiher et al., 2009Kouneiher F. Charron S. Koechlin E. Motivation and cognitive control in the human prefrontal cortex.Nat. Neurosci. 2009; 12: 939-945Crossref PubMed Scopus (242) Google Scholar). This particular study showed that incentives yielded an additive increase in BOLD signal, on top of demand-driven control signals. However, more recent work has shown that incentive information is not merely additive, but interactive: with increasing incentive-related activity under high task-demand conditions, thus more directly implicating incentives in the enhancement of cognitive control (Bahlmann et al., 2015Bahlmann J. Aarts E. D’Esposito M. Influence of motivation on control hierarchy in the human frontal cortex.J. Neurosci. 2015; 35: 3207-3217Crossref PubMed Scopus (0) Google Scholar), cf. Krebs et al., 2012Krebs R.M. Boehler C.N. Roberts K.C. Song A.W. Woldorff M.G. The involvement of the dopaminergic midbrain and cortico-striatal-thalamic circuits in the integration of reward prospect and attentional task demands.Cereb. Cortex. 2012; 22: 607-615Crossref PubMed Scopus (71) Google Scholar). Beyond mean activity, incentives also enhance the fidelity of working memory representations. Task set representations are more distinctive, as revealed by multivariate pattern analysis of BOLD data, during incentivized working memory trials (Etzel et al., 2015Etzel J. Cole M.W. Zacks J.M. Kay K.N. Braver T.S. Reward motivation enhances task coding in frontoparietal cortex.Cereb. Cortex. 2015; (Published online January 19, 2015)https://doi.org/10.1093/cercor/bhu327Crossref PubMed Scopus (3) Google Scholar). Interestingly, increased distinctiveness predicts individual differences in incentive-driven behavioral enhancement. Incentives not only drive more control-related activity, or higher fidelity task set representations, but they also affect the selection of more costly control strategies. For example, cognitive control may be recruited proactively, in advance of imperative events, or reactively, concurrent with event onset (Braver, 2012Braver T.S. The variable nature of cognitive control: a dual mechanisms framework.Trends Cogn. Sci. 2012; 16: 106-113Abstract Full Text Full Text PDF PubMed Scopus (380) Google Scholar). Proactive control has behavioral advantages, but also incurs opportunity costs that bias reliance on reactive control. Incentives appear to offset costs, increasing proactive relative to reactive control, as reflected in sustained increases in BOLD signal prior to imperative events, and attenuated phasic responses at event onsets, and this shift to proactive control predicts performance enhancements (see Jimura et al., 2010Jimura K. Locke H.S. Braver T.S. Prefrontal cortex mediation of cognitive enhancement in rewarding motivational contexts.Proc. Natl. Acad. Sci. USA. 2010; 107: 8871-8876Crossref PubMed Scopus (0) Google Scholar). Moreover, incentive-driven shifts to proactive control are larger among highly reward-sensitive individuals (Jimura et al., 2010Jimura K. Locke H.S. Braver T.S. Prefrontal cortex mediation of cognitive enhancement in rewarding motivational contexts.Proc. Natl. Acad. Sci. USA. 2010; 107: 8871-8876Crossref PubMed Scopus (0) Google Scholar). In sum, working memory operations are treated as subjectively costly. Whether apparent costliness reflects avoidance learning of behaviors with low likelihood of success, or opportunity costs, incentives can counterbalance costs, promoting working memory operations. Cost-benefit decision-making thus underlies working memory allocation for cognitive control. We propose that during goal pursuit, individuals engage in costly control episodes, remaining engaged to the extent that benefits outweigh costs. Moreover, we propose that DA solves a core computational problem of control episodes: namely, value-based management of working memory for cognitive control that reflects not only prior reward learning, but also instantaneous effects of current incentive state. To illustrate, we consider an example control episode involving the demanding task of finding the product of two two-digit numbers, incentivized by points on an examination (without calculators; Figure 1). Control episodes may be initiated by incentive-driven (point-value cued) allocation of working memory to represent the goal state (finding the product). Throughout an episode, the actor must maintain high-level goal information (e.g., the original numbers), resisting interference from distractors, while flexibly updating targeted, lower-level representations of subgoals in a hierarchical fashion. Subgoals in our example include: (1) multiplying the ones column digits; (2) carrying the tens-digit value of that product; (3) adding that value to the product of the tens-digits, etc. Maintaining each subgoal is subjectively costly and thus the stability of goal representations should reflect the value of those goals. Similarly, updating operations, as required when subgoals are completed, are also subjectively costly. As each stage has its own costs, and costs may accumulate in excess of perceived benefits, any stage may result in disengagement. We consider the mental multiplication example for illustrative purposes only; the general notion of a control episode should apply broadly to any hierarchically structured, temporally extended sequence of goal-directed behaviors that require working memory allocation (e.g., planning, problem-solving, and reasoning). In the sections that follow, we describe how DA mediates value-based working memory management during control episodes. Figure 2 provides an overview of critical functions that will be reviewed. Tonic DA, for example, influences the stability of working memory contents by direct action in PFC (Figure 2B), while phasic DA efflux in the striatum trains policies for value-based updating of working memory contents that reflect both the reward value of the goals to which they correspond and effort (updating and maintenance) costs (Figure 2C). While cached value-functions reflect past experience, their implementation is subject to instantaneous modulation by incentive state. Accordingly, we describe how DA and its projection targets encode net incentive state, dynamically accounting for goal state revaluation and generalized motivation. Such information is used to bias policies for working memory allocation actions (Figure 2D). Hence, DA does double duty in translating incentive information into cognitive effort both by functional modulation of working memory circuits (Figures 2B and 2C) and by influencing value-learning and decision-making about effortful action (Figures 2C and 2D). We take up each of these key duties in turn. Successful control episodes demand stable maintenance and also targeted, flexible updating of working memory, with DA appearing to play an important role in both processes. In the PFC, DA influences the stability of recurrent networks (Brunel and Wang, 2001Brunel N. Wang X.-J. Effects of neuromodulation in a cortical network model of object working memory dominated by recurrent inhibition.J. Comput. Neurosci. 2001; 11: 63-85Crossref PubMed Scopus (393) Google Scholar, Seamans and Yang, 2004Seamans J.K. Yang C.R. The principal f