How does the brain translate information signaling potential rewards into motivation to get them? Motivation to obtain reward is thought to depend on the midbrain, (particularly the ventral tegmental area, VTA), the nucleus accumbens (NAcc), and the dorsolateral prefrontal cortex (dlPFC), but it is not clear how the interactions amongst these regions relate to reward-motivated behavior. To study the influence of motivation on these reward-responsive regions and on their interactions, we used Dynamic Causal Modeling (DCM) to analyze functional magnetic resonance imaging (fMRI) data from humans performing a simple task designed to isolate reward anticipation. The use of fMRI permitted the simultaneous measurement of multiple brain regions while human participants anticipated and prepared for opportunities to obtain reward, thus allowing characterization of how information about reward changes physiology underlying motivational drive. Further, we modeled the impact of external reward cues on causal relationships within this network, thus elaborating a link between physiology, connectivity, and motivation. Specifically, our results indicated that dlPFC was the exclusive entry point of information about reward in this network, and that anticipated reward availability caused VTA activation only via its effect on the dlPFC. Anticipated reward thus increased dlPFC activation directly, whereas it influenced VTA and NAcc only indirectly, by enhancing intrinsically weak or inactive pathways from the dlPFC. Our findings of a directional prefrontal influence on dopaminergic regions during reward anticipation suggest a model in which the dlPFC integrates and transmits representations of reward to the mesolimbic and mesocortical dopamine systems, thereby initiating motivated behavior.

Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects, but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge, but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help resolve the fundamental challenge in decision science: explaining why people make the decisions they do.

Intertemporal choices are a ubiquitous class of decisions that involve selecting between outcomes available at different times in the future. We investigated the neural systems supporting intertemporal decisions in healthy younger and older adults. Using functional neuroimaging, we find that aging is associated with a shift in the brain areas that respond to delayed rewards. Although we replicate findings that brain regions associated with the mesolimbic dopamine system respond preferentially to immediate rewards, we find a separate region in the ventral striatum with very modest time dependence in older adults. Activation in this striatal region was relatively insensitive to delay in older but not younger adults. Since the dopamine system is believed to support associative learning about future rewards over time, our observed transfer of function may be due to greater experience with delayed rewards as people age. Identifying differences in the neural systems underlying these decisions may contribute to a more comprehensive model of age-related change in intertemporal choice.

Impulsivity is a variable behavioral trait that depends on numerous factors. For example, increasing the absolute magnitude of available choice options promotes far-sighted decisions. We argue that this magnitude effect arises in part from differential exertion of self-control as the perceived importance of the choice increases. First, we demonstrate that frontal executive control areas are engaged for more difficult decisions and that this effect is enhanced for high magnitude rewards. Second, we show that increased hunger, which is associated with lower self-control, reduces the magnitude effect. Third, we tested an intervention designed to increase self-control and show that it interferes with the magnitude effect. Taken together, our findings challenge existing theories about the magnitude effect and suggest that visceral and cognitive factors affecting choice may do so by influencing self-control.

Background : Reinforcement learning models provide excellent descriptions of learning in multiple species across a variety of tasks. Many researchers are interested in relating parameters of reinforcement learning models to neural measures, psychological variables or experimental manipulations. We demonstrate that parameter identification is difficult because a range of parameter values provide approximately equal quality fits to data. This identification problem has a large impact on power: we show that a researcher who wants to detect a medium sized correlation ( r = .3) with 80% power between a variable and learning rate must collect 60% more subjects than specified by a typical power analysis in order to account for the noise introduced by model fitting.New Method : We derive a Bayesian optimal model fitting technique that takes advantage of information contained in choices and reaction times to constrain parameter estimates.Results : We show using simulation and empirical data that this method substantially improves the ability to recover learning rates.Comparison with Existing Methods : We compare this method against the use of Bayesian priors. We show in simulations that the combined use of Bayesian priors and reaction times confers the highest parameter identifiability. However, in real data where the priors may have been misspecified, the use of Bayesian priors interferes with the ability of reaction time data to improve parameter identifiability.Conclusions : We present a simple technique that takes advantage of readily available data to substantially improve the quality of inferences that can be drawn from parameters of reinforcement learning models. Highlights-Parameters of reinforcement learning models are particularly difficult to estimate -Incorporating reaction times into model fitting improves parameter identifiability -Bayesian weighting of choice and reaction times improves the power of analyses assessing learning rate 2 .

Animals rely on learned associations to make decisions. Associations can be based on relationships between object features (e.g., the three leaflets of poison ivy leaves) and outcomes (e.g., rash). More often, outcomes are linked to multidimensional states (e.g., poison ivy is green in summer but red in spring). Feature-based reinforcement learning fails when the values of individual features depend on the other features present. One solution is to assign value to multi-featural conjunctive representations. Here, we test if the hippocampus forms separable conjunctive representations that enables the learning of response contingencies for stimuli of the form: AB+, B−, AC−, C+. Pattern analyses on functional MRI data show the hippocampus forms conjunctive representations that are dissociable from feature components and that these representations, along with those of cortex, influence striatal prediction errors. Our results establish a novel role for hippocampal pattern separation and conjunctive representation in reinforcement learning.

Novelty detection, a critical computation within the medial temporal lobe (MTL) memory system, necessarily depends on prior experience. The current study used functional magnetic resonance imaging (fMRI) in humans to investigate dynamic changes in MTL activation and functional connectivity as experience with novelty accumulates. fMRI data were collected during a target detection task: Participants monitored a series of trial-unique novel and familiar scene images to detect a repeating target scene. Even though novel images themselves did not repeat, we found that fMRI activations in the hippocampus and surrounding cortical MTL showed a specific, decrementing response with accumulating exposure to novelty. The significant linear decrement occurred for the novel but not the familiar images, and behavioral measures ruled out a corresponding decline in vigilance. Additionally, early in the series, the hippocampus was inversely coupled with the dorsal striatum, lateral and medial prefrontal cortex, and posterior visual processing regions; this inverse coupling also habituated as novelty accumulated. This novel demonstration of a dynamic adjustment in neural responses to novelty suggests a similarly dynamic allocation of neural resources based on recent experience.A fundamental task for organisms is to detect, learn about, and respond to change in the environment. Novelty responses in the brain signal environmental change and predict neural and behavioral adjustments to it; neural responses to novelty are thus a proxy for the salience of environmental change. Evidence from humans, nonhuman primates, and rodents points to a specialized brain system for the detection of novelty, centered around the hippocampus (HPC) and medial temporal lobe (MTL) memory system (Ranganath and Rainer 2003). It is now well documented that the HPC and MTL respond robustly to novel stimuli (Gabrieli et al. 1997;Jessen et al. 2002;Kohler et al. 2005;Bunzeck and Duzel 2006;Yassa and Stark 2008;Blackford et al. 2010;Howard et al. 2011). However, relatively little is known about how these responses are modulated by prior experience. In particular, how does a recent history rich with novel information influence MTL networks specialized for novelty processing?Other literature has described how prior exposure to biologically salient stimuli (or to a repeated feature) changes neural responses to future processing of those stimuli. Throughout the ventral visual stream, neural responses to repeated presentations of visually identical stimuli progressively decrease (Henson et al. 2003). Similarly, the amygdala and other structures implicated in fear processing habituate to cumulative exposure to trialunique stimuli that depict fear (Breiter et al. 1996;Wright et al. 2001;Fischer et al. 2003) or signal threat (Buchel et al. 1998;LaBar et al. 1998). Habituation has been proposed to be biologically adaptive: As a stimulus or feature is repeated, it provides less information about the environment, and demands fewer processing resources (Sokolov 1963;Rankin...

Humans naturally group the world into coherent categories defined by membership rules. Rules can be learned implicitly by building stimulus-response associations using reinforcement learning or by using explicit reasoning. We tested if the striatum, in which activation reliably scales with reward prediction error, would track prediction errors in a task that required explicit rule generation. Using functional magnetic resonance imaging during a categorization task, we show that striatal responses to feedback scale with a "surprise" signal derived from a Bayesian rule-learning model and are inconsistent with RL prediction error. We also find that striatum and caudal inferior frontal sulcus (cIFS) are involved in updating the likelihood of discriminative rules. We conclude that the striatum, in cooperation with the cIFS, is involved in updating the values assigned to categorization rules when people learn using explicit reasoning.