scholarly journals Unlocking the reinforcement-learning circuits of the orbitofrontal cortex.

2021 ◽  
Vol 135 (2) ◽  
pp. 120-128
Author(s):  
Stephanie M. Groman ◽  
Daeyeol Lee ◽  
Jane R. Taylor
Keyword(s):  
Reinforcement Learning ◽  
Orbitofrontal Cortex ◽  
Learning Circuits

scholarly journals Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning

2011 ◽  
Vol 31 (7) ◽  
pp. 2700-2705 ◽  
Author(s):  
M. A. McDannald ◽  
F. Lucantonio ◽  
K. A. Burke ◽  
Y. Niv ◽  
G. Schoenbaum
Keyword(s):  
Reinforcement Learning ◽  
Orbitofrontal Cortex ◽  
Ventral Striatum ◽  
Model Based ◽  
Model Free

scholarly journals Executive modulation of brain reward systems endows goals with value

2020 ◽  
Author(s):  
Samuel D. McDougle ◽  
Ian C. Ballard ◽  
Beth Baribault ◽  
Sonia J. Bishop ◽  
Anne G.E. Collins
Keyword(s):  
Executive Function ◽  
Reinforcement Learning ◽  
Goal Attainment ◽  
Reward System ◽  
Executive Processes ◽  
Learning Contexts ◽  
Brain Reward ◽  
Learning Circuits ◽  
Goal Directed Behavior ◽  
Do So

ABSTRACTRecent evidence suggests that executive processes shape reinforcement learning (RL) computations. Here, we extend this idea to the processing of choice outcomes, asking if executive function and RL interact during learning from novel goals. We designed a task where people learned from familiar rewards or abstract instructed goals. We hypothesized that learning from these goals would produce reliable responses in canonical reward circuits, and would do so by leveraging executive function. Behavioral results pointed to qualitatively similar learning processes when subjects learned from achieving goals versus familiar rewards. Goal learning was robustly and selectively correlated with performance on an independent executive function task. Neuroimaging revealed comparable appetitive responses and computational signatures in reinforcement learning circuits for both goal-based and familiar learning contexts. During goal learning, we observed enhanced correlations between prefrontal cortex and canonical reward-sensitive regions, including hippocampus, striatum, and the midbrain. These findings demonstrate that attaining novel goals produces reliable reward signals in dopaminergic circuits. We propose that learning from goal-directed behavior is mediated by top-down input that primes the reward system to endow value to cues signaling goal attainment.

scholarly journals Meta-reinforcement learning in a thalamo-orbitofrontal circuit

2020 ◽  
Author(s):  
Vijay Mohan K Namboodiri ◽  
Taylor Hobbs ◽  
Ivan Trujillo Pisanty ◽  
Rhiana C Simon ◽  
Garret D Stuber
Keyword(s):  
Reinforcement Learning ◽  
Orbitofrontal Cortex ◽  
Calcium Imaging ◽  
Prediction Error ◽  
Reward Learning ◽  
Two Photon ◽  
Simultaneous Imaging ◽  
Reward Prediction ◽  
Biological Substrate ◽  
The Difference

Learning to predict rewards is essential for the survival of animals. Contemporary views suggest that such learning is driven by a reward prediction error—the difference between received and predicted rewards. Here we show using two-photon calcium imaging and optogenetics in mice that a different class of reward learning signals exists within the orbitofrontal cortex (OFC). Specifically, the reward responses of many OFC neurons exhibit plasticity consistent with filtering out rewards that are less salient for learning (such as predicted rewards, or, unpredicted rewards available in a context containing highly salient aversive stimuli). We show using quasi-simultaneous imaging and optogenetics that this reward response plasticity is sculpted by medial thalamic inputs to OFC. These results provide a biological substrate for emerging theoretical views of meta-reinforcement learning in prefrontal cortex.

scholarly journals A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning

2018 ◽  
Vol 14 (1) ◽  
pp. e1005925 ◽  
Author(s):  
Zhewei Zhang ◽  
Zhenbo Cheng ◽  
Zhongqiao Lin ◽  
Chechang Nie ◽  
Tianming Yang
Keyword(s):  
Neural Network ◽  
Reinforcement Learning ◽  
Network Model ◽  
Neural Network Model ◽  
Orbitofrontal Cortex ◽  
Task Space ◽  
Space Acquisition ◽  
And Task

scholarly journals A State Representation for Reinforcement Learning and Decision-Making in the Orbitofrontal Cortex

2017 ◽  
Author(s):  
Nicolas W. Schuck ◽  
Robert Wilson ◽  
Yael Niv
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Orbitofrontal Cortex ◽  
High Dimensional ◽  
State Representation ◽  
Functional Studies ◽  
Related Information ◽  
Novel Theory ◽  
Partially Observable ◽  
Support Decision Making

AbstractDespite decades of research, the exact ways in which the orbitofrontal cortex (OFC) influences cognitive function have remained mysterious. Anatomically, the OFC is characterized by remarkably broad connectivity to sensory, limbic and subcortical areas, and functional studies have implicated the OFC in a plethora of functions ranging from facial processing to value-guided choice. Notwithstanding such diversity of findings, much research suggests that one important function of the OFC is to support decision making and reinforcement learning. Here, we describe a novel theory that posits that OFC’s specific role in decision-making is to provide an up-to-date representation of task-related information, called a state representation. This representation reflects a mapping between distinct task states and sensory as well as unobservable information. We summarize evidence supporting the existence of such state representations in rodent and human OFC and argue that forming these state representations provides a crucial scaffold that allows animals to efficiently perform decision making and reinforcement learning in high-dimensional and partially observable environments. Finally, we argue that our theory offers an integrating framework for linking the diversity of functions ascribed to OFC and is in line with its wide ranging connectivity.

scholarly journals A probability distribution over latent causes in the orbitofrontal cortex

2016 ◽  
Author(s):  
Stephanie C.Y. Chan ◽  
Yael Niv ◽  
Kenneth A. Norman
Keyword(s):  
Reinforcement Learning ◽  
Probability Distribution ◽  
Episodic Memory ◽  
Orbitofrontal Cortex ◽  
Posterior Probability ◽  
Recent Observation ◽  
Inference Problem ◽  
Relevant Alternatives ◽  
Posterior Probability Distribution ◽  
Pattern Similarity

ABSTRACTThe orbitofrontal cortex (OFC) has been implicated in both the representation of “state”, in studies of reinforcement learning and decision making, and also in the representation of “schemas”, in studies of episodic memory. Both of these cognitive constructs require a similar inference about the underlying situation or “latent cause” that generates our observations at any given time. The statistically optimal solution to this inference problem is to use Bayes rule to compute a posterior probability distribution over latent causes. To test whether such a posterior probability distribution is represented in the OFC, we tasked human participants with inferring a probability distribution over four possible latent causes, based on their observations. Using fMRI pattern similarity analyses, we found that BOLD activity in OFC is best explained as representing the (log-transformed) posterior distribution over latent causes. Furthermore, this pattern explained OFC activity better than other task-relevant alternatives such as the most probable latent cause, the most recent observation, or the uncertainty over latent causes.SIGNIFICANCE STATEMENTOur world is governed by hidden (latent) causes that we cannot observe, but which generate the observations that we do see. A range of high-level cognitive processes require inference of a probability distribution (or “belief distribution”) over the possible latent causes that might be generating our current observations. This is true for reinforcement learning (where the latent cause comprises the true “state” of the task), and for episodic memory (where memories are believed to be organized by the inferred situation or “schema”). Using fMRI, we show that this belief distribution over latent causes is encoded in patterns of brain activity in the orbitofrontal cortex — an area that has been separately implicated in the representations of both states and schemas.CONFLICT OF INTERESTThe authors declare no competing financial interests.

scholarly journals A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning

2017 ◽  
Author(s):  
Zhewei Zhang ◽  
Zhenbo Cheng ◽  
Zhongqiao Lin ◽  
Chechang Nie ◽  
Tianming Yang
Keyword(s):  
Neural Network ◽  
Reinforcement Learning ◽  
State Space ◽  
Network Model ◽  
Neural Network Model ◽  
Orbitofrontal Cortex ◽  
Activity Patterns ◽  
Neural Mechanism ◽  
Theoretical Explanation ◽  
Dynamic Activity

AbstractReinforcement learning has been widely used in explaining animal behavior. In reinforcement learning, the agent learns the value of the states in the task, collectively constituting the task state space, and use the knowledge to choose actions and acquire desired outcomes. It has been proposed that the orbitofrontal cortex (OFC) encodes the task state space during reinforcement learning. However, it is not well understood how the OFC acquires and stores task state information. Here, we propose a neural network model based on reservoir computing. Reservoir networks exhibit heterogeneous and dynamic activity patterns that are suitable to encode task states. The information can be extracted by a linear readout trained with reinforcement learning. We demonstrate how the network acquires and stores the task structures. The network exhibits reinforcement learning behavior and its aspects resemble experimental findings of the OFC. Our study provides a theoretical explanation of how the OFC may contribute to reinforcement learning and a new approach to understanding the neural mechanism underlying reinforcement learning.

Sign in / Sign up

Export Citation Format

Share Document