Learning with reinforcement prediction errors in a model of the Drosophila mushroom body

Effective decision making in a changing environment demands that accurate predictions are learned about decision outcomes. In Drosophila, such learning is orchestrated in part by the mushroom body, where dopamine neurons signal reinforcing stimuli to modulate plasticity presynaptic to mushroom body output neurons. Building on previous mushroom body models, in which dopamine neurons signal absolute reinforcement, we propose instead that dopamine neurons signal reinforcement prediction errors by utilising feedback reinforcement predictions from output neurons. We formulate plasticity rules that minimise prediction errors, verify that output neurons learn accurate reinforcement predictions in simulations, and postulate connectivity that explains more physiological observations than an experimentally constrained model. The constrained and augmented models reproduce a broad range of conditioning and blocking experiments, and we demonstrate that the absence of blocking does not imply the absence of prediction error dependent learning. Our results provide five predictions that can be tested using established experimental methods.

Learning with reward prediction errors in a model of the Drosophila mushroom body

Effective decision making in a changing environment demands that accurate predictions are learned about decision outcomes. In Drosophila, such learning is or-chestrated in part by the mushroom body (MB), where dopamine neurons (DANs) signal reinforcing stimuli to modulate plasticity presynaptic to MB output neurons (MBONs). Here, we extend previous MB models, in which DANs signal absolute rewards, proposing instead that DANs signal reward prediction errors (RPEs) by utilising feedback reward predictions from MBONs. We formulate plasticity rules that minimise RPEs, and use simulations to verify that MBONs learn accurate reward predictions. We postulate as yet unobserved connectivity, which not only overcomes limitations in the experimentally constrained model, but also explains additional experimental observations that connect MB physiology to learning. The original, experimentally constrained model and the augmented model capture a broad range of established fly behaviours, and together make five predictions that can be tested using established experimental methods.

Time-Out Procedures in School Settings

Time-out is a consequence-based strategy that includes altering a child’s environment such that he or she has relatively less contact with reinforcement, contingent upon a target behavior. Time-out may be considered a type of negative punishment procedure because it includes the removal of reinforcing stimuli, resulting in a decrease in the future frequency of a target behavior. This chapter describes time-out and several notable variations of time-out that range along a continuum of intrusiveness. In addition, procedural elements commonly incorporated within time-out are briefly described, followed by additional considerations when using time-out in school settings.

Dynamic salience processing in paraventricular thalamus gates associative learning

The salience of behaviorally relevant stimuli is dynamic and influenced by internal state and external environment. Monitoring such changes is critical for effective learning and flexible behavior, but the neuronal substrate for tracking the dynamics of stimulus salience is obscure. We found that neurons in the paraventricular thalamus (PVT) are robustly activated by a variety of behaviorally relevant events, including novel (“unfamiliar”) stimuli, reinforcing stimuli and their predicting cues, as well as omission of the expected reward. PVT responses are scaled with stimulus intensity and modulated by changes in homeostatic state or behavioral context. Inhibition of the PVT responses suppresses appetitive or aversive associative learning and reward extinction. Our findings demonstrate that the PVT gates associative learning by providing a dynamic representation of stimulus salience.

Shared mushroom body circuits underlie visual and olfactory memories in Drosophila

In nature, animals form memories associating reward or punishment with stimuli from different sensory modalities, such as smells and colors. It is unclear, however, how distinct sensory memories are processed in the brain. We established appetitive and aversive visual learning assays for Drosophila that are comparable to the widely used olfactory learning assays. These assays share critical features, such as reinforcing stimuli (sugar reward and electric shock punishment), and allow direct comparison of the cellular requirements for visual and olfactory memories. We found that the same subsets of dopamine neurons drive formation of both sensory memories. Furthermore, distinct yet partially overlapping subsets of mushroom body intrinsic neurons are required for visual and olfactory memories. Thus, our results suggest that distinct sensory memories are processed in a common brain center. Such centralization of related brain functions is an economical design that avoids the repetition of similar circuit motifs.

Sleep deprivation amplifies striatal activation to monetary reward

BackgroundSleep loss produces abnormal increases in reward seeking but the mechanisms underlying this phenomenon are poorly understood. The present study examined the influence of one night of sleep deprivation on neural responses to a monetary reward task in a sample of late adolescents/young adults.MethodUsing a within-subjects crossover design, 27 healthy, right-handed late adolescents/young adults (16 females, 11 males; mean age 23.1 years) underwent functional magnetic resonance imaging (fMRI) following a night of sleep deprivation and following a night of normal sleep. Participants' recent sleep history was monitored using actigraphy for 1 week prior to each sleep condition.ResultsFollowing sleep deprivation, participants exhibited increased activity in the ventral striatum (VS) and reduced deactivation in the medial prefrontal cortex (mPFC) during the winning of monetary reward, relative to the same task following normal sleep conditions. Shorter total sleep time over the five nights before the sleep-deprived testing condition was associated with reduced deactivation in the mPFC during reward.ConclusionsThese findings support the hypothesis that sleep loss produces aberrant functioning in reward neural circuitry, increasing the salience of positively reinforcing stimuli. Aberrant reward functioning related to insufficient sleep may contribute to the development and maintenance of reward dysfunction-related disorders, such as compulsive gambling, eating, substance abuse and mood disorders.

The conceptualisation, measurement and scope of reinforcement sensitivity in the context of a neuroscience of personality

Reinforcement sensitivity theory (RST) is complex, and there are subtle differences between RST and other approach‐avoidance process theories of personality. However, most such theories posit a common biobehavioural mechanism underlying personality which we must therefore strive to understand: differential sensitivity to reinforcing stimuli. Reinforcement sensitivity is widely assessed using questionnaires, but should we treat such measures as (a) a proxy for reinforcement sensitivity itself (i.e. the underlying causes of personality) or (b) trait constructs potentially manifesting out of reinforcement sensitivity (i.e. the ‘surface’ of personality)? Might neuroscience paradigms, such as those I have reviewed in my target paper, provide an advantage over questionnaires in allowing us to move closer to (a), thereby improving both the measurement and our understanding of reinforcement sensitivity? Assuming we can achieve this, how useful is reinforcement sensitivity—and biological perspectives more generally—for explaining personality? These are the major questions raised in the discussion of my target paper, and among the most pertinent issues in this field today. Copyright © 2008 John Wiley & Sons, Ltd.