Dopamine – Not Just About “Reward”

Here is a good alternative perspective to the now widely accepted view of dopamine as the reward necurotransmitter.  It is more complex, of course.

Dopamine, Behavioral Economics and Effort

Frontiers in Behavioral Neuroscience September 2009 | Volume 3 | Article 13 | 1 BEHAVIORAL NEUROSCIENCE REVIEW ARTICLE published: 07 September 2009 doi: 10.3389

Against the backdrop of these conceptual and terminological issues, there is a tremendous weight of empirical evidence that has built up against the various iterations of the DA hypothesis of “reward”.  It is somewhat ironic that the processes most directly linked to the use of the term reward (i.e., primary motivation, subjective pleasure) are the ones that have proven to be most problematic in terms of demonstrating the involvement of mesolimbic DA (Salamone et al., 2007).

For example – low doses of DA antagonists and depletions of nucleus accumbens DA have been shown to produce effects that do not closely resemble extinction, pre-feeding or appetite suppressant drugs. Also:
– Although it is well known that whole forebrain DA depletions can produce aphagia (i.e., lack of eating), it is DA depletions in the lateral or ventrolateral caudate/putamen, rather than the nucleus accumbens, which have most conclusively been linked to this effect.
– It has been shown repeatedly that nucleus accumbens DA depletions or antagonism do not substantially impair appetite for food, or produce a general disruption of primary food motivation.
– In DA deficient mice, restoration of DA production in caudate putamen, but not nucleus accumbens, was able to rescue feeding behavior

The last several years have seen substantial theoretical developments related to the hypothesized behavioral functions of nucleus accumbens dopamine (DA). It has become evident to many investigators that there are conceptual limitations and empirical problems with the traditional DA hypothesis of “reward”. Even the use of the term “reward” itself often is problematic.

Researchers rarely define what they mean by “reward” when they are using it to describe a psychological process; some use it as though it were a synonym for “reinforcement”, or in reference to “appetite” or “primary motivation”, while still others employ it as a code word to mean “pleasure”. In some papers, the word “reward” seems to be used as a rather monolithic, all-encompassing term that refers to any and all aspects of appetitive learning, motivation and emotion, whether conditioned or unconditioned. Used in this way, the term reward is a rather blunt instrument.

These problems are not merely semantic, as it is difficult to test a hypothesis which maintains that a neurotransmitter mediates such an ill-defined set of functions. It has been suggested that it is advantageous to maintain the distinction between the terms reward and reinforcement; with this usage:
– reinforcement refers more directly to instrumental learning mechanisms
– while reward connotes the primary motivational

There are numerous problems with the hypothesis that brain dopamine (DA) systems, particularly in the nucleus accumbens, directly mediate the rewarding or primary motivational characteristics of natural stimuli such as food. Research and theory related to the functions of mesolimbic DA are undergoing a substantial conceptual restructuring, with the traditional emphasis on hedonia and primary reward yielding to other concepts and lines of inquiry. The present review is focused upon the involvement of nucleus accumbens DA in behavioral activation and effort-related processes.

Viewed from the framework of behavioral economics:
– the effects of accumbens DA depletions and antagonism on food-reinforced behavior are highly dependent upon the work requirements of the instrumental task
– DA depleted rats are more sensitive to increases in response costs (i.e., ratio requirements)
– Moreover, interference with accumbens DA transmission exerts a powerful influence over effort-related choice behavior.

Rats with accumbens DA depletions or antagonism:
– reallocate their instrumental behavior away from food-reinforced tasks that have high response requirements
– instead these rats select a less-effortful type of food-seeking behavior.

Nucleus accumbens DA and adenosine interact in the regulation of effort-related functions, and other brain structures (anterior cingulate cortex, amygdala, ventral pallidum) also are involved.

… alternative conceptual frameworks available for organizing what was known about the behavioral functions of DA systems, particularly mesolimbic DA:

– In a contemporary review of the behavioral functions of DA systems (Salamone, 1987), it was noted that DA in nucleus accumbens could be involved in the “exertion of effort”, and it was suggested that future experiments could “offer animals choices between various reinforcers that are associated with operants of varying difficulty” so that researchers could determine if the allocation of behavioral resources could be biased toward or away from more or less effortful responses by administration of dopaminergic drugs.
– This recognition of dopaminergic involvement in the exertion of effort, and effort-based choices related to cost benefit analyses, fit nicely with an emerging emphasis in the behavioral literature on work, response costs or constraints, and economic models of operant behavior.
– the idea that nucleus accumbens DA mediates the pleasure associated with positive reinforcers has been strongly challenged.
– Moreover, the potential role of DA systems in instrumental behavior or learning is not limited to situations involving appetitive motivation. There is considerable evidence that striatal mechanisms in general, and mesolimbic DA in particular, also participate in aspects of aversive learning and aversive motivation

Although imaging studies often are used to support the idea that nucleus accumbens mediates pleasure this appears to be oversimplified; indeed, research employing various imaging methods has demonstrated that the human nucleus accumbens also responds to stress, aversion and hyperarousal/irritability . Physiological and neurochemical studies in animals clearly indicate that DA neuron activity is not simply tied to the delivery of primary reinforcers or pleasurable stimuli. Rather, VTA neuron activity and DA release can be activated by a number of different appetitive and aversive conditions with changes seen across varying time scales, including tonic, slow phasic and fast phasic signals

Of course, one would not want to throw the baby out with the bathwater. It is apparent that mesolimbic DA participates in several complex functions related to aspects of instrumental behavior, learning and incentive motivation, and pavlovian/ instrumental interactions. The more difficult aspect of research and theory in this area is to ask – which specific aspects? For the purposes of this special issue, the present review will focus upon the role of nucleus accumbens DA in effort-related.

… another way of describing this effect of impaired DA transmission is to say that:

– rats with accumbens DA depletions are more sensitive than control animals to work-related response costs
– , or that they are less likely to trade high levels of work for food. In another study

Over the last two decades, several lines of evidence have converged to strengthen the original observation that the effects of interference with DA transmission interact powerfully with the work requirements of an instrumental task. Economic models of operant behavior have emphasized how a number of factors, including not only reinforcement value, but also conditions related to the characteristics of the instrumental response, can determine behavioral output … in terms of behavioral economics, the price of food reinforcement as a commodity is a cost/benefi t ratio expressed as the effort expended per unit of food value consumed. Optimal foraging theory was proposed to account for the observation that the amount of effort or time expended to obtain motivational stimuli was an important determinant of foraging choice, an idea that is still very influential in the ethology research today.

One of the ways of controlling work requirements in an operant schedule is to vary the ratio requirement (i.e., the number of times the animal must press the lever to receive a unit of reinforcement). The effects of the DA antagonist haloperidol on food-reinforced behavior were shown to be dependent upon the particular ratio schedule that was used. Accumbens DA depletions also produce effects that interact powerfully with the ratio requirement of the schedule employed…accumbens DA depletions exacerbate an effect known as ratio strain:

– In untreated animals, the overall relation between ratio size and response output is inverted-U shaped
– Up to a point, as ratio requirements get larger, animals adjust to this challenge by increasing response output.
– However, if the ratio requirement is high enough (i.e., if the cost is too high), the animal reaches the point at which additional responses being required actually tend to suppress responding.

…A completely different function is shown by rats with accumbens DA depletions, which are much more sensitive to the size of the ratio requirement. In behavioral economic terms, this pattern can be described as reflecting an increase in the elasticity of demand for food reinforcement. The term elasticity is widely used in economics, but price elasticity of demand refers to the sensitivity of consumption to changes in price.

DA-depleted rats showed markedly reduced consumption relative to the control group at higher ratio levels. In discussing the effects of dopaminergic manipulations on ratio performance, it is useful to mention the term “reinforcement efficacy”, which is sometimes used to describe the effects of drug manipulations on progressive ratio performance:
– With progressive ratio schedules, the ratio requirement increases as successive ratios are completed, and the “break point” is said to occur at the point at which the animal essentially ceases to respond.
– One can operationally define reinforcement efficacy in terms of the break point in a progressive ratio schedule (and also by measuring ratio strain in rats responding across different FR schedules).

Used in this manner, reinforcement efficacy is essentially being employed as an empirical descriptor of a particular behavioral outcome. Nevertheless, given the terminological problems mentioned above, it is worth emphasizing that the term “reinforcement efficacy” should not be used simply as a replacement for “reward”, nor should progressive ratio breakpoints be viewed as necessarily providing some direct and unambiguous measure related to the subjective pleasure produced by the stimulus. Changes in progressive ratio break points can reflect more than just changes in the appetitive motivational properties of a reinforcing stimulus.

For example, changing the kinetic requirements of the instrumental response (e.g., increasing the height of the lever) was shown to decrease progressive ratio break points. Although some researchers have maintained that the break point provides a direct measure of the appetitive motivational characteristics of a stimulus, it is, as explicitly stated in a classic review by Stewart (1974), more directly a measure of how much work the organism will do in order to obtain that stimulus.

Progressive ratio break points and measures of ratio strain are essentially outcomes that result from effort-related decision making processes. The animal is making a cost/benefit choice about whether or not to continue to respond, based partly on factors related to the reinforce itself, but also upon the work-related response costs and time constraints imposed by the ratio schedule. For these reasons, interpretations of the actions of drugs or lesions on progressive ratio break points should be done with caution, as should be the case for any individual task.

A drug that alters the break point could do so for many different reasons; it may be affecting functions related to the processing of reward value, or alternatively it could be affecting exertion of effort, or decision making processes.

The ability to exert effort, sustain work, overcome obstacles, and attain access to motivationally relevant stimuli is necessary for survival. But it is only part of the story. In a complex environment, which affords many opportunities for obtaining significant stimuli, and multiple paths for accessing them, organisms must make choices.

The variables that need to be evaluated to make these decisions are complex..Thus, both the magnitude and the organization of the ratio requirement appear to be critical determinants of the sensitivity of an operant schedule to the effects of accumbens DA depletions. In order to be sure that these results reflected the influence of ratio size, as opposed to other variables such as time, additional studies examined the effects of accumbens DA depletions on tandem schedules, in which a ratio requirement was attached to an interval requirement.

– In a conventional variable interval (VI) schedule, a time interval must elapse before the first response is reinforced, and the particular time interval varies around an average value. A tandem VI/FR schedule has an additional ratio requirement attached to the interval.
– Research employing tandem VI/FR schedules with varying combinationshas yielded a consistent pattern; accumbens DA depletions do not impair overall response output in rats responding on the conventional VI schedules (i.e., those requiring only one response after the interval), but do substantially reduce responding on the corresponding VI schedule with the higher ratio requirement attached.
– These results are consistent with research showing that accumbens DA antagonism did not impair performance on a progressive interval task and suggest that interval requirements per se do not pose a severe constraint to rats with compromised DA transmission in nucleus accumbens.

This serves to underscore the critical importance of ratio requirements as providing a work-related challenge to rats with accumbens DA depletions or antagonism.

In summarizing these results, Salamone and Correa (2002) stated that nucleus accumbens DA depletions appear to have two major effects:
(1) they reduce the response-enhancing effects that moderate-size ratio requirements have on operant responding (i.e., the ascending limb of the function relating ratio requirement to response output), and
(2) they enhance the response-suppressing effects that very large ratios have on operant responding (i.e., the descending limb of the function, enhancing ratio strain).

Furthermore, finer grained analyses of detailed patterns of responding reveal more insights into the behavioral manifestations of accumbens DA depletions.

– Accumbens DA depletions produce a slight reduction in the local rate of responding, as indicated by the distribution of inter-response times
– In addition, they enhance pauses in responding. The latter may indicate a fragmentation in the pattern of responding, a reduction in the ability to sustain uninterrupted response output, or a lack of engagement in the task

Recently, computational approaches have been used to analyze these effects of accumbens DA depletions on response rate. This relation between response output and DA function has been interpreted to mean that DA release in nucleus accumbens could provide a window of opportunistic drive but among the most important are those involving cost/benefi t assessments based upon effort and reinforcement value. Considerable evidence indicates that nucleus accumbens DA, along with other transmitters and structures, participates in the overall circuitry that regulates effort-based choice behavior.

One of the procedures that has been used to assess the contribution of accumbens DA to response allocation and effort-related choice behavior is a task that offers rats the option of either lever pressing to obtain a relatively preferred food, or approaching and consuming a less preferred food (lab chow) that is concurrently available in the chamber:

– Well trained rats under baseline conditions typically get most of their food by lever pressing, and consume only small quantities of chow
– Low-to-moderate doses of DA antagonists, which block either D1 or D2 family receptor subtypes, produce a substantial alteration of response allocation in rats performing on this task. The DA antagonists…all decreased lever pressing for food but substantially increased intake of the concurrently available chow…Although DA antagonists have been shown to reduce lever pressing and increase chow intake, appetite suppressants from different classes, including amphetamine failed to increase chow intake at doses that suppressed lever pressing. Similarly, pre-feeding to reduce food motivation was shown to suppress both lever pressing and chow intake. Furthermore, attachment of higher ratio requirements caused animals that were not drug treated to shift from lever pressing to chow intake (Salamone et al., 1997), indicating that this task is sensitive to work load.

. these findings demonstrate that interference with DA transmission does not simply reduce appetite, but does act to alter response allocation between alternative sources of food that can be obtained through different instrumental responses:
– The shift from lever pressing to chow intake in rats performing on this task is associated with DA depletions in nucleus accumbens, but not the neostriatum
– Although it has been suggested that caudate/ putamen DA may have some types of motivational functions related to feeding,
– DA depletions in anteroventromedial neostriatum, which is dorsal to nucleus accumbens, had no behavioral effect,
– while ventrolateral neostriatal DA depletions produced severe motor impairments that merely decreased both lever pressing and feeding
– In contrast, decreases in lever pressing and increases in chow intake occur as a result of accumbens DA depletions, as well as intra-accumbens injections of D1 or D2 antagonists
– The shift from lever pressing to chow intake on this task has been shown to occur in rats if D1 or D2 family antagonist are injected into the medial core, lateral core, or dorsal shell subregions of the accumbens

Thus, although lever pressing is decreased by accumbens DA antagonism or depletions, the rats show a compensatory reallocation of behavior and select a new path to an alternative food source…Thus, dopaminergic manipulations did not alter the preference for the high density of food reward over the low density, and did not affect discrimination or memory processes related to arm preference…low doses of DA antagonists and accumbens DA depletions cause animals to reallocate their instrumental response selection based upon the response requirements of the task, and select lower effort alternatives for obtaining rewards … Adenosine A2A receptors also are involved in aspects of behavioral activation and effort-related processes…Consistent with the observation that an adenosine A2A agonist could produce actions similar to those resulting from DA depletion or blockade,…. This suggests that A2A antagonists exert an overall greater effect than A1 antagonists on effort-related functions of nucleus accumbens.

Research on the brain mechanisms involved in effort-related processes may lead to new ways of thinking about behavioral analysis and theory in behavioral economics. One of the contributions that behavioral neuroscience can make to behavioral theory is to use manipulations (e.g., drugs, lesions) that dissociate complex behavioral processes into component parts (Salamone et al., 2007). In this regard, it is useful to consider that a given parameter that is generated from curve-fitting analyses, when viewed in terms of its biological characteristics, has many factors that contribute to it.

…Within the last few years, there has been considerable progress in characterizing the functional anatomy underlying this important aspect of motivation and decision making. Several transmitters across multiple brain regions are involved in effort-related functions, and researchers are only beginning to piece together the complex puzzle of all the potential brain systems that are involved. Presently, the specific way in which each structure contributes to the overall function of the system is unclear. It is uncertain which brain areas are involved in the exertion of effort, or the perception of effort, vs. the actual decision making process itself.

For example, it is possible that nucleus accumbens is involved in the actual decision making processes, but it also is possible that it is mainly involved in regulating energy output, or setting effort-related constraints or feedback that in turn influences decisions made at other levels in the system. If the latter is true, then it is possible that the decision making effects of drug or lesion manipulations of nucleus accumbens are an outcome reflecting the constraints that are set after compromised DA function in accumbens, rather than a direct effect upon decision making processes per se.

Future research will be necessary to tease apart these distinct aspects of effort-related function. In addition to providing insights into aspects of animal behavior and natural motivation, research on effort-related processes also has clinical implications.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s