Saturday, March 9, 2013

Dopamine and Reward Prediction Error

I am back from the IBAGS conference and full of new information! I plan to blog about tons of amazing things over the next month or so, but today we'll start with some foundation building.

Dopamine nails (source)
The IBAGS (international basal ganglia society) meeting is all about the basal ganglia (which includes the striatum), and as you may know, dopamine is a super important molecule for the proper function of the striatum (it is the dopamine cells that die in Parkinson's Disease).

There were many fantastic talks during the IBAGS meeting and almost a third of them showed the exact same figure on one of their slides. So much so that everyone would start to laugh when someone showed it. And as you may have guessed, it is about dopamine. Here it is:

Schultz 1998 Figure 2
This figure is the basis for the belief that dopamine represents a 'reward prediction error'.  Let me explain. The scattered dots on the lower half of each panel represent action potentials from individual dopaminergic neurons. The x axis it time in seconds. The black columns above them are a histogram showing how much firing is going on at each point in time. When the black columns are tall, there was more dopamine neuron firing. You can see that the height of the black columns matches up with the density of the scattered dots below them.

During 'reward learning', an animal is trained to associate a stimulus (like a tone or a flash of light) with getting a reward (like a drop of water or juice). These three panels show how dopamine responds to this whole process. The first panel shows that when there is no stimulus (CS) and the reward is a surprise, the dopamine neurons respond very strongly to it. The second panel shows that when there is a stimulus that tells the animal that a reward is soon to come, the dopamine neurons respond to the stimulus, but not to the reward. Finally the third panel shows that when there is a stimulus the dopamine neurons respond to it, but if the reward (R) never comes, the dopamine neurons actually stop firing when the reward should have happened.

What is so fascinating about this is that it shows dopamine neurons do not just fire in response to reward, they encode the actual reward with respect to the expected reward. In the author's words:
"Dopamine neurons report rewards relative to their prediction rather than signaling primary rewards unconditionally (Fig. 2). The dopamine response is positive (activation) when primary rewards occur without being predicted. The response is nil when rewards occur as predicted. The response is negative (depression) when predicted rewards are omitted. Thus dopamine neurons report primary rewards according to the difference between the occurrence and the prediction of reward, which can be termed an error in the prediction of reward..." Schultz 1998
This finding is so important to researchers now because it shows that dopamine neurons can encode learning rules. Dopamine neurons constantly and dynamically tell the rest of the brain which stimuli lead to reward, and which stimuli don't. The implications here for pathological learning are huge as well. Mis-signalling in dopamine neurons could lead to an inability to tell what is rewarding and what is not.

© TheCellularScale
Schultz W (1998). Predictive reward signal of dopamine neurons. Journal of neurophysiology, 80 (1), 1-27 PMID: 9658025


  1. I'm curious if there were any dissenting voices regarding dopamine in RPE? The timing argument put forth by Peter Redgrave (e.g., The short-latency dopamine signal:
    a role in discovering novel actions?
    ) and associated evidence is quite convincing.

  2. Great question. I don't know of anyone who disagrees with this exactly, but new research keeps showing that dopamine does more than this. Soon I'll blog about Hikosaka's findings that some dopamine neurons encode the 'importance' or salience of a signal while a separate group of dopamine neurons encode value.

  3. Hah, I was at the Cosyne workshop on foraging and literally every single talk had this slide too... Then Michael Platt got up and tried to convince everyone that at future foraging meetings, that we'd all be showing a figure from his paper on neural foraging ;)

  4. I'm glad you liked my dopamine nails! ;-)