marginal utility, value and the brain

Economics assumes the principle of diminishing marginal utility, i.e. the utility of a good increases more and more slowly as the quantity consumed increases (Wikipedia). Mathematically, it means that the value of a monetary gain is not a linear function of the monetary value. Before Bernouilli St-Petersburg Paradox (1738]1954), the expected value of a possible gamble was construed as the product of the objective (for instance, monetary) value of its outcomes and its probability. Suppose, then, a gambler is offered the following lottery:

A fair coin is tossed. If the outcome is heads, the lottery ends and you win 2$. If the outcome is tail, toss the coin again. It the outcome is heads, the lottery ends and you win 4$, etc. If the nth outcome is heads, you win 2n.

Summing the products of probability and value leads to an infinite expected value:

(0.5 x 2) + (0.25 x 4) + (0.125 x 8)…. =
1+1+1 …

After 30 tosses, the gambler could win more than 1 billion $. How much would it be worth paying for a ticket? If a rational agent maximizes expected value, he or she must be willing to buy a ticket for this lottery at any finite price, considering that the expected value of this prospect if infinite. But, as Hacking pointed out, “few of us would pay even $25 to enter such a game” (Hacking, 1980). When Bernoulli offered scholars in St-Petersburg to play this lottery, nobody was interested in it. Bernoulli concluded that the utility function is not linear, but logarithmic. Hence the subjective value of 10$ is different, depending whether you are Bill Gates or a homeless. Bernoulli’s discussion of the St-Petersburg paradox is often considered as one of the first economic experiment (Roth, 1993, p. 3).

A new study in neuroeconomics (Tobler et al.) indicates that the brain's valuation mechanisms follow this principle. Subjects in the experiments had to learn whether a particular abstract shape--shown on a computer screen--predicts a monetary reward (a picture of a 20 pence coin) or not (scrambled picture of the coin). If the utility of money has a diminishing marginal value, then money should be more important for poorer people than for richer. "More important" meaning that the former would learn reward prediction partterns faster and would display more activity in reward-related area. Bingo! That's exactly what happened. Midbain dopaminergic regions were more solicited in the poorer. The valuation mechanisms obey diminishing marginal utility.

This suggest that midbain dopaminergic systems (about which I blogged earlier; see also references at the end of this post) are the seat of our natural rationality, or at least one of its major component. These systems compute utility, stimulate motivation and attention, send reward-prediction error signals, learn from these signals and devise behavioral policies. They do not encode anticipated or experienced utility (other zones are recruited for these: the amygdala and nucleus accumbens for experienced utility, the OFC for anticipated utility, etc.), but decision utility, the cost/benefits analysis of a possible decision.


References

• Bernoulli, D. (1738]1954). Exposition of a new theory on the measurement of risk. Econometrica, 22, 23-36.
• Hacking, I. (1980). Strange expectations. Philosophy of Science, 47, 562-567.
• Roth, A. E. (1993). On the early history of experimental economics. Journal of the History of Economic Thought, 15, 184-209.
• Tobler, P. N., Fletcher, P. C., Bullmore, E. T., & Schultz, W. (2007). Learning-related human brain activations reflecting individual finances. Neuron, 54(1), 167-175.

On dopaminergic systems:

• Ahmed, S. H. (2004). Neuroscience. Addiction as compulsive reward prediction. Science, 306(5703), 1901-1902.
• Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129.
• Berridge, K. C. (2003). Pleasures of the brain. Brain and Cognition, 52(1), 106.
• Berridge, K. C., & Robinson, T. E. (1998). What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev, 28(3), 309-369.
• Cohen, J. D., & Blum, K. I. (2002). Reward and decision. Neuron, 36(2), 193-198.
• Daw, N. D., & Doya, K. (2006). The computational neurobiology of learning and reward. Curr Opin Neurobiol, 16(2), 199-204.
• Daw, N. D., & Touretzky, D. S. (2002). Long-term reward prediction in td models of the dopamine system. Neural Comput, 14(11), 2567-2583.
• Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36(2), 285-298.
• Di Chiara, G., & Bassareo, V. (2007). Reward system and addiction: What dopamine does and doesn't do. Curr Opin Pharmacol, 7(1), 69-76.
• Egelman, D. M., Person, C., & Montague, P. R. (1998). A computational role for dopamine delivery in human decision-making. J Cogn Neurosci, 10(5), 623-630.
• Floresco, S. B., & Magyar, O. (2006). Mesocortical dopamine modulation of executive functions: Beyond working memory. Psychopharmacology (Berl), 188(4), 567-585.
• Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science, 306(5703), 1940-1943.
• Joel, D., Niv, Y., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Netw, 15(4-6), 535-547.
• Kakade, S., & Dayan, P. (2002). Dopamine: Generalization and bonuses. Neural Netw, 15(4-6), 549-559.
• McCoy, A. N., & Platt, M. L. (2004). Expectations and outcomes: Decision-making in the primate brain. J Comp Physiol A Neuroethol Sens Neural Behav Physiol.
• Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004). Computational roles for dopamine in behavioural control. Nature, 431(7010), 760.
• Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nat Neurosci, 9(8), 1057-1063.
• Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004). Dopamine neurons can represent context-dependent prediction error. Neuron, 41(2), 269-280.
• Nieoullon, A. (2002). Dopamine and the regulation of cognition and attention. Progress in Neurobiology, 67(1), 53.
• Niv, Y., Daw, N. D., & Dayan, P. (2006). Choice values. Nat Neurosci, 9(8), 987-988.
• Niv, Y., Duff, M. O., & Dayan, P. (2005). Dopamine, uncertainty and td learning. Behav Brain Funct, 1, 6.
• Redish, A. D. (2004). Addiction as a computational process gone awry. Science, 306(5703), 1944-1947.
• Schultz, W. (1999). The reward signal of midbrain dopamine neurons. News Physiol Sci, 14(6), 249-255.
• Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599.
• Schultz, W., & Dickinson, A. (2000). Neuronal coding of prediction errors. Annu Rev Neurosci, 23, 473-500.
• Self, D. (2003). Neurobiology: Dopamine as chicken and egg. Nature, 422(6932), 573-574.
• Suri, R. E. (2002). Td models of reward predictive responses in dopamine neurons. Neural Netw, 15(4-6), 523-533.
• Tom, S. M., Fox, C. R., Trepel, C., & Poldrack, R. A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315(5811), 515-518.
• Ungless, M. A. (2004). Dopamine: The salient issue. Trends Neurosci, 27(12), 706.