It would seem, then, that we need to look beyond the merely 'technical' level if we wish to solve the Prisoner's Dilemma. However, this is not to say that rationality is no use to us but that, rather, our conventional understanding of what it means to be 'rational' is simply too narrow and needs broadening a little. Thus, we come back to the quote at the start of this article. Irrationality, says Douglas Hofstadter, is the 'square root' - ie, the cause - of all evil. Fair enough - but how can we distinguish, once and for all, what is rational from what is irrational? A possible answer lies in Hofstadter's concept of superrationality - that is, looking outside one's own decision and taking into account the decisions of others too, and consequently making the decision that one would hope they would also make. In other words, the 'superrationalist' thinks 'globally' - in the wider interest - rather than 'locally', simply with his/her own interest in mind.

After a decade of painstaking research, federal and university scientists have reconstructed the 1918 influenza virus that killed 50 million people worldwide. Like the flu viruses now raising alarm bells in Asia, the 1918 virus was a bird flu that jumped directly to humans, the scientists reported. To shed light on how the virus evolved, the United States Department of Health and Human Services published the full genome of the 1918 influenza virus on the Internet in the GenBank database.

This is extremely foolish. The genome is essentially the design of a weapon of mass destruction. No responsible scientist would advocate publishing precise designs for an atomic bomb, and in two ways revealing the sequence for the flu virus is even more dangerous.

These ideas fed into our work on indirect reciprocity, a concept that was first introduced by Robert Trivers in a famous paper in the 1970s. I recall that he mentioned this idea obliquely when he wrote about something he called "general altruism". Here you give something back not to the person to whom you owe something, but to somebody else in society. He pointed out that this also works with regard to cooperation at a high level. Trivers didn't go into details, because at the time it was not really at the center of his thinking. He was mostly interested in animal behavior, and so far indirect reciprocity has not been proven to exist in animal behavior. It might exist in some cases, but ethologists are still debating the pros and cons.

In human societies, however, indirect reciprocity has a very striking effect. There is a famous anecdote about the American baseball player Yogi Berra, who said something to the effect of, "I make a point of going to other people's funerals because otherwise they won't come to mine." This is not as nonsensical as it seems. If a colleague of the university, for instance, goes faithfully to every faculty member's funeral, then the faculty will turn out strongly at his. Others reciprocate. It works. We think instinctively in terms of direct reciprocation — when I do something for you, you do something for me — but the same principle can apply in situations of indirect reciprocity. I do something for you and somebody else helps me in return.

There is, to be sure, a genuine problem in the phenomenon of quantum measurement, but I will not discuss it here. It concerns introspective systems, where subject = object so that the basic conception of a single subject observing an ensemble of objects must be modified.
- David Finkelstein

It is time we steered by the stars, not by the lights of each passing ship.
- General Omar Bradley

Co-operation has brought the human race a long way in a staggeringly short time

“Our everyday life is much stranger than we imagine, and rests on fragile foundations.” This is the intriguing first sentence of a very unusual new book about economics, and much else besides: “The Company of Strangers”, by Paul Seabright, a professor of economics at the University of Toulouse. (The book is published by Princeton University Press.) Why is everyday life so strange? Because, explains Mr Seabright, it is so much at odds with what would have seemed, as recently as 10,000 years ago, our evolutionary destiny. It was only then that “one of the most aggressive and elusive bandit species in the entire animal kingdom” decided to settle down. In no more than the blink of an eye, in evolutionary time, these suspicious and untrusting creatures, these “shy, murderous apes”, developed co-operative networks of staggering scope and complexity—networks that rely on trust among strangers. When you come to think about it, it was an extraordinarily improbable outcome.

I teach a seminar titled "Ideas of Human Nature" in which I restrict enrollment to 15 students. Others typically want to get in, and yet much of the class's popularity stems from the benefits that come from keeping it small. Most students understand the advantages of small classes, and they wouldn't want everyone who wishes admittance to get in; just themselves! If everyone gets in, the class becomes too big and discussion is inhibited, to everyone's disadvantage. And so, each year I find myself in the difficult position of telling a number of students that there simply isn't room for them. Each student turned away from this class understands the logic, but nonetheless, each would like the limit to be expanded -- by just one.

"The major shift in human evolution is from behaving like an animal struggling to survive to behaving like an animal choosing to evolve. In fact, in order to survive, we have to evolve. And to evolve, we need a new kind of thinking and a new kind of behavior, a new ethic and a new morality. It will be that of the evolution of everyone rather than the survival of the fittest."
- Jonas Salk

A human being is part of the whole, called by us 'Universe,' a part limited in time and space. He experiences himself, his thoughts and feelings, as something separated from the rest, a kind of optical delusion of his consciousness. This delusion is a kind of prison for us, restricting us to our personal desires and affection for a few persons nearest to us. Our task must be to free ourselves from this prison by widening our circles of compassion to embrace all living creatures and the whole of nature in its beauty.
- Albert Einstein

Beware of altruism. It is based on self-deception, the root of all evil.
- Lazarus Long

It's my understanding that Hofstadterian superrationality is not generally
accepted within the game theory research community as a valid principle of
decision making. Do you have any information to the contrary, or some
other reason to think that it will be commonly used by transhumans?

You yourself articulated, very precisely, the structure underlying Hofstadterian superrationality: "Expected utility of a course of action is defined as the average of the utility function evaluated on each possible state of the multiverse, weighted by the probability of that state being the actual state if the course was chosen." The key precise phrasing is "weighted by the probability of that state being the actual state if the course was chosen". This view of decisionmaking is applicable to a timeless universe; it provides clear recommendations in the case of, e.g., Newcomb's Paradox.

Superrationality----------------

Not really intuition. I guess I have to go into detail after all.

I'll discuss a "toy model" of rational choice, it is instructive, but clearly not realistic.

The model will be as follows: I imagine two rational beings in two rooms, and there is a button on each wall. If one of the two presses the button, the other is killed. If they don't, they survive. If both of them press the button, one of them is killed at random.

What course of action maximizes each individuals probability of survival?

This seems like a question that has a definite answer, and it is surprising that when you do the analysis, you run into a problem. Well, I won't talk aboutit, I'll do it.

That would be a good idea.
- Gandhi (when asked what he thought of western civilisation)