Collective intelligence

In the beginning, there was music. Childhood and young adulthood floated by to a soundtrack of lyrics and rhythms and searing guitar riffs that consumed you, became you, constituted your identity, galvanized your intent, spoke your soul.

But time passes, classrooms fade to cubicles, and a vast landscape of new music turns foreign and unexplored. For Jeff Hersh, 31, the stereo came to double as Proust's madeleine, its purpose to invoke memories rather than create them.

"Finding music was easier when I was younger," says Hersh, a vice president at Smith Barney in New York. "In college I lived in a fraternity house with 70 guys all around me at all times, listening to various kinds of music. But as you get older, you work more, you get isolated."

Then in November, a friend told Hersh about Pandora.com, an inventive "Internet radio" website that generates music streams — "stations" — based on one's favorite artists or songs. He started his own private thread of music that was a combination of Neil Young and Pearl Jam, Hersh says, and in an hour he heard more new music he liked than he had in the last decade, much of it from obscure bands that shared musical traits with Young and Pearl Jam.

At Google, we're constantly trying to find new ways to organize the world's information, including information relevant to our business. Building on the ideas of Friedrich Hayek and the Iowa Electronic Markets, a few Googlers (Doug Banks, Patri Friedman, Ilya Kirnos, Piaw Na and me, with some help from Hal Varian), set up a predictive market system inside the company.

The markets were designed to forecast product launch dates, new office openings, and many other things of strategic importance to Google. So far, more than a thousand Googlers have bid on 146 events in 43 different subject areas (no payment is required to play).

We designed the market so that the price of an event should, in theory, reflect a consensus probability that the event will occur. To determine accuracy of the market, we looked at the connection between prices of events and the frequency with which they actually occurred. If prices are correct, events priced at 10 cents should occur about 10 percent of the time.

In the graph below, the X-axis indicates the price ranges for the group. The orange line represents the average price, which is how often outcomes in that group should actually happen according to market prices. The purple line is how often they did happen. Ideally these would be equal, and as you can see they're pretty close. So our prices really do represent probabilities - very exciting!

The Netscape IPO wasn't really about dot-commerce. At its heart was a new cultural force based on mass collaboration. Blogs, Wikipedia, open source, peer-to-peer - behold the power of the people.

Ten years ago, Netscape's explosive IPO ignited huge piles of money. The brilliant flash revealed what had been invisible only a moment before: the World Wide Web. As Eric Schmidt (then at Sun, now at Google) noted, the day before the IPO, nothing about the Web; the day after, everything.

Computing pioneer Vannevar Bush outlined the Web's core idea - hyperlinked pages - in 1945, but the first person to try to build out the concept was a freethinker named Ted Nelson who envisioned his own scheme in 1965. However, he had little success connecting digital bits on a useful scale, and his efforts were known only to an isolated group of disciples. Few of the hackers writing code for the emerging Web in the 1990s knew about Nelson or his hyperlinked dream machine.

At the suggestion of a computer-savvy friend, I got in touch with Nelson in 1984, a decade before Netscape. We met in a dark dockside bar in Sausalito, California. He was renting a houseboat nearby and had the air of someone with time on his hands. Folded notes erupted from his pockets, and long strips of paper slipped from overstuffed notebooks. Wearing a ballpoint pen on a string around his neck, he told me - way too earnestly for a bar at 4 o'clock in the afternoon - about his scheme for organizing all the knowledge of humanity. Salvation lay in cutting up 3 x 5 cards, of which he had plenty.

Although Nelson was polite, charming, and smooth, I was too slow for his fast talk. But I got an aha! from his marvelous notion of hypertext. He was certain that every document in the world should be a footnote to some other document, and computers could make the links between them visible and permanent. But that was just the beginning! Scribbling on index cards, he sketched out complicated notions of transferring authorship back to creators and tracking payments as readers hopped along networks of documents, what he called the docuverse. He spoke of "transclusion" and "intertwingularity" as he described the grand utopian benefits of his embedded structure. It was going to save the world from stupidity.

I believed him. Despite his quirks, it was clear to me that a hyperlinked world was inevitable - someday. But looking back now, after 10 years of living online, what surprises me about the genesis of the Web is how much was missing from Vannevar Bush's vision, Nelson's docuverse, and my own expectations. We all missed the big story. The revolution launched by Netscape's IPO was only marginally about hypertext and human knowledge. At its heart was a new kind of participation that has since developed into an emerging culture based on sharing. And the ways of participating unleashed by hyperlinks are creating a new type of thinking - part human and part machine - found nowhere else on the planet or in history.

Not only did we fail to imagine what the Web would become, we still don't see it today! We are blind to the miracle it has blossomed into. And as a result of ignoring what the Web really is, we are likely to miss what it will grow into over the next 10 years. Any hope of discerning the state of the Web in 2015 requires that we own up to how wrong we were 10 years ago.

Although not approved by the search company, information on crime, sexual predators and cheap gas prices is graphically displayed.

Tracking sexual predators in Florida. Guiding travelers to the cheapest gas. Pinpointing $ 1,500 studio apartments for rent in Manhattan.

Geeks, tinkerers and innovators are crashing the Google party, having discovered how to tinker with the search engine’s mapping service to graphically illustrate vital information that might otherwise be ignored, overlooked or not perceived as clearly.

Yahoo and other sites also offer maps, but Google Inc.’ s 4-month-old mapping service is more easily accessible and manipulated by outsiders, the tinkerers say.

In this expanded interview transcript, inventor Ray Kurzweil discusses birth, death, and the potential offered by non-biological thinking processes.
By: Lucas Conley

Fast Company: First off, without death, CEOs will never give up their jobs. There won't be any succession plans.

Ray Kurzweil: I don't think we need to kill people off to provide opportunity for new leadership and creativity. The marketplace of ideas and technologies is going to expand -- it has been for years. Look at the computer industry. 60 years ago it was a handful of research projects, and now it's a trillion-dollar industry.

FC: But biotech? Who's to say how quickly it will advance?

Kurzweil: A lot of people say you can't really tell the future, and there are certain things that are hard to predict. What will Google's stock be three years from now? That's hard to predict. But if you ask me what it will cost to sequence a base pair of DNA in 2010 or the cost to move a megabyte of data wirelessly in 2015, those things turn out to be remarkably predictable.

Suddenly, the computer world is interesting again. The last three months of 2004 brought more innovation, faster, than users have seen in years. The recent flow of products and services differs from those of previous hotly competitive eras in two ways. The most attractive offerings are free, and they are concentrated in the newly sexy field of "search."

Google, current heavyweight among systems for searching the Internet, has not let up from its pattern of introducing features and products every few weeks. Apart from its celebrated plan to index the contents of several university libraries, Google has recently released "beta" (trial) versions of Google Scholar, which returns abstracts of academic papers and shows how often they are cited by other scholars, and Google Suggest, a weirdly intriguing feature that tries to guess the object of your search after you have typed only a letter or two. Give it "po" and it will show shortcuts to poetry, Pokémon, post office, and other popular searches. (If you stop after "p" it will suggest "Paris Hilton.") In practice, this is more useful than it sounds.

Microsoft, heavyweight of the rest of computerdom, has scrambled to catch up with search innovations from Google and others. On Dec. 10, a company official made a shocking disclosure. For years Microsoft had emphasized the importance of "WinFS," a fundamentally new file system that would make it much easier for users to search and manage information on their own computers. Last summer, the company said that WinFS would not be ready in time for inclusion with its next version of Windows, called Longhorn. The latest news was that WinFS would not be ready even for the release after that, which pushed its likely delivery at least five years into the future. This seemed to put Microsoft entirely out of the running in desktop search. But within three days, it had released a beta version of its new desktop search utility, which it had previously said would not be available for months.

Meanwhile, a flurry of mergers, announcements and deals from smaller players produced a dazzling variety of new search possibilities. Early this month Yahoo said it would use the excellent indexing program X1 as the basis for its own desktop search system, which it would distribute free to its users. The search company Autonomy, which has specialized in indexing corporate data, also got into the new competition, as did Ask Jeeves, EarthLink, and smaller companies like dTSearch, Copernic, Accoona and many others.

I have most of these systems running all at once on my computer, and if they don't melt it down or blow it up I will report later on how each works. But today's subject is the virtually unpublicized search strategy of another industry heavyweight: I.B.M.

From Jon Udell's blog:

Dave Winer today points to an Scott Rosenberg's excellent take on Google's new library venture. Scott concludes:

The public has a big interest in making sure that no one business has a chokehold on the flow of human knowledge. As long as Google's amazing project puts more knowledge in more hands and heads, who could object? But in this area, taking the long view is not just smart -- it's ethically essential. So as details of Google's project emerge, it will be important not just to rely on Google's assurances but to keep an eye out for public guarantees of access, freedom of expression and limits to censorship. Scott Rosenberg

Google, the operator of the world's most popular Internet search service, plans to announce an agreement today with some of the nation's leading research libraries and Oxford University to begin converting their holdings into digital files that would be freely searchable over the Web.

It may be only a step on a long road toward the long-predicted global virtual library. But the collaboration of Google and research institutions that also include Harvard, the University of Michigan, Stanford and the New York Public Library is a major stride in an ambitious Internet effort by various parties. The goal is to expand the Web beyond its current valuable, if eclectic, body of material and create a digital card catalog and searchable library for the world's books, scholarly papers and special collections.

It took a million years to move from counting pebbles to the elaborations of quantum mechanics. Certainly this was an arduous migration of the multitude -- not a private party of physicists, but the Long March of the entire human race.
- Anonymous

Google Inc. plans to announce on Thursday that it is adding a new search service aimed at scientists and academic researchers.

Google Scholar, which was scheduled to go online Wednesday evening at scholar.google.com, is a result of the company's collaboration with a number of scientific and academic publishers and is intended as a first stop for researchers looking for scholarly literature like peer-reviewed papers, books, abstracts and technical reports.

The tech guru sees a "new economic system" in the unconscious cooperation embodied by Google links and Amazon lists.

Howard Rheingold is on the hunt again. With his last book, Smart Mobs: The Next Social Revolution, in 2001, the longtime observer of technology trends made a persuasive case that pervasive mobile communications, combined with always-on Internet connections, will produce new kinds of ad-hoc social groups. Now, he's starting to take the leap beyond smart mobs, trying to weave some threads out of such seemingly disparate developments as Web logs, open-source software development, and Google.

At the same time, Rheingold is worried that established companies could quash such nascent innovations as file-sharing -- and potentially put the U.S. at risk of falling behind the rest of the world. He recently spoke with Robert D. Hof, BusinessWeek's Silicon Valley bureau chief. Here are excerpts from their conversation:

Q: Where do you see the social revolution you've been talking about going next?

A: It's too early to say. The question is: What does it point toward? Some kind of collective action...in which the individuals aren't consciously cooperating. A market is a great example as a mechanism for determining price based on demand. People aren't saying, "I'm contributing to the market," they say they're just selling something. But it adds up.

With experimental markets, workers are betting on their company's future — and moving in on the boss's domain.

The end of management just might look something like this. You show up for work, boot up your computer and log onto your company's Intranet to make a few trades before getting down to work. You see how your stocks did the day before and then execute a few new orders. You think your company should step up production next month, and you trade on that thought. You sell stock for the production of 20,000 units and buy stock that represents an order for 30,000 instead. All around you, as co-workers arrive at their cubicles, they too flick on their computers and trade.

Together, you are buyers and sellers of your company's future. Through your trades, you determine what is going to happen and then decide how your company should respond. With employees in the trading pits betting on the future, who needs the manager in the corner office?

The cell phone may be bringing us into a new renaissance, but it may end up differently than what we're expecting. Instead of becoming more empowered as individuals, we may give up on the notion of individuality altogether.

The Renaissance -- the great big one they taught us about in school -- is known for a lot of great inventions: perspective painting, the printing press, ships that could circumnavigate the earth, modern banking and even the sonnet. What we tend to forget about the 15th and 16th centuries, though, is that this was also when we invented the "individual."

Sure, we knew that people existed in their individual bodies for a long time. Even cavemen knew that hitting the guy over there meant hitting someone else. But people were so highly identified with their tribes, clans or fiefdoms, that they didn't really think of themselves as individuals. Anyone who was a true individual was pretty much an outcast -- either banished, mutant, a leper or, at best, a shaman, whose individuality was as much a curse as a blessing.

No, the real individual, as he or she is known today, was born as a 'he' during the renaissance. The mad genius Dr. Faustus is often cited as the first full-fledged individual character in drama; he's the scientist who has reached the height of knowledge and capability and must make a deal with the devil in order to reach to even higher levels of power.