The possibility of developing truly intelligent machines, and their potential to be friend or foe to humanity, gets the Hollywood treatment in a new blockbuster film I, Robot, which opens in the US on Friday.

At the heart of the movie are Isaac Asimov's "Three Laws of Robotics", invented as a simple, but immutable moral code for robots See below. The film's plot revolves around an apparent breaking of the laws, when a robot is suspected of murdering a famous scientist.

Yet, while the movie is an enjoyable action romp, robotics and artificial intelligence experts admit they are a long way from having to worry about such rules yet. "The difficulty is building something that would understand them," says Alan Bundy, at Edinburgh University's Artificial Intelligence Institute in the UK. "That is well beyond the state of the art at the moment."

The scientists and engineers spearheading the creation of artificial beings and bionic people are responding to the magnetism of the technological imperative, the pull of a scientific problem as challenging as any imaginable.

Fascinating scientific puzzle though it is, the creation of artificial beings is also expected to meet important needs for society and individuals. Industrial robots are already widely used in factories and on assembly lines. Robots for hazardous duty, from dealing with terrorist threats to exploring hostile environments, including distant planets, are in place or on the drawing boards. Such duty could include military postings because there is a longstanding interest in self-guided battlefield mechanisms that reduce the exposure of human soldiers, and in artificially enhanced soldiers with increased combat effectiveness. (For this reason, the Department of Defense, largely through its research arm — the Defense Advanced Research Projects Agency — is the main U.S. funding source for research in artificial creatures.) Artificial creatures can also be used in less hostile environments: homes, classrooms, and hospitals and rest homes, serving as all-purpose household servants, helping to teach, and caring for the ill or elderly.

Among these possibilities, the connection between artificial creatures and human implants might be the most important because it promises enormous medical benefits. This connection might be the single greatest motivation to develop artificial beings. Yet regardless of their potential good uses, and apart from any issues of blasphemy, we have concerns about robots and androids. One fear is that the limitations we think to design out of our creations, from cosmetic deficiencies to the existential realities of illness and death, are essential human attributes, and that to abandon them is somehow to abandon our humanity. Something in us, it seems, fears perfection, and artificial beings threaten us with an unwelcome perfection, expressed as rigid unfeeling precision.

There is another menace first conveyed nearly 200 years ago in “Frankenstein,” and now more compelling than ever: the fear that technology will grow out of control and diminish humanity for all of us. That concern is hardly limited to artificial creatures. It appears in many arenas — the loss of privacy associated with new forms of surveillance and data manipulation; the depersonalization of human relationships; the incidence of human-made ecological disaster; the growing gap between the world’s technological “haves” and “have-nots.” It is especially and deeply unsettling, however, to contemplate the literal displacement of humanity by beings made in the human image, only better.

Since Weld and Etzioni's "The First Law of Robotics" at AAAI in 1994, there has been growing concern with the safety of deploying intelligent agents in the real world. Perhaps HAL in Kubrick's 2001: A Space Odyssey is the best image of such an agent gone wrong.

One area often missing from such discussions is the safety of learning agents. This is an important omission since learning/adaptation is a component in most definitions of what it means to be an agent. Some recent work has begun to address some of the issues involved, but the field is still in the initial stages of defining the problem.

A safe agent is one which can efficiently find and execute acceptable solutions for its target problems. Learning can adversely affect which problems the agent can solve, the efficiency with which they come up with plans to solve them, and the quality of the solutions. Thus learning can cause a "safe" agent to become "unsafe."

Since we would like the agents to learn from their environment, either the agent must be able to quickly/cheaply check that its new learning hasn't made it unsafe or the learning method must guarantee that it will preserve the agent's safeness.

Unfortunately, it is unlikely that for problems in general, that any learning method would be able to guarantee it won't decrease coverage, efficiency, and solution quality. So, at least initially, we must identify learning methods that make guarantees with respect to at least one of the performance dimensions and/or with respect to some restricted class of problems. While it may not be possible to guarantee monotonicity for these performance dimensions, we may be able to bind the degradation for some of these.

In this symposium we are interested in peoples' experiences with agent learning going wrong, how to prevent it, and with what end-users both want and fear from learning agents.

Even before the advent of Artificial Intelligence, science fiction writer Isaac Asimov recognized that an agent must place the protection of humans from harm at a higher priority than obeying human orders. Inspired by Asimov, we pose the following fundamental questions: (1) How should one formalize the rich, but informal, notion of "harm"? (2) How can an agent avoid performing harmful actions, and do so in a computationally tractable manner? (3) How should an agent resolve conflict between its goals and the need to avoid harm? (4) When should an agent prevent a human from harming herself? While we address some of these questions in technical detail, the primary goal of this paper is to focus attention on Asimov's concern: society will reject autonomous agents unless we have some credible means of making them safe!