may I ask how, exactly? is this not the singularity in which computers get the idea that they're superior to us, and what-all-not? seriously, I'm confused.

No, the singularity is the point when a greater than human intelligence is achieved. This could be you. :-) There has been a long running debate in the transhumanists circles for years about IA vs AI (Intelligence Augmentation vs. Artificial Intelligence). I think the evidence from brain science and neurological complexity is that the future of intelligence will be a SYMBIOSIS between man and machine. Together as equals we will both go to the stars.

just a minor modification to the above very interesting discussion:

i reckon singularity refers to the fusion of 3 (rather than 2) distinct forms: man, machine and nature.

and by definition, none of the 3 will retain their autonomy. neither man nor machine (nor nature for that matter) will be more powerful and capable of dominating the others - the resultant being will be all of these three and none of them - it'll be altogether a new (bio? organic?) form of life.

now, as regards to how we choose to see this transformation, this depends. some see it as a good thing worth pursuing; some see as an evil progression that should be stopped by all means; others see it as inevitable but they're not gonna bother, others see it as pure science fiction. and some others try to make sense out of it. which is the smartest thing to do? i dunno. but certainly, overcoming certain limitations imposed upon us by nature is a tantalising prospect worth exploring and thinking aloud.

george,

your trichotomy assumes that man, machine, and nature are somehow autonomous at present. this i think is just plain false.

man is ultimately just one of our biosphere's ways of organizing itself, right? it's been doing it for billions of years already, and the arrival of humans simply ushered in a new "level of organization", if you don't mind the ontological hand-waving. and machines, of course, exist only in terms of their human-interactive function (at least at present). to argue that man is autonomous from nature because man controls nature presupposes an ontological rift between the two, which is of course begging the question.

that said, i'm sympathetic with the general scenario you allude to. at some point, we'll learn to construct machines that have some degree of autonomy (not autonomy from nature, but at least from our ability to predict and control them), which, in all likelihood, will live symbiotically with humanity. the resulting superorganism is presumably the focus of the inconceivable post-(or trans-)human state of life on earth characterized by the singularity.

coming back to the topic, i think yudkowski's remarks are comically reminiscent of christian theology. given a set of traits that we define as being "good" in a human (e.g. power, wisdom, compassion, etc.), we extrapolate these qualities to their logical extreme and from there posit the necessary existence of some perfect being (e.g. the christian god). similarly, yudkowski seems to be taking all the things we consider "good" about technological/social progress, extrapolates them to their logical extreme and calls this the singularity.

personally, i don't think the singularity will usher in a new age of human prosperity any more than the advent of multicellular life on earth ushered in a new age of bacterial prosperity. it simply changed the name of the game, made things inconceivably more complex, and likewise more interesting. but my point is, i see no reason to think that the post-singularity future will be any kind of paradise. certainly, i think it will be much nicer and more interesting than the state of being we currently occupy, but appeals to "cosmic bliss" and the like are pretty unimaginative.

yeah, i think that a planetary consciousness will probably emerge at some point, but let's not fool ourselves into thinking this is the end of the story. paradise is a very simple-minded anthropocentric concept.

.maak,

you stand correct; i wrongly implied that humans are now autonomous, as we're obviously dependent upon the surrounding biosphere and technosphere.

i also agree with you that no paradise will be realised merely because of the advent of superhuman intelligence. however, this paradigm shift in how we command and change our form will greatly enhance our abilities to choose consciously how we will lead our lives, whether that may be inside a supercomputer as part of a cosmic simulation or on new planetary systems. and this very ability may appear closely related to some more or less well accepted doctrine of life in paradise. A great resource that supports this view; indeed from wich i have derived this opinion is roger williams's "the metamorphosis of prime intellect". the book is, of course, science-fiction, however the main thesis of the book - mind uploading - is nonetheless not a distant reality.

George,

I am unable to find the title "Metamorphosis of Prime Intellect" by Roger Williams. Can you find it on the net for me and post the URL here for me?

hosting by courtesy of corrosion:

R. Williams. Metamorphosis of Prime Intellect

http://www.kuro5hin.org/prime-intellect/

good book. although pure fiction, it made me rethink my previous attitude toward mind uploading.

The Singularity, as Eliezer describes it, is a possible future event: the creation of a smarter-than-human intelligence. This may or may not happen. If it does happen, the consequences will depend on the kind of intelligence that is created: its abilities and its desires.

Unfortunately it appears that the simplest minds we can create, through AI, will have rather alien desires. They will most likely be indifferent to the concepts we call good, and right (perhaps despite programmer effort otherwise.) This is simply because the structure underlying our sense of morality, shared between all humans by virtue of belonging to the same species with the same mental hardware, is complex and non-obvious. It's not the kind of moral disputes we humans argue over, but the kind of thing unravelled by cognitive science, evolutionary psychology, etc. If the AI designers fail to transfer this structure it's pretty unlikely to magically materialise. (As you might guess I'm glossing over details here to give a snap summary. Try http://www.singinst.org/friendly/ for more details.)

As such, Eliezer's comment refers not to the general result of creating any kind of smarter than human intelligence, but the possible result given some kind of "nice" intelligence that, like many of us, wants to "make things better."

good comments on singularity from what I've been reading this could be the make or break point. I shudder to think that we could wipe ourselves out in the process of trying to become superintelligent here is where most people would disagree with transhumanist ideology. On a side not I like how you've combined rational science, utopian futurism, along with real world notions such as the vivation that you've talked to me about I'll come up with some good articles soon for you blog!

Nick,

It's been my running theory that Strong AI will never be successful without a merging with the human mind. The future of intelligence will be a symbiosis of man and machine, evolving forward together as merged equals.

I wish Eli luck in creating a truly benevolent AI, as it would neatly solve the bottleneck problem that we are approaching of rampant decentralized nanotechnology and the chance of even one malicious person destroying the world. I think there are other ways to mitigate these dangers, but at the moment I know of no method that is sure proof.

Devon,

This is what I'm incrasingly referring to as the Bottleneck problem. As are technology advances, especially the ability of a singular individual to do great harm (via bio/nanotech), as well as governments engaged in some nanowar, this is the bottleneck we have to pass in order to make it. And as I have also said if we do make it, it means utopian for every survivor. If there is even a small shred of malevolence it will be enough to destroy the planet. Utopia or Oblivion, there is no third way that I can see.