The CEO of a prominent group promoting
the manifestation of a technological singularity, Luke Muehlhauser
from the Singularity Institute for Artificial Intelligence, recently
came out with a
very surprising statement regarding the dangers which would accompany
the creation of artificial super-intelligence:
“Unfortunately, the singularity may not be what you're hoping for. By default the singularity (intelligence explosion) will go very badly for humans, because what humans want is a very, very specific set of things in the vast space of possible motivations, and it's very hard to translate what we want into sufficiently precise math, so by default superhuman AIs will end up optimizing the world around us for something other than what we want, and using up all our resources to do so.”
For those of you unfamiliar with the
concept of the technological singularity... it has to do (generally
speaking) with programming a thinking computer that initially has the
same cognitive abilities as a human being. Due to computers
regularly becoming able to process evermore information faster, in a
very short time, after a computer achieved a human level of
intellect, it would, conceivably, surpass that level – arguably in
the next moment and almost certainly within the next few years. What
would start with a computer being able to pass a Turing test
(basically being able to fool human observers as to whether or not
they were having a dialogue with a human or a computer) would then
shortly be followed by a type of self-consciousness machine that
would intellectually be capable of manipulating humans and taking
human rationality to its furthest degree.
