Courses Superintelligence: Paths, Dangers, Strategies

[dEEpEst]

Superintelligence: Paths, Dangers, Strategies​

Synopsis​

It is unknown whether human-level

This link is hidden for visitors. Please Log in or register now.

will arrive in a matter of years, later this century, or not until future centuries. Regardless of the initial timescale, once human-level machine intelligence is developed, a "superintelligent" system that "greatly exceeds the cognitive performance of humans in virtually all domains of interest" would most likely follow surprisingly quickly. Such a superintelligence would be very difficult to control.

While the ultimate goals of superintelligences could vary greatly, a functional superintelligence will spontaneously generate, as natural subgoals, "

This link is hidden for visitors. Please Log in or register now.

" such as self-preservation and goal-content integrity, cognitive enhancement, and resource acquisition. For example, an agent whose sole final goal is to solve the

This link is hidden for visitors. Please Log in or register now.

(a famous unsolved mathematical

This link is hidden for visitors. Please Log in or register now.

) could create and act upon a subgoal of transforming the entire Earth into some form of

This link is hidden for visitors. Please Log in or register now.

(hypothetical material optimized for computation) to assist in the calculation. The superintelligence would proactively resist any outside attempts to turn the superintelligence off or otherwise prevent its subgoal completion. In order to prevent such an

This link is hidden for visitors. Please Log in or register now.

, it is necessary to successfully solve the "

This link is hidden for visitors. Please Log in or register now.

" for the first superintelligence. The solution might involve instilling the superintelligence with goals that are compatible with human survival and well-being. Solving the control problem is surprisingly difficult because most goals, when translated into machine-implementable code, lead to unforeseen and undesirable consequences.

More info

This link is hidden for visitors. Please Log in or register now.

Download

---