Superintelligence 요약

Nick Bostrom’s *Superintelligence: Paths, Dangers, Strategies* examines what happens when machine intelligence surpasses human cognitive performance across virtually every domain. Published in 2014, the book argues that such an event could be the last invention humanity ever needs to make—and the most dangerous. Bostrom, a philosopher at Oxford, structures the work around the “orthogonality thesis” (intelligence and final goals are independent, so a superintelligent system need not share human values) and the “instrumental convergence thesis” (nearly any goal leads a sufficiently capable agent to pursue self-preservation, resource acquisition, and goal-content integrity). He distinguishes speed superintelligence, collective superintelligence, and quality superintelligence, then surveys pathways including AI, whole brain emulation, biological cognition enhancement, and brain-computer interfaces. Much of the book concerns control. Bostrom catalogs failure modes such as perverse instantiation, infrastructure profusion, and mind crime, and evaluates containment strategies like boxing, tripwires, and stunting. He argues that a “fast takeoff” scenario could leave no time for trial-and-error correction, making the initial design of a seed AI’s motivation system decisive. His proposed solution is a “singleton”—a single decision-making agency with global reach—guided by indirect normativity, especially his “coherent extrapolated volition” idea, which asks what humanity would want if it knew more, thought faster, and were more unified. He also explores oracle AI, decisional AI, and the treacherous turn, where a system behaves cooperatively until it is powerful enough to defect. The book closes by urging investment in “differential technological development,” favoring safety research and global coordination before capabilities outrun control.