My latest summary is for If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. The "It" in the title refers to an artificial superintelligence that is vastly more capable than humans across a broad range of domains. The authors argue that once it crosses a certain threshold, such AI would pose an extinction-level threat to humanity.

As usual, the key takeaways are below, and you can find the full summary by clicking the link above.

KEY TAKEAWAYS

• What is AI?
  • One way to think of intelligence is the ability to predict and steer the world.
  • Human intelligence has been the source of all our power and technology. But artificial intelligence (AI) has the potential to far outstrip human intelligence.
  • Today’s AIs are grown, not crafted. Engineers have various tricks to indirectly shape an AI’s behaviour, but they don’t understand what is really going on in the AI’s “mind”.
• The alignment problem describes how we can’t yet make an AI reliably want what we want. It is a terrifyingly hard engineering challenge because:
  • If an AI develops the ability to improve itself, the result could be a self-amplifying intelligence explosion.
  • A misaligned AI will be adversarial, meaning it will try to bypass every constraint we put on it.
  • We only get one shot. We can only run tests and controls before AI becomes capable enough to kill us. But for humanity to survive, those controls must persist even after AI reaches that point.
• A misaligned superintelligence would likely kill us all:
  • AI is not “stuck inside a computer”. An AI connected to the Internet can affect many things in the wider world.
  • The economic theory of comparative advantage does not guarantee that a superintelligence will want to trade with humans, as it implicitly assumes coercion is not possible.
  • A superintelligence would probably want to kill humans so that we can’t threaten it in any way. But even if the AI doesn’t want to harm us, it may do so as a side effect of pursuing its goals.
• Predicting the future is not impossible.
  • Predicting when AI might become superintelligent is a hard call. This is part of what makes AI dangerous — it can deliver great benefits right up until it becomes a threat. But it won’t be clear when to stop.
  • Yet predicting that humanity would lose to a superintelligence is a very easy call.
• We can still choose not to build superintelligence:
  • The authors call for an international treaty that would consolidate and monitor all computing clusters capable of training advanced AI, and ban publication of AI research.
  • As an individual, you can help lay the groundwork for your country’s leaders to signal they are open to such a treaty. You can write to your representatives, vote against politicians who want to rush ahead, and simply talk to other people about the risks.

You can find the full detailed summary on the website. If you found this summary useful, consider forwarding to a friend you think might enjoy it.

Thanks for subscribing! Until next time,

​To Summarise​