The problem of ensuring that super-intelligent artificial intelligences (AIs) pursue our moral goals. This is harder than it sounds, since AIs may find loopholes in their instructions; suppose, for instance, that an AI told to maximize human happiness decides the best way to do so is by constantly drugging us.

A thought experiment that illustrates why it’s difficult to tell whether an AI is conscious. If we were to feed Chinese messages into a box and get Chinese answers, we might assume that we’re talking to a real Chinese speaker — but inside the box is a non–Chinese speaker using an algorithm to transform inputs into outputs, without understanding the “conversation” at all.

A Silicon Valley–adjacent philosophical movement that originally asked adherents to use their money to produce the greatest good for the greatest number. Lately it has prioritized “longterm-ism,” the idea that speculative risks to future generations are more urgent to address than current suffering.