Welcome to the Most Important Conversation of Our Time

• Life, defined as a process that can retain its complexity and replicate, can develop through three stages: a biological stage (1.0), where its hardware and software are evolved, a cultural stage (2.0), where it can design its software (through learning) and a technological stage (3.0), where it can design its hardware as well, becoming the master of its own destiny.

• Artificial intelligence may enable us to launch Life 3.0 this century, and a fascinating conversation has sprung up regarding what future we should aim for and how this can be accomplished. There are three main camps in the controversy: techno-skeptics, digital utopians and the beneficial-AI movement.

• Techno-skeptics view building superhuman AGI as so hard that it won’t happen for hundreds of years, making it silly to worry about it (and Life 3.0) now.

• Digital utopians view it as likely this century and wholeheartedly welcome Life 3.0, viewing it as the natural and desirable next step in the cosmic evolution.

• The beneficial-AI movement also views it as likely this century, but views a good outcome not as guaranteed, but as something that needs to be ensured by hard work in the form of AI-safety research.

Matter Turns Intelligent

• Intelligence, defined as ability to accomplish complex goals, can’t be measured by a single IQ, only by an ability spectrum across all goals.

• Today’s artificial intelligence tends to be narrow, with each system able to accomplish only very specific goals, while human intelligence is remarkably broad.

• Memory, computation, learning and intelligence have an abstract, intangible and ethereal feel to them because they’re substrate-independent: able to take on a life of their own that doesn’t depend on or reflect the details of their underlying material substrate.

• Any chunk of matter can be the substrate for memory as long as it has many different stable states.

• Any matter can be computronium, the substrate for computation, as long as it contains certain universal building blocks that can be combined to implement any function. NAND gates and neurons are two important examples of such universal “computational atoms.”

• A neural network is a powerful substrate for learning because, simply by obeying the laws of physics, it can rearrange itself to get better and better at implementing desired computations.

The Near Future: Breakthroughs, Bugs, Laws, Weapons and Jobs

• Near-term AI progress has the potential to greatly improve our lives in myriad ways, from making our personal lives, power grids and financial markets more efficient to saving lives with self-driving cars, surgical bots and AI diagnosis systems.

• When we allow real-world systems to be controlled by AI, it’s crucial that we learn to make AI more robust, doing what we want it to do. This boils down to solving tough technical problems related to verification, validation, security and control.

• This need for improved robustness is particularly pressing for AI-controlled weapon systems, where the stakes can be huge.

• Many leading AI researchers and roboticists have called for an international treaty banning certain kinds of autonomous weapons, to avoid an out-of-control arms race that could end up making convenient assassination machines available to everybody with a full wallet and an axe to grind.

• AI can make our legal systems more fair and efficient if we can figure out how to make robojudges transparent and unbiased.

• Our laws need rapid updating to keep up with AI, which poses tough legal questions involving privacy, liability and regulation.

• Long before we need to worry about intelligent machines replacing us altogether, they may increasingly replace us on the job market.

• This need not be a bad thing, as long as society redistributes a fraction of the AI-created wealth to make everyone better off.

• Otherwise, many economists argue, inequality will greatly increase.

• With advance planning, a low-employment society should be able to flourish not only financially, with people getting their sense of purpose from activities other than jobs.

• Career advice for today’s kids: Go into professions that machines are bad at—those involving people, unpredictability and creativity.

Intelligence Explosion?

• If we one day succeed in building human-level AGI, this may trigger an intelligence explosion, leaving us far behind.

• If a group of humans manage to control an intelligence explosion, they may be able to take over the world in a matter of years.

• If humans fail to control an intelligence explosion, the AI itself may take over the world even faster.

• Whereas a rapid intelligence explosion is likely to lead to a single world power, a slow one dragging on for years or decades may be more likely to lead to a multipolar scenario with a balance of power between a large number of rather independent entities.

• The history of life shows it self-organizing into an ever more complex hierarchy shaped by collaboration, competition and control. Superintelligence is likely to enable coordination on ever-larger cosmic scales, but it’s unclear whether it will ultimately lead to more totalitarian top-down control or more individual empowerment.

• Cyborgs and uploads are plausible, but arguably not the fastest route to advanced machine intelligence.

Aftermath: The Next 10,000 Years

The current race toward AGI can end in a fascinatingly broad range of aftermath scenarios for upcoming millennia. There’s absolutely no consensus on which, if any, of these scenarios are desirable, and all involve objectionable elements. This makes it all the more important to continue and deepen the conversation around our future goals, so that we don’t inadvertently drift or steer in an unfortunate direction.

AI Aftermath Scenarios

Libertarian utopia	Humans, cyborgs, uploads and superintelligences coexist peacefully thanks to property rights.
Benevolent dictator	Everybody knows that the AI runs society and enforces strict rules, but most people view this as a good thing.
Egalitarian utopia	Humans, cyborgs and uploads coexist peacefully thanks to property abolition and guaranteed income.
Gatekeeper	A superintelligent AI is created with the goal of interfering as little as necessary to prevent the creation of another superintelligence. As a result, helper robots with slightly subhuman intelligence abound, and human-machine cyborgs exist, but technological progress is forever stymied.
Protector god	Essentially omniscient and omnipotent AI maximizes human happiness by intervening only in ways that preserve our feeling of control of our own destiny and hides well enough that many humans even doubt the AI’s existence.
Enslaved god	A superintelligent AI is confined by humans, who use it to produce unimaginable technology and wealth that can be used for good or bad depending on the human controllers.
Conquerors	AI takes control, decides that humans are a threat/nuisance/waste of resources, and gets rid of us by a method that we don’t even understand.
Descendants	AIs replace humans, but give us a graceful exit, making us view them as our worthy descendants, much as parents feel happy and proud to have a child who’s smarter than them, who learns from them and then accomplishes what they could only dream of—even if they can’t live to see it all.
Zookeeper	An omnipotent AI keeps some humans around, who feel treated like zoo animals and lament their fate.
1984	Technological progress toward superintelligence is permanently curtailed not by an AI but by a human-led Orwellian surveillance state where certain kinds of AI research are banned.
Reversion	Technological progress toward superintelligence is prevented by reverting to a pre-technological society in the style of the Amish.
Self-destruction	Superintelligence is never created because humanity drives itself extinct by other means (say nuclear and/or biotech mayhem fueled by climate crisis).

Our Cosmic Endowment: The Next Billion Years and Beyond

• Compared to cosmic timescales of billions of years, an intelligence explosion is a sudden event where technology rapidly plateaus at a level limited only by the laws of physics.

• This technological plateau is vastly higher than today’s technology, allowing a given amount of matter to generate about ten billion times more energy (using sphalerons or black holes), store 12–18 orders of magnitude more information or compute 31–41 orders of magnitude faster—or to be converted to any other desired form of matter.

• Superintelligent life would not only make such dramatically more efficient use of its existing resources, but would also be able to grow today’s biosphere by about 32 orders of magnitude by acquiring more resources through cosmic settlement at near light speed.

• Dark energy limits the cosmic expansion of superintelligent life and also protects it from distant expanding death bubbles or hostile civilizations. The threat of dark energy tearing cosmic civilizations apart motivates massive cosmic engineering projects, including wormhole construction if this turns out to be feasible.

• The main commodity shared or traded across cosmic distances is likely to be information.

• Barring wormholes, the light-speed limit on communication poses severe challenges for coordination and control across a cosmic civilization. A distant central hub may incentivize its superintelligent “nodes” to cooperate either through rewards or through threats, say by deploying a local guard AI programmed to destroy the node by setting off a supernova or quasar unless the rules are obeyed.

• The collision of two expanding civilizations may result in assimilation, cooperation or war, where the latter is arguably less likely than it is between today’s civilizations.

• Despite popular belief to the contrary, it’s quite plausible that we’re the only life form capable of making our observable Universe come alive in the future.

• If we don’t improve our technology, the question isn’t whether humanity will go extinct, but merely how: will an asteroid, a supervolcano, the burning heat of the aging Sun or some other calamity get us first?

• If we do keep improving our technology with enough care, foresight and planning to avoid pitfalls, life has the potential to flourish on Earth and far beyond for many billions of years, beyond the wildest dreams of our ancestors.

Goals

• The ultimate origin of goal-oriented behavior lies in the laws of physics, which involve optimization.

• Thermodynamics has the built-in goal of dissipation: to increase a measure of messiness that’s called entropy.

• Life is a phenomenon that can help dissipate (increase overall messiness) even faster by retaining or growing its complexity and replicating while increasing the messiness of its environment.

• Darwinian evolution shifts the goal-oriented behavior from dissipation to replication.

• Intelligence is the ability to accomplish complex goals.

• Since we humans don’t always have the resources to figure out the truly optimal replication strategy, we’ve evolved useful rules of thumb that guide our decisions: feelings such as hunger, thirst, pain, lust and compassion.

• We therefore no longer have a simple goal such as replication; when our feelings conflict with the goal of our genes, we obey our feelings, as by using birth control.

• We’re building increasingly intelligent machines to help us accomplish our goals. Insofar as we build such machines to exhibit goal-oriented behavior, we strive to align the machine goals with ours.

• Aligning machine goals with our own involves three unsolved problems: making machines learn them, adopt them and retain them.

• AI can be created to have virtually any goal, but almost any sufficiently ambitious goal can lead to subgoals of self-preservation, resource acquisition and curiosity to understand the world better—the former two may potentially lead a superintelligent AI to cause problems for humans, and the latter may prevent it from retaining the goals we give it.

• Although many broad ethical principles are agreed upon by most humans, it’s unclear how to apply them to other entities, such as non-human animals and future AIs.

• It’s unclear how to imbue a superintelligent AI with an ultimate goal that neither is undefined nor leads to the elimination of humanity, making it timely to rekindle research on some of the thorniest issues in philosophy!

Consciousness

• There’s no undisputed definition of “consciousness.” I use the broad and non-anthropocentric definition consciousness = subjective experience.

• Whether AIs are conscious in that sense is what matters for the thorniest ethical and philosophical problems posed by the rise of AI: Can AIs suffer? Should they have rights? Is uploading a subjective suicide? Could a future cosmos teeming with AIs be the ultimate zombie apocalypse?

• The problem of understanding intelligence shouldn’t be conflated with three separate problems of consciousness: the “pretty hard problem” of predicting which physical systems are conscious, the “even harder problem” of predicting qualia, and the “really hard problem” of why anything at all is conscious.

• The “pretty hard problem” of consciousness is scientific, since a theory that predicts which of your brain processes are conscious is experimentally testable and falsifiable, while it’s currently unclear how science could fully resolve the two harder problems.

• Neuroscience experiments suggest that many behaviors and brain regions are unconscious, with much of our conscious experience representing an after-the-fact summary of vastly larger amounts of unconscious information.

• Generalizing consciousness predictions from brains to machines requires a theory. Consciousness appears to require not a particular kind of particle or field, but a particular kind of information processing that’s fairly autonomous and integrated, so that the whole system is rather autonomous but its parts aren’t.

• Consciousness might feel so non-physical because it’s doubly substrate-independent: if consciousness is the way information feels when being processed in certain complex ways, then it’s merely the structure of the information processing that matters, not the structure of the matter doing the information processing.

• If artificial consciousness is possible, then the space of possible AI experiences is likely to be huge compared to what we humans can experience, spanning a vast spectrum of qualia and timescales—all sharing a feeling of having free will.

• Since there can be no meaning without consciousness, it’s not our Universe giving meaning to conscious beings, but conscious beings giving meaning to our Universe.

• This suggests that as we humans prepare to be humbled by ever smarter machines, we take comfort mainly in being Homo sentiens, not Homo sapiens.

The Asilomar AI Principles

Artificial intelligence has already provided beneficial tools that are used every day by people around the world. Its continued development, guided by the following principles, will offer amazing opportunities to help and empower people in the decades and centuries ahead.

RESEARCH ISSUES

§1 Research Goal: The goal of AI research should be to create not undirected intelligence, but beneficial intelligence.
§2 Research Funding: Investments in AI should be accompanied by funding for research on ensuring its beneficial use, including thorny questions in computer science, economics, law, ethics, and social studies, such as:
(a) How can we make future AI systems highly robust, so that they do what we want without malfunctioning or getting hacked?
(b) How can we grow our prosperity through automation while maintaining people’s resources and purpose?
(c) How can we update our legal systems to be more fair and efficient, to keep pace with AI, and to manage the risks associated with AI?
(d) What set of values should AI be aligned with, and what legal and ethical status should it have?
§3 Science-Policy Link: There should be constructive and healthy exchange between AI researchers and policy-makers.
§4 Research Culture: A culture of cooperation, trust, and transparency should be fostered among researchers and developers of AI.
§5 Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards.

ETHICS AND VALUES

§6 Safety: AI systems should be safe and secure throughout their operational lifetime, and verifiably so where applicable and feasible.
§7 Failure Transparency: If an AI system causes harm, it should be possible to ascertain why.
§8 Judicial Transparency: Any involvement by an autonomous system in judicial decision-making should provide a satisfactory explanation auditable by a competent human authority.
§9 Responsibility: Designers and builders of advanced AI systems are stakeholders in the moral implications of their use, misuse, and actions, with a responsibility and opportunity to shape those implications.
§10 Value Alignment: Highly autonomous AI systems should be designed so that their goals and behaviors can be assured to align with human values throughout their operation.
§11 Human Values: AI systems should be designed and operated so as to be compatible with ideals of human dignity, rights, freedoms, and cultural diversity.
§12 Personal Privacy: People should have the right to access, manage, and control the data they generate, given AI systems’ power to analyze and utilize that data.
§13 Liberty and Privacy: The application of AI to personal data must not unreasonably curtail people’s real or perceived liberty.
§14 Shared Benefit: AI technologies should benefit and empower as many people as possible.
§15 Shared Prosperity: The economic prosperity created by AI should be shared broadly, to benefit all of humanity.
§16 Human Control: Humans should choose how and whether to delegate decisions to AI systems, to accomplish human-chosen objectives.
§17 Non-subversion: The power conferred by control of highly advanced AI systems should respect and improve, rather than subvert, the social and civic processes on which the health of society depends.
§18 AI Arms Race: An arms race in lethal autonomous weapons should be avoided.

LONGER-TERM ISSUES

§19 Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.
§20 Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.
§21 Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.
§22 Recursive Self-Improvement: AI systems designed to recursively self-improve or self-replicate in a manner that could lead to rapidly increasing quality or quantity must be subject to strict safety and control measures.
§23 Common Good: Superintelligence should only be developed in the service of widely shared ethical ideals, and for the benefit of all humanity rather than one state or organization.