The Key Images and Ideas that Lie at the Core of GEB

In a word, GEB is a very personal attempt to say how it is that animate beings can come out of inanimate matter. What is a self, and how can a self come out of stuff that is as selfless as a stone or a puddle? What is an "I", and why are such things found (at least so far) only in association with, as poet Russell Edson once wonderfully phrased it, "teetering bulbs of dread and dream"—that is, only in association with certain kinds of gooey lumps encased in hard protective shells mounted atop mobile pedestals that roam the world on pairs of slightly fuzzy, jointed suits?

GEB approaches these questions by slowly building up an analogy that likens inanimate molecules to meaningless symbols, and further likens selves (or "I"s or "souls", if you prefer—whatever it is that distinguishes animate from inanimate matter) to certain special swirly, twisty, vortex-like, and meaningful patterns that arise only in particular types of systems of meaningless symbols. It is these strange, twisty patterns that the book spends so much time on, because they are little known, little appreciated, counterintuitive, and quite filled with mystery. And for reasons that should not be too difficult to fathom, I call such strange, loopy patterns "strange loops" throughout the book, although in later chapters, I also use the phrase "tangled hierarchies" to describe basically the same idea.

The Gödelian strange loop that arises in formal systems in mathematics (i.e., collections of rules for churning out an endless series of mathematical truths solely by mechanical symbol-shunting without any regard to meanings or ideas hidden in the shapes being manipulated) is a loop that allows such a system to "percieve itself", to talk about itself, to become "self-aware", and in a sense it would not be going too far to say that by virtue of having such a loop, a formal system acquires a self.

What is so weird in this is that the formal systems where these skeletal "selves" come to exist are built out of nothing but meaningless symbols. The self, such as it is, arises solely because of a special type of swirly, tangled pattern among the meaningless symbols. A crucial part of my book's argument rests on the idea that meaning cannot be kept out of formal systems when sufficiently complex isomorphisms arise. Meaning comes in despite one's best efforts to keep symbols meaningless! When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with some degree of meaning—indeed, such tracking or mirroring is no less and no more than what meaning is. Depending on how complex and subtle and reliable the tracking is, different degrees of meaningfulness arise.

As I see it, the only way of overcoming a magical view of what "I" and consciousness are is to keep reminding oneself, unpleasant though it may seem, that the "teetering bulb of dread and dream" that nestles safely inside one's own cranium is a purely physical object made up of completely sterile and inanimate components, all of which obey exactly the same laws as those that govern all the rest of the universe, such as pieces of text, or CD-ROM's, or computers. Only if one keeps on bashing up against this disturbing fact can one slowly begin to develop a feel for the way out of the mystery of consciousness: that the key is not the stuff out of which brains are made, but the patterns that can come to exist inside the stuff of a brain. This is a liberating shift, because it allows one to move to a different level of considering what brains are: as media that support complex patterns that mirror, albeit far from perfectly, the world, of which, needless to say, those brains are themselves denizens—and it is in the inevitable self-mirroring that arises, however impartial or imperfect it may be, that the strange loops of consciousness start to swirl.

Something very strange emerges from the Gödelian loop: the revelation of the causal power of meaning in a rule-bound but meaning-free universe. And this is where my analogy to brains and selves comes back in, suggesting that the twisted loop of selfhood trapped inside an inanimate bulb called a "brain" also has causal power—or, put another way, that a mere pattern called "I" can shove around inanimate particles in the brain no less than inanimate particles in the brain can shove around patterns. In short, an "I" comes about—in my view, at least—via a kind of vortex whereby patterns in a brain mirror the brain's mirroring of the world, and eventually mirror themselves, whereup the vortex of "I" becomes a real, causal entity. When and only when such a loop arises in a brain or in any other substrate, is a person—a unique new "I"—brought into being. Moreover, the more self-referentially rich such a loop is, the more conscious is the self to which it gives rise. Yes, shocking though this might sound, consciousness is not an on/off phenomenon, but admits of degrees, grades, shades. Or, to put it more bluntly, there are bigger souls and smaller souls.

Strange loops are an abstract structure that crops up in various media and in varying degrees of richness. GEB is in essence a long proposal of strange loops as a metaphor for how selfhood originates, a metaphor by which to begin to grab a hold of just what it is that makes an "I" seem, at one and the same time, so terribly real and tangible to its own possessor, and yet also so vague, so impenetrable, so deeply elusive. I personally cannot imagine that consciousness will be fully understood without reference to Gödelian strange loops or level-crossing feedback loops.

Introduction: A Musico-Logical Offering

The idea of a canon is that one single theme is played against itself. This is done by having "copies" of the theme played by the various participating voices. But there are many ways to do this. The most straightforward of all canons is the round, such as "Three Blind Mice", "Row, Row, Row Your Boat", or "Frere Jacques". Here, the theme enters in the first voice and, after a fixed time-delay, a "copy" of it enters, in precisely the same key. After the same fixed time-delay in the second voice, the third voice enters carrying the theme, and so on. Most themes will not harmonize with themselves in this way. In order for a theme to work as a canon theme, each of its notes must be able to serve in a dual (or triple, or quadruple) role: it must firstly be part of a melody, and secondly it must be part of a harmonization of the same melody. When there are three canonical voices, for instance, each note of the theme must act in two distinct harmonic ways, as well as melodically. Thus, each note in a canon has more than one musical meaning; the listener's ear and brain automatically figure out the appropriate meaning, by referring to context.

Bach included a crab canon in the Musical Offering, needless to say. Notice that every type of "copy" preserves all the information in the original theme, in the sense that the theme is fully recoverable from any of the copies. Such an information preserving transformation is often called an isomorphism, and we will have much traffic with isomorphisms in this book.

A fugue is like a canon, in that it is usually based on one theme which gets played in different voices and different keys, and occasionally at different speeds or upside down or backwards. However, the notion of fugue is much less rigid than that of canon, and consequently it allows for more emotional and artistic expression. The telltale sign of a fugue is the way it begins: with a single voice singing its theme. When it is done, then a second voice enters, either five scale-notes up, or four down. Meanwhile the first voice goes on, singing the "countersubject": a secondary theme, chosen to provide rhythmic, harmonic, and melodic contrasts to the subject. Each of the voices enters in turn, singing the theme, often to the accompaniment of the countersubject in some other voice, with the remaining voices doing whatever fanciful things entered the composer's mind. When all the voices have "arrived", then there are no rules. There are, to be sure, standard kinds of things to do—but not so standard that one can merely compose a fugue by formula.

In this canon, Bach has given us our first example of the notion of Strange Loops. The "Strange Loop" phenomenon occurs whenever, by moving upwards (or downwards) through the levels of some hierarchical system, we unexpectedly find ourselves right back where we started. (Here, the system is that of musical keys.) Sometimes I use the term Tangled Hierarchy to describe a system in which a Strange Loop occurs. As we go on, the theme of Strange Loops will recur again and again. Sometimes it will be hidden, other times it will be out in the open; sometimes it will be right side up, other times it will be upside down, or backwards. "Quaerendo invenietis" is my advice to the reader.

To my mind, the most beautiful and powerful visual realizations of this notion of Strange Loops exist in the work of the Dutch graphic artist M. C. Escher, who lived from 1902 to 1972. Escher was the creator of some of the most intellectually stimulating drawings of all time. Many of them have their origin in paradox, illusion, or double-meaning. Mathematicians were among the first admirers of Escher's drawings, and this is understandable because they often are based on mathematical principles of symmetry or pattern ... But there is much more to a typical Escher drawing than just symmetry or pattern; there is often an underlying idea, realized in artistic form. And in particular, the Strange Loop is one of the most recurrent themes in Escher's work.

In the examples we have seen of Strange Loops by Bach and Escher, there is a conflict between the finite and the infinite, and hence a strong sense of paradox. Intuition senses that there is something mathematical involved here. And indeed in our own century a mathematical counterpart was discovered, with the most enormous repercussions. And, just as the Bach and Escher loops appeal to very simple and ancient intuitions—a musical scale, a staircase—so this discovery, by K. Gödel, of a Strange Loop in mathematical systems has its origins in simple and ancient intuitions. In its absolutely barest form, Gödel's discovery involves the translation of an ancient paradox in philosophy into mathematical terms. That paradox is the so-called Epimenides paradox, or liar paradox. Epimenides was a Cretan who made one immortal statement: "All Cretans are liars." A sharper version of the statement is simply "I am lying"; or, "This statement is false". It is that last version which I will usually mean when I speak of the Epimenides paradox. It is a statement which rudely violates the usually assumed dichotomy of statements into true and false, because if you tentatively think it is true, then it immediately backfires on you and makes you think it is false. But once you've decided it is false, a similar backfiring returns you to the idea that it must be true.

Gödel had the insight that a statement of number theory could be about a statement of number theory (possibly even itself), if only numbers could somehow stand for statements. The idea of a code, in other words, is at the heart of his construction. In the Gödel Code, usually called "Gödel-numbering", numbers are made to stand for symbols and sequences of symbols. That way, each statement of number theory, being a sequence of specialized symbols, acquires a Gödel number, something like a telephone number or a license plate, by which it can be referred to. And this coding trick enables statements of number theory to be understood on two different levels: as statements of number theory, and also as statements about statements of number theory. Once Gödel had invented this coding scheme, he had to work out in detail a way of transporting the Epimenides paradox into a numbertheoretical formalism. His final transplant of Epimenides did not say, "This statement of number theory is false", but rather, "This statement of number theory does not have any proof".

The Gödel sentence G should more properly be written in English as:

This statement of number theory does not have any proof in the system of Principia Mathematica.

Incidentally, this Gödel sentence G is not Gödel's Theorem—no more than the Epimenides sentence is the observation that "The Epimenides sentence is a paradox." We can now state what the effect of discovering G is. Whereas the Epimenides statement creates a paradox since it is neither true nor false, the Gödel sentence G is unprovable (inside P.M.) but true. The grand conclusion? That the system of Principia Mathematica is "incomplete"—there are true statements of number theory which its methods of proof are too weak to demonstrate.

In short, Gödel showed that provability is a weaker notion than truth, no matter what axiomatic system is involved.

Chapter I: The MU-puzzle

One of the most central notions in this book is that of a formal system. The type of formal system I use was invented by the American logician Emil Post in the 1920's, and is often called a "Post production system".

Strings, producible by the rules of the system, are called theorems. The term "theorem" has, of course, a common usage mathematics which is quite different from this one. It means some statement in ordinary language which has been proven to be true by a rigorous argument, such as Zeno's Theorem about the "unexistence" of motion, or Euclid's Theorem about the infinitude of primes. But in formal system theorems need not be thought of as statements—they are merely strings of symbols. And instead of being proven, theorems are merely produced, as if by a machine, according to certain typographical rules.

A formal system may have zero, or several, or even infinitely many axioms.

Every formal system has symbol-shunting rules, such as the four rules of the MIU-system. These rules are called either rules of production or rules of inference.

A derivation of a theorem is an explicit, line-by-line demonstration of how to produce that theorem according to the rules of the formal system. The concept of derivation is modeled on that of proof, but a derivation is an austere cousin of a proof.

It is possible to program a machine to do a routine task in such a way that the machine will never notice even the most obvious facts about what it is doing; but it is inherent in human consciousness to notice some facts about the things one is doing. It is an inherent property of intelligence that it can jump out of the task which it is performing, and survey what it has done; it is always looking for and often finding, patterns.

It is very important when studying formal systems to distinguish working within the system from making statements or observations about the system. We will occasionally refer back to these two modes of dealing with a formal system, and we will call them the Mechanic mode (M-mode) and the Intelligent mode (I-mode). To round out our mode with one for each letter of the MIU-system, I will also mention a final mode-the Un-mode (U-mode), which is the Zen way of approaching things.

If there is a test for theoremhood, a test which does always terminate in a finite amount of time, then that test is called a decision procedure for the given formal system. When you have a decision procedure, then you have a very concrete characterization of the nature of all theorems in the system. A decision procedure is a "litmus test" for theoremhood! Incidentally, one requirement on formal systems is that the set of axioms must be characterized by a decision procedure—there must be a litmus test for axiomhood. This ensures that there is no problem in getting off the ground at the beginning, at least. That is the difference between the set of axioms and the set of theorems: the former always has a decision procedure, but the latter may not.

CHAPTER II: Meaning and Form in Mathematics

Do words and thoughts follow formal rules, or do they not? That problem is the problem of this book.

In the Introduction, the word "isomorphism" was defined as an information preserving transformation. We can now go into that notion a little more deeply, and see it from another perspective. The word "isomorphism" applies when two complex structures can be mapped onto each other, in such a way that to each part of one structure there is a corresponding part in the other structure, where "corresponding" means that the two part play similar roles in their respective structures. This usage of the word "isomorphism" is derived from a more precise notion in mathematics.

It is cause for joy when a mathematician discovers an isomorphism between two structures which he knows. It is often a "bolt from the blue", and a source of wonderment. The perception of an isomorphism between two known structures is a significant advance in knowledge—and I claim that it is such perceptions of isomorphism which create meanings in the minds of people. A final word on the perception of isomorphisms: since they come in many shapes and sizes, figuratively speaking, it is not always totally clear when you really have found an isomorphism. Thus, "isomorphism" is a word with all the usual vagueness of words—which is a defect but an advantage as well.

When you confront a formal system you know nothing of, and if you hope to discover some hidden meaning in it, your problem is how to assign interpretations to its symbols in a meaningful way—that is, in such a way that a higher-level correspondence emerges between true statements and theorems. You may make several tentative stabs in the dark before finding a good set of words to associate with the symbols. It is very similar to attempts to crack a code, or to decipher inscriptions in an unknown language: the only way to proceed is by trial and error, based on educated guesses. When you hit a good choice, a "meaningful" choice, all of a sudden things just feel right, and work speeds up enormously. Pretty soon everything falls into place.

Probably the most significant fact of this Chapter, if understood deeply, is this: the pq-system seems to force us into recognizing that symbols of a formal system, though initially without meaning, cannot avoid taking on "meaning" of sorts—at least if an isomorphism is found.

In Chapters to come, we will lay out a formal system that (1) includes a stylized vocabulary in which all statements about natural numbers can be expressed, and (2) has rules corresponding to all the types of reasoning which seem necessary. A very important question will be whether the rules for symbol manipulation which we have then formulated are really of equal power (as far as number theory is concerned) to our usual mental reasoning abilities—or, more generally, whether it is theoretically possible to attain the level of our thinking abilities, by using some formal system.

CHAPTER III: Figure and Ground

There is a famous artistic distinction between figure and ground. When a figure or "positive space" (e.g., a human form, or a letter, or a still life) is drawn inside a frame, an unavoidable consequence is that its complementary shape—also called the "ground", or "background", or "negative space"—has also been drawn. In most drawings, however, this figure-ground relationship plays little role. The artist is much less interested in ground than in the figure. But sometimes, an artist will take interest in ground as well.

Let us now officially distinguish between two kinds of figures: cursively drawable ones, and recursive ones (by the way, these are my own terms are not in common usage). A cursively drawable figure is one whose ground is merely an accidental by-product of the drawing act. A recursive figure is one whose ground can be seen as a figure in its own right. Usually this is quite deliberate on the part of the artist. The "re" in "recursive" represents the fact that both foreground and background are cursively drawable – the figure is "twice-cursive". Each figure-ground boundary in a recursive figure is a double-edged sword. M. C. Escher was a master at drawing recursive figures.

One may also look for figures and grounds in music. One analogue is the distinction between melody and accompaniment—for the melody is always in the forefront of our attention, and the accompaniment is subsidiary, in some sense. Therefore it is surprising when we find, in the lower lines of a piece of music, recognizable melodies. This does not happen too often in post-baroque music. Usually the harmonies are not thought of as foreground. But in baroque music—in Bach above all—the distinct lines, whether high or low or in between, all act as "figures". In this sense, pieces by Bach can be called "recursive". Another figure-ground distinction exists in music: that between on-beat and off-beat. If you count notes in a measure "one-and, two-and, three-and, four-and", most melody-notes will come on numbers, not on "and"s. But sometimes, a melody will be deliberately pushed onto the "and"s, for the sheer effect of it.

CHAPTER IV: Consistency, Completeness, and Geometry

In Chapter II, we saw how meaning—at least in the relatively simple context of formal systems—arises when there is an isomorphism between rule-governed symbols, and things in the real world. The more complex the isomorphism, in general, the more "equipment"—both hardware and software—is required to extract the meaning from the symbols. If an isomorphism is very simple (or very familiar), we are tempted to say that the meaning which it allows us to see is explicit. We see the meaning without seeing the isomorphism. The most blatant example is human language, where people often attribute meaning to words in themselves, without being in the slightest aware of the very complex "isomorphism" that imbues them with meanings. This is an easy enough error to make. It attributes all the meaning to the object (the word), rather than to the link between that object and the real world. You might compare it to the naive belief that noise is a necessary side effect of any collision of two objects. This is a false belief; if two objects collide in a vacuum, there will be no noise at all. Here again, the error stems from attributing the noise exclusively to the collision, and not recognizing the role of the medium, which carries it from the objects to the ear.

The Tortoise says that no sufficiently powerful record player can be perfect, in the sense of being able to reproduce every possible sound from a record. Gödel says that no sufficiently powerful formal system can be perfect, in the sense of reproducing every single true statement as a theorem. But as the Tortoise pointed out with respect to phonographs, this fact only seems like a defect if you have unrealistic expectations of what formal systems should be able to do. Nevertheless, mathematicians began this century with just such unrealistic expectations, thinking that axiomatic reasoning was the cure to all ills. They found out otherwise in 1931. The fact that truth transcends theoremhood, in any given formal system, is called "incompleteness" of that system.

It now becomes clear that consistency is not a property of a formal system per se, but depends on the interpretation which is proposed for it. By the same token, inconsistency is not an intrinsic property of any formal system.

We have been speaking of "consistency" and "inconsistency" all along, without defining them. We have just relied on good old everyday notions. But now let us say exactly what is meant by consistency of a formal system (together with an interpretation): that every theorem, when interpreted, becomes a true statement. And we will say that inconsistency occurs when there is at least one false statement among the interpreted theorems. This definition appears to be talking about inconsistency with the external world—what about internal inconsistencies? Presumably, a system would be internally inconsistent if it contained two or more theorems whose interpretations were incompatible with one another, and internally consistent if all interpreted theorems were compatible with one another.

We have given two ways of looking at consistency: the first says that a system-plus-interpretation is consistent with the external world if every theorem comes out true when interpreted; the second says that a system-plus-interpretation is internally consistent if all theorems come out mutually compatible when interpreted. Now there is a close relationship between these two types of consistency. In order to determine whether several statements at mutually compatible, you try to imagine a world in which all of them could be simultaneously true. Therefore, internal consistency depends upon consistency with the external world—only now, "the external world" is allowed to be any imaginable world, instead of the one we live in. But this is an extremely vague, unsatisfactory conclusion. What constitutes an “imaginable" world?

Roughly, then, it should be possible to establish different brands of consistency. For instance, the most lenient would be "logical consistency", putting no restraints on things at all, except those of logic. More specifically, a system-plus-interpretation would be logically consistent just as long as no two of its theorems, when interpreted as statements, directly contradict each other; and mathematically consistent just as long as interpreted theorems do not violate mathematics; and physically consistent just as long as all its interpreted theorems are compatible with physical law; then comes biological consistency, and so on.

Consistency: when every theorem, upon interpretation, comes out true (in some imaginable world).
Completeness: when all statements which are true (in some imaginable world), and which can be expressed as well-formed strings of the system, are theorems.

CHAPTER V: Recursive Structures and Processes

What is recursion? It is nesting, and variations on nesting. The concept is very general. (Stories inside stories, movies inside movies, paintings inside paintings, Russian dolls inside Russian dolls (even parenthetical comments in. side parenthetical comments!)—these are just a few of the charms of recursion.) Sometimes recursion seems to brush paradox very closely. For example, there are recursive definitions. Such a definition may give the casual viewer the impression that something is being defined in terms of itself. That would be circular and lead to infinite regress, if not to paradox proper. Actually, a recursive definition (when properly formulated) never leads to infinite regress or paradox. This is because a recursive definition never defines something in terms of itself, but always in terms of simpler versions of itself.

This is the crucial fact which distinguishes recursive definitions from circular ones. There is always some part of the definition which avoids reference, so that the action of constructing an object which satisfies the definition will eventually "bottom out".

We keep on running up against "sameness-in-differentness", and the question When are two things the same? It will recur over and over again in this book. We shall come at it from all sorts of skew angles, and in the end, we shall see how deeply this simple question is connected with the nature of intelligence. That this issue arose in the Chapter on recursion is no accident, for recursion is a domain where "sameness-in-differentness" plays a central role. Recursion is based on the "same" thing happening on several different levels at once. But the events on different levels aren't exactly the same—rather, we find some invariant feature in them, despite many ways in which they differ.

Now what is the connection between the recursive processes of this Chapter, and the recursive sets of the preceding Chapter? The answer involves the notion of a recursively enumerable set. For a set to be r.e. means that it can be generated from a set of starting points (axioms), by the repeated application of rules of inference. Thus, the set grows and grows, each new element being compounded somehow out of previous elements, in a sort of "mathematical snowball". But this is the essence of recursion—something being defined in terms of simpler versions of itself, instead of explicitly.

It might seem that recursively defined sequences of that type possess some sort of inherently increasing complexity of behavior, so that the further out you go, the less predictable they get. This kind of thought carried a little further suggests that suitably complicated recursive systems might be strong enough to break out of any predetermined patterns. And isn't this one of the defining properties of intelligence? Instead of just considering programs composed of procedures which can recursively call themselves, why not get really sophisticated, and invent programs which can modify themselves—programs which can act on programs, extending them, improving them, generalizing them, fixing them, and so on? This kind of "tangled recursion" probably lies at the heart of intelligence.

CHAPTER VI: The Location of Meaning

Last chapter, we came upon the question, "When are two things the same?" In this Chapter, we will deal with the flip side of that question: "When is one thing not always the same?" The issue we are broaching is whether meaning can be said to be inherent in a message, or whether meaning is always manufactured by the interaction of a mind or a mechanism with a message. In the latter case, meaning could not said to be located in any single place, nor could it be said that a message has any universal, or objective, meaning, since each observer could bring its own meaning to each message. But in the former case, meaning would have both location and universality. In this Chapter, I want to present the case for the universality of at least some messages, without, to be sure, claiming it for all messages. The idea of an "objective meaning" of a message will turn out to be related, in an interesting way, to the simplicity with which intelligence can be described.

One gets the impression that isomorphisms and decoding mechanisms (i.e., information-revealers) simply reveal information which is intrinsically inside the structures, waiting to be "pulled out". This leads to the idea that for each structure, there are certain pieces of information which can be pulled out of it, while there are other pieces of information which cannot be pulled out of it. But what does this phrase "pull out" really mean? How hard are you allowed to pull? There are cases where by investing sufficient effort, you can pull very recondite piece of information out of certain structures. In fact, the pulling-out may involve such complicated operations that it makes you feel you are putting in more information than you are pulling out.

Nowadays, the idea of decoding is extremely widespread; it is a significant part of the activity of astronomers, linguists, archaeologists, military specialists, and so on. It is often suggested that we may be floating in a sea of radio messages from other civilizations, messages which we do not yet know how to decipher. And much serious thought has been given to the techniques of deciphering such a message. One of the main problems—perhaps the deepest problem—is the question, "How will we recognize the fact that there is a message at all? How to identify a frame?"

We can separate out fairly clearly three levels of information: (1) the frame message; (2) the outer message; (3) the inner message. The one we are most familiar with is (3), the inner message; it is the message which is supposed to be transmitted: the emotional experiences in music, the phenotype in genetics, the royalty and rites of ancient civilizations in tablets, etc. To understand the inner message is to have extracted the meaning intended by the sender.

The frame message is the message "I am a message; decode me if you can!"; and it is implicitly conveyed by the gross structural aspects of any information-bearer. To understand the frame message is to recognize the need for a decoding-mechanism.

If the frame message is recognized as such, then attention is switched to level (2), the outer message. This is information, implicitly carried by symbol-patterns and structures in the message, which tells how to decode the inner message. To understand the outer message is to build, or know how to build, the correct decoding mechanism for the inner message. This outer level is perforce an implicit message, in the sense that the sender cannot ensure that it will be understood. It would be a vain effort to send instructions which tell how to decode the outer message, for they would have to be part of the inner message, which can only be understood once the decoding mechanism has been found. For this reason, the outer message is necessarily a set of triggers, rather than a message which can be revealed by a known decoder.

It is in the nature of outer messages that they are not conveyed in any explicit language. To find an explicit language in which to convey outer messages would not be a breakthrough—it would be a contradiction in terms! It is always the listener's burden to understand the outer message. Success lets him break through into the inside, at which point the ratio of triggers to explicit meanings shifts drastically towards the latter. By comparison with the previous stages, understanding the inner message seems effortless. It is as if it just gets pumped in.

These examples may appear to be evidence for the viewpoint that no message has intrinsic meaning, for in order to understand any inner message, no matter how simple it is, one must first understand its frame message and its outer message, both of which are carried only by triggers (such as being written in the Japanese alphabet, or having spiraling grooves, etc.). It begins to seem, then, that one cannot get away from a "jukebox" theory of meaning—the doctrine that no message contains inherent meaning, because, before any message can be understood, it has to be used as the input to some "jukebox", which means that information contained in the "jukebox" must be added to the message before it acquires meaning. However, rules do get used, and messages do get understood. How come?

This happens because our intelligence is not disembodied, but is instantiated in physical objects: our brains. Their structure is due to the long process of evolution, and their operations are governed by the laws of physics. Since they are physical entities, our brains run without being told how to run. So it is at the level where a brain interprets incoming data as a message that the message-paradox breaks down. It seems that brains come equipped with "hardware" for recognizing that certain things are messages, and for decoding those messages. This minimal inborn ability to extract inner meaning is what allows the highly recursive, snowballing process of language acquisition to take place. The inborn hardware is like a jukebox: it supplies the additional information which turns mere triggers into complete messages.

It would be nice if we could define intelligence in some other way than "that which gets the same meaning out of a sequence of symbols as we do". For if we can only define it this one way, then our argument that meaning is an intrinsic property is circular, hence content-free. We should try to formulate in some independent way a set of characteristics which deserve the name "intelligence". Such characteristics would constitute the uniform core of intelligence, shared by humans. At this point in history we do not yet have a well-defined list of those characteristics. However, it appears likely that within the next few decades there will be much progress made in elucidating what human intelligence is. In particular, perhaps cognitive psychologists, workers in Artificial Intelligence, and neuroscientists will be able to synthesize their understandings, and come up with a definition of intelligence. It may still be human-chauvinistic; there is no way around that. But to counterbalance that, there may be some elegant and beautiful—and perhaps even simple—abstract ways of characterizing the essence of intelligence. This would serve to lessen the feeling of having formulated an anthropocentric concept. And of course, if contact were established with an alien civilization from another star system, we feel supported in our belief that our own type of intelligence is not just a fluke, but an example of a basic form which reappears in nature in contexts, like stars and uranium nuclei. This in turn would support the idea of meaning being an inherent property.

CHAPTER VII: The Propositional Calculus

The Propositional Calculus gives us a set of rules for producing statements which would be true in all conceivable worlds. That is why all of its theorems sound so simple-minded; it seems that they have absolutely no content! Looked at this way, the Propositional Calculus might seem to be a waste of time, since what it tells us is absolutely trivial. On the other hand, it does it by specifying the form of statements that are universally true, and this throws a new kind of light onto the core truths of the universe: they are not only fundamental, but also regular: they can be produced by one set of typographical rules. To put it another way, they are all "cut from the same cloth".

Despite its quirks, the Propositional Calculus has some features to recommend itself. If one embeds it into a larger system (as we will do next Chapter), and if one is sure that the larger system contains no contradictions (and we will be), then the Propositional Calculus does all that one could hope: it provides valid propositional inferences—all that can be made. So if ever an incompleteness or an inconsistency is uncovered, one can be sure that it will be the fault of the larger system, and not of its subsystem which is the Propositional Calculus.

CHAPTER VIII: Typographical Number Theory

Gödel's construction depends on describing the form, as well as the content, of strings of the formal system we shall define in this Chapter—Typographical Number Theory (TNT). The unexpected twist is that, because of the subtle mapping which Gödel discovered, the form of strings can be described in the formal system itself.

It is worthwhile pausing for breath and contemplating what it would mean to have a formal system that could sift out the true ones from the false ones. This system would treat all these strings—which to us look like statements—as designs having form, but no content. And this system would be like a sieve through which could pass only designs with a special style—the "style of truth". The boundary separating the set of true statements from the set of false statements (as written in the TNT-notation) is anything but straight; it is a boundary with many treacherous curves, a boundary of which mathematicians have delineated stretches, here and there, working over hundreds of years. Just think what a coup it would be to have a typographical method which was guaranteed to place any formula on the proper side of the border!

CHAPTER IX: Mumon and Gödel

One of the basic tenets of Zen Buddhism is that there is no way to characterize what Zen is. No matter what verbal space you try to enclose Zen in, it resists, and spills over. It might seem, then, that all efforts to explain Zen are complete wastes of time. But that is not the attitude of Zen masters and students. For instance, Zen koans are a central part of Zen study, verbal though they are. Koans are supposed to be "triggers" which, though they do not contain enough information in themselves to impart enlightenment, may possibly be sufficient to unlock the mechanisms inside one's mind that lead to enlightenment. But in general, the Zen attitude is that words and truth are incompatible, or at least that no words can capture truth. Only by stepping outside of logic, so the theory goes, can one make the leap to enlightenment. But what is so bad about logic? Why does it prevent the leap to enlightenment?

To answer that, one needs to understand something about what enlightenment is. Perhaps the most concise summary of enlightenment would be: transcending dualism. Now what is dualism? Dualism is the conceptual division of the world into categories. Is it possible to transcend this natural tendency? By prefixing the word "division" by the word "conceptual", I may have made it seem that this is an intellectual or conscious effort, and perhaps thereby given the impression that dualism could be overcome simply by suppressing thought (as if to suppress thinking act were simple!). But the breaking of the world into categories takes place below the upper strata of thought; in fact, dualism is just as a perceptual division of the world into categories as it is a conceptual division. In other words, human perception is by nature a dualistic phenomenon—which makes the quest for enlightenment an uphill struggle, to say the least.

At the core of dualism, according to Zen, are words—just plain words. The use of words is inherently dualistic, since each word represents, obviously, a conceptual category. Therefore, a major part of Zen is the fight against reliance on words. To combat the use of words, one of the devices is the kōan, where words are so deeply abused that one's mind is practically left reeling, if one takes the kōans seriously. Therefore it is perhaps wrong to say that the enemy of enlightenment is logic; rather it is dualistic, verbal thinking. In fact, it is even more basic than that: it is perception. As soon as you perceive an object, you draw a line between it and the rest of the world; you divide the world, artificially, into parts, and you thereby miss the Way.

I have a name for what Zen strives for: ism. Ism is an antiphilosophy, a way of being without thinking. The masters of ism are rocks, trees, clams; but it is the fate of higher animal species to have to strive for ism, without ever being able to attain it fully. Still, one is occasionally granted glimpses of ism. To suppress perception, to suppress logical, verbal, dualistic thinking—this is the essence of Zen, the essence of ism. This is the Unmode—not Intelligent, not Mechanical, just "Un".

Zen is holism, carried to its logical extreme. If holism claims that things can only be understood as wholes, not as sums of their parts, Zen goes one further, in maintaining that the world cannot be broken into parts at all. To divide the world into parts is to be deluded, and to miss enlightenment.

CENTRAL PROPOSITION: If there is a typographical rule which tells how certain digits are to be shifted, changed, dropped, or inserted in any number represented decimally, then this rule can be represented equally well by an arithmetical counterpart which involves arithmetical operations with powers of 10 as well as additions, subtractions, and so forth.

More briefly:

Typographical rules for manipulating numerals are actually arithmetical rules for operating on numbers.

This simple observation is at the heart of Gödel's method, and it will have an absolutely shattering effect. It tells us that once we have a Gödel numbering for any formal system, we can straightaway form a set of arithmetical rules which complete the Gödel isomorphism. The upshot is that we can transfer the study of any formal system—in fact the study of all formal systems—into number theory.

Now it could be objected here that a coded message, unlike an uncod message, does not express anything on its own—it requires knowledge of the code. But in reality there is no such thing as an uncoded message. There are only messages written in more familiar codes, and message written in less familiar codes. If the meaning of a message is to be revealed it must be pulled out of the code by some sort of mechanism, or isomorphism. It may be difficult to discover the method by which the decoding should be done; but once that method has been discovered, the message becomes transparent as water. When a code is familiar enough, it ceases appearing like a code; one forgets that there is a decoding mechanism. The message is identified with its meaning.

CHAPTER X: Levels of Description, and Computer Systems

Gödel's string G, and a Bach fugue: they both have the property that they can be understood on different levels. We are all familiar with this kind of thing; and yet in some cases it confuses us, while in others we handle it without any difficulty at all. For example, we all know that we human beings are composed of an enormous number of cells (around twenty-five trillion), and therefore that everything we do could in principle be described in terms of cells. Or it could even be described on the level of molecules. Most of us accept this in a rather matter-of-fact way; we go to the doctor, who looks at us on lower levels than we think of ourselves. We read about DNA and "genetic engineering" and sip our coffee. We seem to have reconciled these two inconceivably different pictures of ourselves simply by disconnecting them from each other. We have almost no way to relate a microscopic description of ourselves to that which we feel ourselves to be, and hence it is possible to store separate representations of ourselves in quite separate "compartments" of our minds. Seldom do we have to flip back and forth between these two concepts of ourselves, wondering "How can these two totally different things be the same me?"

One of the major problems of Artificial Intelligence research is to figure out how to bridge the gap between these two descriptions; how to construe a system which can accept one level of description, and produce the other.

There is another place where many levels of description coexist for a system, and where all the levels are conceptually quite close to one another. I am referring to computer systems. When a computer program is running, it can be viewed on a number of levels. On each level, the description is given in the language of computer science, which makes all the descriptions similar in some ways to each other—yet there are extremely important differences between the views one gets on the different levels. At the lowest level, the description can be so complicated that it is like the dot-description of a television picture. For some purposes, however, this is by far the important view. At the highest level, the description is greatly chunked and takes on a completely different feel, despite the fact that many of the same concepts appear on the lowest and highest levels. The chunks on the high-level description are like a chess expert's chunks, and like the chunked description of the image on a screen: they summarize in capsule form a number of things which on lower levels are seen as separate.

The many levels in a complex computer system have the combined effect of "cushioning" the user, preventing him from having to think about the many lower- level goings-on which are most likely totally irrelevant to him anyway. A passenger in an airplane does not usually want to be aware of the levels of fuel in the tanks, or the wind speeds, or how many chicken dinners are to be served, or the status of the rest of the air traffic around the destination—this is all left to employees on different levels of the airlines hierarchy, and the passenger simply gets from one place to another. Here again, it is when something goes wrong—such as his baggage not arriving that the passenger is made aware of the confusing system of levels underneath him.

It is striking how tight the connection is between progress in computer science (particularly Artificial Intelligence) and the development of new languages. A clear trend has emerged in the last decade: the trend to consolidate new types of discoveries in new languages. One key for the understanding and creation of intelligence lies in the constant development and refinement of the languages in terms of which processes for symbol manipulation are describable. Today, there are probably three or four dozen experimental languages which have been developed exclusively for Artificial Intelligence research. It is important to realize that any program which can be written in one of these languages is in principle programmable in lower-level languages, but it would require a supreme effort for a human; and the resulting program would be so long that it would exceed the grasp of humans. It is not that each higher level extends the potential of the computer; the full potential of the computer already exists in its machine language instruction set. It is that the new concepts in a high-level language suggest directions and perspectives by their very nature.

The "space" of all possible programs is so huge that no one can have a sense of what is possible. Each higher-level language is naturally suited for exploring certain regions of "program space"; thus the programmer, by using that language, is channeled into those areas of program space. He is not forced by the language into writing programs of any particular type, but the language makes it easy for him to do certain kinds of things. Proximity to a concept, and a gentle shove, are often all that is needed for a major discovery—and that is the reason for the drive towards languages of ever higher levels.

What is this proverbial distinction between software and hardware? It is the distinction between programs and machines—between long complicated sequences of instructions, and the physical machines which carry them out. I like to think of software as "anything which you could send over the telephone lines", and hardware as "anything else". A piano is hardware, but printed music is software. A telephone set is hardware, but a telephone number is software. The distinction is a useful one, but not always so clear-cut.

We humans also have "software" and "hardware" aspects, and the difference is second nature to us. We are used to the rigidity of our physiology: the fact that we cannot, at will, cure ourselves of diseases, or grow hair of any color—to mention just a couple of simple examples. We can, however, "reprogram" our minds so that we operate in new conceptual frameworks. The amazing flexibility of our minds seems nearly irreconcilable with the notion that our brains must be made out of fixed-rule hardware, which cannot be reprogrammed. We cannot make our neurons fire faster or slower, we cannot rewire our brains, we cannot redesign the interior of a neuron, we cannot make any choices about the hardware—and yet, we can control how we think.

But there are clearly aspects of thought which are beyond our control. We cannot make ourselves smarter by an act of will; we cannot learn a new language as fast as we want; we cannot make ourselves think faster than we do; we cannot make ourselves think about several things at once; and so on. This is a kind of primordial self-knowledge which is so obvious that it is hard to see it at all; it is like being conscious that the air is there. We never really bother to think about what might cause these "defects" of our minds: namely, the organization of our brains. To suggest ways of reconciling the software of mind with the hardware of brain is a main goal of this book.

Each level of science is, in some sense, "sealed off" from the levels below it, recalling the way in which a submarine is built in compartments, so that if one part is damaged, and water begins pouring in, the trouble can be prevented from spreading, by closing the doors, thereby sealing off the damaged compartment from neighboring compartments. Although there is always some "leakage" between the hierarchical levels of science, so that a chemist cannot afford to ignore lower-level physics totally, or a biologist to ignore chemistry totally, there is almost no leakage from one level to a distant level. That is why people can have intuitive understandings of other people without necessarily understanding the quark model, the structure of nuclei, the nature of electron orbits, the chemical bond, the structure of proteins, the organelles in a cell, the methods of intercellular communication, the physiology of the various organs of the human body, or the complex interactions among organs. All at a person needs is a chunked model of how the highest level acts; and as all know, such models are very realistic and successful.

There is, however, perhaps one significant negative feature of a chunked model: it usually does not have exact predictive power. That is, we save ourselves from the impossible task of seeing people as collections of quarks (or whatever is at the lowest level) by using chunked models: but of course such models only give us probabilistic estimates of how other people feel, will react to what we say or do, and so on. In short, in using chunked high-level models, we sacrifice determinism for simplicity. Despite not being sure how people will react to a joke, we tell it with the expectation at they will do something such as laugh, or not laugh—rather than, say, climb the nearest flagpole. (Zen masters might well do the latter!) A chunked model defines a "space" within which behavior is expected to fall, and specifies probabilities of its falling in different parts of that space.

In coming Chapters, where we discuss the brain, we shall examine whether the brain's top level—the mind—can be understood without understanding the lower levels on which it both depends and does not depend. Are there laws of thinking which are "sealed off" from the lower laws that govern the microscopic activity in the cells of the brain? Can mind be "skimmed" off of brain and transplanted into other systems? Or is it impossible to unravel thinking processes into neat and modular subsystems? Is the brain more like an atom, a renormalized electron, a nucleus, a neutron, or a quark? Is consciousness an epiphenomenon? To understand the mind, must one go all the way down to the level of nerve cells?

CHAPTER XI: Brains and Thoughts

It was only with the advent of computers that people actually tried to create "thinking" machines, and witnessed bizarre variations on the theme, of thought. Programs were devised whose "thinking" was to human thinking as a slinky flipping end over end down a staircase is to human locomotion. All of a sudden the idiosyncrasies, the weaknesses and powers, the vagaries and vicissitudes of human thought were hinted at by the newfound ability to experiment with alien, yet hand-tailored forms of thought—or approximations of thought.

Thought must depend on representing reality in the hardware of the brain. In the preceding Chapters, we have developed formal systems which represent domains of mathematical reality in their symbolisms. To what extent is it reasonable to use such formal systems as models for how the brain might manipulate ideas? We saw, in the pq-system and then in other more complicated systems, how meaning, in a limited sense of the term, arose as a result of an isomorphism which maps typographical symbols onto numbers, operations, and relations; and strings of typographical symbols onto statements. Now in the brain we don't have typographical symbols, but we have something even better: active elements which can store information and transmit it and receive it from other active elements. Thus we have active symbols, rather than passive typographical symbols.

Not all descriptions of a person need be attached to some central symbol for that person, which stores the person's name. Descriptions can be manufactured and manipulated in themselves. We can invent nonexistent people by making descriptions of them; we can merge two descriptions when we find they represent a single entity; we can split one description into two when we find it represents two things, not one—and so on. This "calculus of descriptions" is at the heart of thinking. It is said to be intensional and not extensional, which means that descriptions can "float" without being anchored down to specific, known objects. The intensionality of thought is connected to its flexibility; it gives us the ability to imagine hypothetical worlds, to amalgamate different descriptions or chop one description into separate pieces, and so on. A flexible, intensional representation of the world is what thinking is all about. Now how can a physiological system such as the brain support such a system?

The most important cells in the brain are nerve cells, or neurons, of which there are about ten billion. (Curiously, outnumbering the neurons by about ten to one are the glial cells, or glia. Glia are believed to play more of a supporting role to the neurons' starring role, and therefore we will not discuss them.) Each neuron possesses a number of synapses ("entry ports") and one axon ("output channel"). The input and output are electrochemical flows: that is, moving ions. In between the entry ports of a neuron and its output channel is its cell body, where "decisions" are made.

The type of decision which a neuron faces—and this can take place up to a thousand times per second—is this: whether or not to fire—that is, to ease ions down its axon, which eventually will cross over into the entry its of one or more other neurons, thus causing them to make the same sort of decision. The decision is made in a very simple manner: if the sum all inputs exceeds a certain threshold, yes; otherwise, no.

Now we have described the brain's "ants". What about "teams", or "signals"? What about "symbols"? We make the following observation: despite the complexity of its input, a single neuron can respond only in a very primitive way—by firing, or not firing. This is a very small amount of information. Certainly for large amounts of information to be carried or processed, many neurons must be involved. And therefore one might guess at larger structures, composed from many neurons, would exist, which handle concepts on a higher level. This is undoubtedly true, but the most naive assumption—that there is a fixed group of neurons for each different concept—is almost certainly false.

If we looked at my neurons' interconnections, could we find various structures that could be identified as coding for specific things I know, specific beliefs I have, specific hopes, fears, likes and dislikes I harbor? If mental experiences can be attributed to the brain, can knowledge and other aspects of mental life likewise be traced to specific locations inside the brain, or to specific physical subsystems of the brain? This will be a central question to which we will often return in this Chapter and the next.

People have looked for evidence of the "funneling" of many low-level neural responses into fewer and fewer higher-level ones, culminating in something such as the proverbial grandmother cell, or some kind of multineuron network, as mentioned above. It is evident that this will not be found in some gross anatomical division of the brain, but rather in a more microscopic analysis. One possible alternative to the the grandmother cell might be a fixed set of neurons, say a few dozen, at the thin end of the "funnel", all of which fire when Granny comes into view. And for each different recognizable object, there would be a unique network and a funneling process that would focus down onto that network. There are more complicated alternatives along similar lines, involving networks which can be excited in different manners, instead of in a fixed manner. Such networks would be the "symbols" in our brains. But is such funneling necessary? Perhaps an object being looked at is implicitly identified by its "signature" in the visual cortex—that is, the collected responses of simple, complex, and hypercomplex cells. Perhaps the brain does not need any further recognizer for a particular form. This theory, however, poses the following problem. Suppose you are looking at a scene. It registers its signature on your visual cortex; but then how do you get from that signature to a verbal description of the scene?

Philosophically, the most important question of all is this: What would the existence of modules—for instance, a grandmother module—tell us? Would this give us any insight into the phenomenon of our own consciousness? Or would it still leave us as much in the dark about what consciousness is, as does knowledge that a brain is built out of neurons and glia? As you might guess, my feeling is that it would go a long way towards giving us an understanding of the phenomenon of consciousness. The crucial step that needs to be taken is from a low-level—neuron-by-neuron—description of the state of a brain, to a high-level—module-by-module—description of the same state of the same brain. Or, we want to shift the description of the brain state from the signal level to the symbol, level.

Let us from now on refer to these hypothetical neural complexes, neural modules, neural packets, neural networks, multineuron units—call them what you will, whether they come in the form of pancakes, garden rakes, rattlesnakes, snowflakes, or even ripples on lakes—as symbols.

The first thing to emphasize is that symbols can be either dormant, or awake (activated). An active symbol is one which has been triggered—that is, one in which a threshold number of neurons have been caused to fire by stimuli coming from outside. Since a symbol can be triggered in many different ways, it can act in many different ways when awakened. This suggests that we should think of a symbol not as a fixed entity, but as a variable entity.

A high-level description of what makes a symbol active, as distinguished from dormant, would be, "It sends out messages, or signals, whose purpose is to try to awaken, or trigger, other symbols." Of course these messages would be carried as streams of nerve impulses, by neurons—but to the extent that we can avoid such phraseology, we should, for it represents a low-level way of looking at things, and we hope that we can get along on purely a high level. In other words, we hope at thought processes can be thought of as being sealed off from neural events in the same way that the behavior of a clock is sealed off from the laws of quantum mechanics, or the biology of cells is sealed off from the laws of quarks. But what is the advantage of this high-level picture? Why is it better to say, "Symbols A and B triggered symbol C" than to say, "Neurons 183 through 612 excited neuron 75 and caused it to fire"?

It is better because symbols symbolize things, and neurons don't. Symbols are the hardware realizations of concepts. Whereas group of neurons triggering another neuron corresponds to no outer event, the triggering of some symbol by other symbols bears a relation to events in the real world—or in an imaginary world.

Incidentally, the requirement that symbols should be able to pass sophisticated messages to and fro is probably sufficient to exclude neurons themselves from playing the role of symbols. Since a neuron has only a single way of sending information out of itself, and has no way of selectively selecting a signal now in one direction, now in another, it simply does not have the kind of selective triggering power which a symbol must have to act like an object in the real world. In his book The Insect Societies, E. O. Wilson makes a similar point about how messages propagate around inside ant colonies: "[Mass communication] is defined as the transfer, among groups, of information that a single individual could not pass to another."

Let us consider the issue of the size of concepts represented by symbols. Most thoughts expressed in sentences are made up out of basic, quasi-atomic components which we do not usually analyze further. These are of word size, roughly—sometimes a little longer, sometimes a little shorter. For instance, the noun "waterfall", the proper noun "Niagara Falls", the past-tense suffix "-ed", the verb "to catch up with", and longer idiomatic phrases are all close to atomic. These are typical elementary brush strokes which we use in painting portraits of more complex concepts, such as the plot of a movie, the flavor of a city, the nature of consciousness, etc. Such complex ideas are not single brush strokes. It seems reasonable to think that the brush strokes of language are also brush strokes of thought, and therefore that symbols represent concepts of about this size. Thus a symbol would be roughly something for which you know a word or stock phrase, or with which you associate a proper name. And the representation in the brain of a more complex idea, such as a problem in a love affair, would be a very complicated sequence of activations of various symbols by other symbols.

There is a general distinction concerning thinking: that between categories and individuals, or classes and instances. (Two other terms sometimes used are "types" and "tokens".) It might seem at first sight that a given symbol would inherently be either a symbol for a class or a symbol for an instance—but that is an oversimplification. Actually, most symbols may play either role, depending on the context of their activation.

However, the idea that a "class" must always be enormously broad and abstract is far too limited. The reason is that our thought makes use of an ingenious principle, which might be called the prototype principle: The most specific event can serve as a general example of a class of events. Everyone knows that specific events have a vividness which imprints them so strongly on the memory that they can later be used as models for other events which are like them in some way. Thus in each specific event, there is the germ of a whole class of similar events. This idea that there is generality in the specific is of far-reaching importance.

With the enormous and ever-growing repertoire of symbols that exist in each brain, you might wonder whether there eventually comes a point when the brain is saturated—when there is just no more room for a new symbol. This would come about, presumably, if symbols never overlapped each other—if a given neuron never served a double function, so that symbols would be like people getting into an elevator. "Warning: This brain has a maximum capacity of 350,275 symbols!" This is not a necessary feature of the symbol model of brain function, however. In fact, overlapping and completely tangled symbols are probably the rule, so that each neuron, far from being a member of a unique symbol, is probably a functioning part of hundreds of symbols.

Thus we are left with two basic problems in the unraveling of thought processes, as they take place in the brain. One is to explain how the low-level traffic of neuron firings gives rise to the high-level traffic of symbol activations. The other is to explain the high-level traffic of symbol activation in its own terms—to make a theory which does not talk about the low-level neural events. If this latter is possible—and it is a key assumption at the basis of all present research into Artificial Intelligence—then intelligence can be realized in other types of hardware than brains. Then intelligence will have been shown to be a property that can be "lifted" right out of the hardware in which it resides—or in other words, intelligence will be a software property. This will mean that the phenomena of consciousness and intelligence are indeed high-level in the same sense as most other complex phenomena of nature: they have their own high-level laws which depend on, yet are "liftable" out of, the lower levels. If, on the other hand, there is absolutely no way to realize symbol-triggering patterns without having all the hardware of neurons (or simulated neurons), this will imply that intelligence is a brain-bound phenomenon, and much more difficult to unravel than one which owes its existence to a hierarchy of laws on several different levels.

The network by which symbols can potentially trigger each other constitutes the brain's working model of the real universe, as well as of the alternate universes which it considers (and which are every bit as important for the individual's survival in the real world as the real world is).

When the story has been completely told, you have built up quite an elaborate mental model of a scene, and in this model all the objects obey physical law. This means that physical law itself must be implicitly present in the triggering patterns of the symbols. Of course, the phrase "physical law" here does not mean "the laws of physics as expounded by a physicist", but rather the intuitive, chunked laws which all of us have to have in our minds in order to survive.

Needless to say, we have in our brains chunked laws not only of how inanimate objects act, but also of how plants, animals, people and societies act—in other words, chunked laws of biology, psychology, sociology, and so on. All of the internal representations of such entities involve the inevitable feature of chunked models: determinism is sacrificed for simplicity. Our representation of reality ends up being able only to predict probabilities of ending up in certain parts of abstract spaces of behavior—not to predict anything with the precision of physics.

A distinction which is made in Artificial Intelligence is that between procedural and declarative types of knowledge. A piece of knowledge is said to be declarative if it is stored explicitly, so that not only the programmer but also the program can "read" it as if it were in an encyclopedia or an almanac. This usually means that it is encoded locally, not spread around. By contrast, procedural knowledge is not encoded as facts—only as programs. A programmer may be able to peer in and say, "I see that because of these procedures here, the program `knows' how to write English sentences"—but the program itself may have no explicit awareness of how it writes those sentences. For instance, its vocabulary may include none of the words "English", "sentence", and "write" at all! Thus procedural knowledge is usually spread around in pieces, and you can't retrieve it, or "key" on it. It is a global consequence of how the program works, not a local detail. In other words, a piece of purely procedural knowledge is an epiphenomenon. The intuitive or chunked laws of physics and other disciplines mentioned earlier fall mainly on the procedural side; the knowledge that an octopus has eight tentacles falls mainly on the declarative side. In between the declarative and procedural extremes, there are all possible shades.

CHAPTER XII: Minds and Thoughts

Now that we have hypothesized the existence of very high-level active subsystems of the brain (symbols), we may return to the matter of a possible isomorphism, or partial isomorphism, between two brains. Instead of asking about an isomorphism on the neural level (which surely does not exist), or on the macroscopic suborgan level (which surely does exist but does not tell us very much), we ask about the possibility of an isomorphism between brains on the symbol level: a correspondence which not only maps symbols in one brain onto symbols in another brain, but also maps triggering patterns onto triggering patterns. This means that corresponding symbols in the two brains are linked in corresponding ways. This would be a true functional isomorphism—the same type of isomorphism as we spoke of when trying to characterize what it is that is invariant about all butterflies.

On the one hand, we can drop all hopes of finding exactly isomorphic software in humans, but on the other, it is clear that some people think more alike than others do. It would seem an obvious conclusion that there is some sort of partial software isomorphism connecting the brains of people whose style of thinking is similar—in particular, a correspondence of (1) the repertoire of symbols, and (2) the triggering patterns of symbols.

The fact is that a large proportion of every human's network of symbols is universal. We simply take what is common to all of us so much for granted that it is hard to see how much we have in common with other people. It takes the conscious effort of imagining how much—or how little—we have in common with other types of entities, such as stones, cars, restaurants, ants, and so forth, to make evident the large amount of overlap that we have with randomly chosen people. What we notice about another person immediately is not the standard overlap, because that is taken for granted as soon as we recognize the humanity of the other person; rather, we look beyond the standard overlap and generally find some major differences, as well as some unexpected, additional overlap.

If a person can provide a chunked description of any part of his own brain, why shouldn't an outsider too, given some nondestructive means of access to the same brain, not only be able to chunk limited portions of the brain, but actually to give a complete chunked description of it—in other words, a complete documentation of the beliefs of the person whose brain is accessible? It is obvious that such a description would have an astronomical size, but that is not of concern here. We are interested in the question of whether, in principle, there exists a well-defined, high-level description of a brain, or whether, conversely, the neuron-level description—or something equally physiological and intuitively unenlightening—is the best description that in principle exists. Surely, to answer this question would be of the highest importance if we seek to know whether we can ever understand ourselves.

It is my contention that a chunked description is possible, but when we get it, all will not suddenly be clear and light. The problem is that in order to pull a chunked description out of the brain state, we need a language to describe our findings. Now the most appropriate way to describe a brain, it would seem, would be to enumerate the kinds of thoughts it could entertain, and the kinds of thoughts it could not entertain—or, perhaps, to enumerate its beliefs and the things which it does not believe. If that is the kind of goal we will be striving for in a chunked description, then it is easy to see what kinds of troubles we will run up against.

What does this imply? It implies that thoughts which clash totally may be produced by a single brain, depending on the circumstances. And any high-level readout of the brain state which is worth its salt must contain all such conflicting versions. Actually this is quite obvious—that we all are bundles of contradictions, and we manage to hang together by bringing out only one side of ourselves at a given time. The selection cannot be predicted in advance, because the conditions which will force the selection are not known in advance. What the brain state can provide, if properly read, is a conditional description of the selection of routes.

In summary, then, a chunked description of a brain state will consist of a probabilistic catalogue, in which are listed those beliefs which are most likely to be induced (and those symbols which are most likely to be activated) by various sets of "reasonably likely" circumstances, themselves described on a chunked level. Trying to chunk someone's beliefs without referring to context is precisely as silly as trying to describe the range of a single person's "potential progeny" without referring to the mate.

Looking back on what we have discussed, you might think to yourself, "These speculations about brain and mind are all well and good, but what about the feelings involved in consciousness, These symbols may trigger each other all they want, but unless someone perceives the whole thing, there's no consciousness." This makes sense to our intuition on some level, but it does not make much sense logically. For we would then be compelled to look for an explanation of the mechanism which does the perceiving of all the active symbols, if it is not covered by what we have described so far. Of course, a "soulist" would not have to look any further—he would merely assert that the perceiver of all this neural action is the soul, which cannot be described in physical terms, and that is that. However, we shall try to give a "nonsoulist" explanation of where consciousness arises. Our alternative to the soulist explanation—and a disconcerting one it is, too—is to stop at the symbol level and say, "This is it—this is what consciousness is. Consciousness is that property of a system that arises whenever there exist symbols in the system which obey triggering patterns somewhat like the ones described in the past several sections." Put so starkly, this may seem inadequate. How does it account for the sense of "I", the sense of self?

There is no reason to expect that "I", or "the self", should not be represented by a symbol. In fact, the symbol for the self is probably the most complex of all the symbols in the brain. For this reason, I choose to put it on a new level of the hierarchy and call it a subsystem, rather than a symbol. To be precise, by "subsystem", I mean a constellation of symbols, each of which can be separately activated under the control of the subsystem itself. The image I wish to convey of a subsystem is that it functions almost as an independent "subbrain", equipped with its own repertoire of symbols which can trigger each other internally. Of course, there is also much communication between the subsystem and the "outside" world—that is, the rest of the brain. "Subsystem" is just another name for an overgrown symbol, one which has gotten so complicated that it has many subsymbols which interact among themselves. Thus, there is no strict level distinction between symbols and subsystems.

Because of the extensive links between a subsystem and the rest of the brain, it would be very difficult to draw a sharp boundary between the subsystem and the outside; but even if the border is fuzzy, the subsystem is quite a real thing. The interesting thing about a subsystem is that, once activated and left to its own devices, it can work on its own. Thus, two or more subsystems of the brain of an individual may operate simultaneously. I have noticed this happening on occasion in my own brain: sometimes I become aware that two different melodies are running through my mind, competing for "my" attention. Somehow, each melody is being manufactured, or "played", in a separate compartment of my brain. Each of the systems responsible for drawing a melody out of my brain is presumably activating a number of symbols, one after another, completely oblivious to the other system doing the same thing. Then they both attempt to communicate with a third subsystem of my brain—my self-symbol—and it is at that point that the "I" inside my brain gets wind of what's going on; in other words, it starts picking up a chunked description of the activities of those two subsystems.

A very important side effect of the self-subsystem is that it can play the role of "soul", in the following sense: in communicating constantly with the rest of the subsystems and symbols in the brain, it keeps track of what symbols are active, and in what way. This means that it has to have symbols for mental activity—in other words, symbols for symbols, and symbols for the actions of symbols. Of course, this does not elevate consciousness or awareness to any "magical", nonphysical level. Awareness here is a direct effect of the complex hardware and software we have described. Still, despite its earthly origin, this way of describing awareness—as the monitoring of brain activity by a subsystem of the brain itself—seems to resemble the nearly indescribable sensation which we all know and call "consciousness". Certainly one can see that the complexity here is enough that many unexpected effects could be created. For instance, it is quite plausible that a computer program with this kind of structure would make statements about itself which would have a great deal of resemblance to statements which people commonly make about themselves. This includes insisting that it has free will, that it is not explicable as a "sum of its parts", and so on.

What kind of guarantee is there that a subsystem, such as I have here postulated, which represents the self, actually exists in our brains? Could a whole complex network of symbols such as has been described above evolve without a self-symbol evolving? How could these symbols and their activities play out "isomorphic" mental events to real events in the surrounding universe, if there were no symbol for the host organism, All the stimuli coming into the system are centered on one small mass in space. It would be quite a glaring hole in a brain's symbolic structure not to have a symbol for the physical object in which it is housed, and which plays a larger role in the events it mirrors than any other object. In fact, upon reflection, it seems that the only way one could make sense of the world surrounding a localized animate object is to understand the role of that object in relation to the other objects around it. This necessitates the existence of a self-symbol; and the step from symbol to subsystem is merely a reflection of the importance of the self- symbol, and is not a qualitative change.

CHAPTER XIII: BlooP and FlooP and GlooP

BlooP, FlooP, and GlooP are three computer languages, each one with is own special purpose. These languages were invented specially for this chapter. They will be of use in explaining some new senses of the word "recursive"—in particular, the notions of primitive recursivity and general recursivity. They will prove very helpful in clarifying the machinery of self-reference in TNT.

From here on out, the representability of all primitive recursive truths will be the criterion for calling a system "sufficiently powerful". The significance of the notion is shown by the following key fact: If you have a sufficiently powerful formalization of number theory, then Gödel's method is applicable, and consequently your system is incomplete. If, on the other hand, your system is not sufficiently powerful (i.e., not all primitive recursive truths are theorems), then your system is, precisely by virtue of that lack, incomplete.

BlooP is our language for defining predictably terminating calculations. The standard name for functions which are BlooP-computable is primitive recursive functions; and the standard name for properties which can be detected by BlooP-tests is primitive recursive predicates.

If a BlooP test can be written for some property of natural numbers, then that property is represented in TNT.

BlooP's defining feature is the boundedness of its loops. What if we drop that requirement on loops, and invent a second language, called "FlooP" ('F' for "free")? FlooP will be identical to BlooP except in one respect: we may have loops without ceilings, as well as loops with ceilings (although the only reason one would include a ceiling when writing a loop-statement in FlooP would be for the sake of elegance). It would seem extremely desirable to be able to separate FlooP procedures into two classes: terminators and nonterminators. A terminator will eventually halt no matter what its input. A nonterminator will go on and on forever, for at least one choice of input. If we could always tell, by some kind of complicated inspection of a FlooP program, to which class it belonged, there would be some remarkable repercussions. In a sense, it would be like having a magical dowsing rod which could solve all problems of number theory in one swell FlooP.

If FlooP is BlooP unchained, then GlooP must be FlooP unchained. But how can you take the chains off twice? How do you make a language whose power transcends that of FlooP? We have found a function whose values we humans know how to calculate—the method of doing so has been explicitly described in English—but which seemingly cannot be programmed in the language FlooP. This is a serious dilemma because no one has ever found any more powerful computer language than FlooP. Careful investigation into the power of computer languages has been carried out. We need not do it ourselves; let it just be reported that there is a vast class of computer languages all of which can be proven to have exactly the same expressive power as FlooP does, in this sense: any calculation which can be programmed in any one of the languages can be programmed in them all. The curious thing is that almost any sensible attempt at designing a computer language ends up by creating a member of this class—which is to say, a language of power equal to that of FlooP. The point is that there are some extremely natural ways to go about inventing algorithmic languages; and different people, following independent routes, usually wind up creating equivalent languages, with the only difference being style, rather than power.

In fact, it is widely believed that there cannot be any more powerful language for describing calculations than languages that are equivalent to FlooP. This hypothesis was formulated in the 1930's by two people, independently of each other: Alan Turing and Alonzo Church, one of the eminent logicians of this century. It is called the Church-Turing Thesis. If we accept the CT-Thesis, we have to conclude that "GlooP" is a myth—there are no restrictions to remove in FlooP, no ways to increase its power by "unshackling" it, as we did BlooP. Now we have double the reason for believing that any termination test is a myth—that there is no way to put FlooP programs in a centrifuge and separate out the terminators from the nonterminators.

Here are three related ways to state the CT-Thesis:

1. What is human-computable is machine-computable.

2. What is machine-computable is FlooP-computable.

3. What is human-computable is FlooP-computable (i.e., general or partial recursive).

As has already been mentioned, “BlooP-computable” is synonymous with “primitive recursive”. Now FlooP computable functions can be divided into two realms: (1) those which are computable by terminating FlooP programs: these are said to be general recursive; and (2) those which are computable only by nonterminating FlooP programs: these are said to be partial recursive. (Similarly for predicates.) People often just say "recursive" when they mean "general recursive".

CHAPTER XIV: On Formally Undecidable Propositions of TNT and Related Systems

I will stress the two key ideas which are at the core of Gödel's proof. The first key idea is the deep discovery that there are strings of TNT which can be interpreted as speaking about other strings of TNT; in short, that TNT, as a language, is capable of "introspection", or self-scrutiny. This is what comes from Gödel-numbering. The second key idea is that the property of self scrutiny can be entirely concentrated into a single string; thus that string's sole focus of attention is itself. This "focusing trick" is traceable, in essence, to the Cantor diagonal method.

In my opinion, if one is interested in understanding Gödel's proof in a deep way, then one must recognize that the proof, in its essence, consists of a fusion of these two main ideas. Each of them alone is a master stroke; to put them together took an act of genius. If I were to choose, however, which of the two key ideas is deeper, I would unhesitatingly pick the first one—the idea of Gödel-numbering, for that idea is related to the whole notion of what meaning and reference are, in symbol-manipulating systems. This is an idea which goes far beyond the confines of mathematical logic, whereas the Cantor trick, rich though it is in mathematical consequences, has little if any relation to issues in real life.

CHAPTER XV: Jumping out of the System

Any system, no matter how complex or tricky it is, can be Gödel-numbered, and then the notion of its proof-pairs can be defined—and this is the petard by which it is hoist. Once a system is well-defined, or "boxed", it becomes vulnerable. So what is to be done? There is no end in sight. It appears that TNT, even when extended ad infinitum, cannot be made complete. TNT is therefore said to suffer from essential incompleteness because the incomepleteness here is part and parcel of TNT; it is an essential part of the nature of TNT and cannot be eradicated in any way, whether simpleminded or ingenious. What's more, this problem will haunt any formal version of number theory, whether it is an extension of TNT, a modification of TNT, or an alternative to TNT.

The fascinating thing is that any such system digs its own hole; the system's own richness brings about its own downfall. The downfall occurs essentially because the system is powerful enough to have self-referential sentences. In physics, the notion exists of a "critical mass" of a fissionable substance, such as uranium. A solid lump of the substance will just sit there, if its mass is less than critical. But beyond the critical mass, such a lump will undergo a chain reaction, and blow up. It seems that with formal systems there is an analogous critical point. Below that point, a system is "harmless" and does not even approach defining arithmetical truth formally; but beyond the critical point, the system suddenly attains the capacity for self-reference, and thereby dooms itself to incompleteness. Once this ability for self-reference is attained, the system has a hole which is tailor-made for itself; the hole takes the features of the system into account and uses them against the system.

It is still of great interest to ponder whether we humans ever can jump out of ourselves—or whether computer programs can jump out of themselves. Certainly it is possible for a program to modify itself—but such modifiability has to be inherent in the program to start with, so that cannot be counted as an example of "jumping out of the system". No matter how a program twists and turns to get out of itself, it is still following the rules inherent in itself. It is no more possible for it to escape than it is for a human being to decide voluntarily not to obey the laws of physics. Physics is an overriding system, from which there can be no escape. However, there is a lesser ambition which it is possible to achieve: that is, one can certainly jump from a subsystem of one's brain into a wider subsystem. One can step out of ruts on occasion. This is still due to the interaction of various subsystems of one's brain, but it can feel very much like stepping entirely out of oneself. Similarly, it is entirely conceivable that a partial ability to "step outside of itself" could be embodied in a computer program.

However, it is important to see the distinction between perceiving oneself, and transcending oneself. You can gain visions of yourself in all sorts of ways—in a mirror, in photos or movies, on tape, through the descriptions if others, by getting psychoanalyzed, and so on. But you cannot quite break out of your own skin and be on the outside of yourself. TNT can talk about itself, but it cannot jump out of itself. A computer program can modify itself but it cannot violate its own instructions—it can at best change some parts of itself by obeying its own instructions. This is reminiscent of the humerous paradoxical question, "Can God make a stone so heavy that he can't lift it?"

In Zen, too, we can see this preoccupation with the concept of transcending the system. Perhaps, self-transcendence is even the central theme of Zen. A Zen person is always trying to understand more deeply what he is, by stepping more and more out of what he sees himself to be, by breaking every rule and convention which he perceives himself to be chained by—needless to say, including those of Zen itself. Somewhere along this elusive path may come enlightenment. In any case (as I see it), the hope is that by gradually deepening one's self-awareness, by gradually widening the scope of "the system", one will in the end come to a feeling of being at one with the entire universe.

CHAPTER XVI: Self-Ref and Self-Rep

Let us look at sentences which, at first glance, may seem to provide the simplest examples of self-reference. Some such sentences are these:

1. This sentence contains five words.

2. This sentence is meaningless because it is self-referential.

3. This sentence no verb.

4. This sentence is false. (Epimenides paradox)

5. The sentence I am now writing is the sentence you are now reading.

All but the last one (which is an anomaly) involve the simple-seeming mechanism contained in the phrase "this sentence". But that mechanism is in reality far from simple. All of these sentences are "floating" in the context of the English language. They can be compared to icebergs, whose tips only are visible. The word sequences are the tips of the icebergs, and the processing which must be done to understand them is the hidden part. In this sense their meaning is implicit, not explicit. Of course, no sentence's meaning is completely explicit, but the more explicit the self-reference is, the more exposed will be the mechanisms underlying it. In this case, for the self-reference of the sentences above to be recognized, not only has one to be comfortable with a language such as English which can deal with linguistic subject matter, but also one has to be able to figure out the referent of the phrase "this sentence". It seems simple, but it depends on our very complex yet totally assimilated ability to handle English.

Besides the question "What constitutes a copy?", there is another fundamental philosophical question concerning self-reps. That is the obverse side of the coin: "What is the original?" This can best be explained by referring to some examples:

1. a program which, when interpreted by some interpreter running on some computer, prints itself out;

2. a program which, when interpreted by some interpreter running on some computer, prints itself out along with a complete copy of the interpreter (which, after all, is also a program);

3. a program which, when interpreted by some interpreter running on some computer, not only prints itself out along with a complete copy of the interpreter, but also directs a mechanical assembly process in which a second computer, identical to the one on which the interpreter and program are running, is put together.

It is clear that in (1), the program is the self-rep. But in (3), is it the program which is the self-rep, or the compound system of program plus interpreter, or the union of program, interpreter, and processor?

Clearly, a self-rep can involve more than just printing itself out. In fact, most of the rest of this Chapter is a discussion of self-reps in which data, program, interpreter, and processor are all extremely intertwined, and in which self-replication involves replicating all of them at once.

The Central Dogmap establishes an analogy between two fundamental Tangled Hierarchies: that of molecular biology and that of mathematical logic. There is something almost mystical in seeing the deep sharing of such an abstract structure by these two esoteric, yet fundamental, advances in knowledge achieved in our century. This Central Dogmap is by no means a rigorous proof of identity of the two theories; but it clearly shows a profound kinship, which is worth deeper exploration.

Dogma I (Molecular Biology)	Dogma II (Mathematical Logic)
strands of DNA	strings of TNT
strands of mRNA	statements of N
proteins	statements of meta-TNT
proteins which act on proteins	statements about statements of meta-TNT
proteins which act on proteins which act on proteins	statements about statements about statements of meta-TNT
transcription (DNA => RNA)	interpretation (TNT => N)
Translation (RNA => proteins)	Arithmetization (N => meta-TNT)
Crick	Gödel
Genetic Code (arbitrary convention)	Gödel Code (arbitrary convention)
codon (triplet of bases)	codon (triplet of digits)
amino acid	quoted symbol of TNT used in meta-TNT
self-reproduction	self-reference
sufficiently strong cellular support system to permit self-rep	sufficiently powerful arithmetical formal system to permit self-ref

One of the more interesting similarities between the two sides of the map is the way in which "loops" of arbitrary complexity arise on the top level of both: on the left, proteins which act on proteins which act on proteins and so on, ad infinitum; and on the right, statements about statements about statements of meta-TNT and so on, ad infinitum. These are like heterarchies, where a sufficiently complex substratum allows high-level Strange Loops to occur and to cycle around, totally sealed off from lower levels.

CHAPTER XVII: Church, Turing, Tarski, and Others

We have come to the point where we can develop one of the main theses of this book: that every aspect of thinking can be viewed as a high-level description of a system which, on a low level, is governed by simple, even formal, rules. The "system", of course, is a brain—unless one is speaking of thought processes flowing in another medium, such as a computer's circuits. The image is that of a formal system underlying an "informal system"—a system which can, for instance, make puns, discover number patterns, forget names, make awful blunders in chess, and so forth. This is what one sees from the outside: its informal, overt, software level. By contrast, it has a formal, hidden, hardware level (or "substrate") which is a formidably complex mechanism that makes transitions from state to state according to definite rules physically embodied in it, and according to the input of signals which impinge on it.

Artificial Intelligence is often referred to as "AI". Often, when I try to explain what is meant by the term, I say that the letters "AI" could just as well stand for "Artificial Intuition", or even "Artificial Imagery". The aim of Al is to get at what is happening when one's mind silently and invisibly chooses, from a myriad alternatives, which one makes most sense in a very complex situation. In many real-life situations, deductive reasoning is inappropriate, not because it would give wrong answers, but because there are too many correct but irrelevant statements which can be made; there are just too many things to take into account simultaneously for reasoning alone to be sufficient. A sense of judgment—"What is important here, and what is not?"—is called for. Tied up with this is a sense of simplicity, a sense of beauty. Where do these intuitions come from? How can they emerge from an underlying formal system?

CHURCH-TURING THESIS, ISOMORPHISM VERSION: Suppose there is a method which a sentient being follows in order to sort numbers into two classes. Suppose further that this method always yields an answer within a finite amount of time, and that it always gives the same answer for a given number. Then: Some terminating FlooP program (i.e., general recursive function) exists which gives exactly the same answers as the sentient being's method does. Moreover: The mental process and the FlooP program are isomorphic in the sense that on some level there is a correspondence between the steps being carried out in both computer and brain.

CHURCH-TURING THESIS, REDUCTIONIST'S VERSION: All brain processes are derived from a computable substrate.

This statement is about the strongest theoretical underpinning one could give in support of the eventual possibility of realizing Artificial Intelligence. Of course, Artificial Intelligence research is not aimed at simulating neural networks, for it is based on another kind of faith: that probably there are significant features of intelligence which can be floated on top of entirely different sorts of substrates than those of organic brains.

There is no reason to believe that a computer's faultlessly functioning hardware could not support high-level symbolic behavior which would represent such complex states as confusion, forgetting, or appreciation of beauty. It would require that there exist massive subsystems interacting with each other according to a complex "logic". The overt behavior could appear either rational or irrational; but underneath it would be the performance of reliable, logical hardware.

CHURCH-TURING THESIS, AI VERSION: Mental processes of any sort can be simulated by a computer program whose underlying language is of power equal to that of FlooP—that is, in which all partial recursive functions can be programmed.

It should also be pointed out that in practice, many AI researchers rely on another article of faith which is closely related to the CT-Thesis, and which I call the AI Thesis. It runs something like this:

AI THESIS: As the intelligence of machines evolves, its underlying mechanisms will gradually converge to the mechanisms underlying human intelligence.

In other words, all intelligences are just variations on a single theme; to create true intelligence, AI workers will just have to keep pushing to ever lower levels, closer and closer to brain mechanisms, if they wish their machines to attain the capabilities which we have.

CHAPTER XVIII: Artificial Intelligence: Retrospects

There is a "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem. "AI is whatever hasn't been done yet."

The issue of a program outdoing its programmer is connected with the question of "originality" in AI. What if an AI program comes up with an idea, or a line of play in a game, which its programmer has never entertained—who should get the credit? It seems reasonable to say that if one can ascribe the performance to certain operations which are easily traced in the program, then in some sense the program was just revealing ideas which were in essence hidden—though not too deeply—inside the programmer's own mind. Conversely, if following the program does not serve to enlighten one as to why this particular discovery popped out, then perhaps one should begin to separate the program's "mind" from that of its programmer. The human gets credit for having invented the program, but not for having had inside his own head the ideas produced by the program. In such cases, the human can be referred to as the "meta-author"—the author of the author of the result, and the program as the (just plain) author.

This brings up a question which is a slight digression from AI, but actually not a huge one. It is this: When you see the word "I" or "me" in a text, what do you take it to be referring to? How far back do we ordinarily trace the "I" in a sentence? The answer, it seems to me, is that we look for a sentient being to attach the authorship to. But what is a sentient being? Something onto which we can map ourselves comfortably.

This brings us back to the issue of the "who" who composes computer music. In most circumstances, the driving force behind such pieces is a human intellect, and the computer has been employed, with more or less ingenuity, as a tool for realizing an idea devised by the human. The program which carries this out is not anything which we can identify with. It is a simple and single-minded piece of software with no flexibility, no perspective on what it is doing, and no sense of self. If and when, however, people develop programs which have those attributes, and pieces of music start issuing forth from them, then I suggest that will be the appropriate, time to start splitting up one's admiration: some to the programmer for creating such an amazing program, and some to the program itself for its sense of music. And it seems to me that that will only take place when the internal structure of such a program is based on something similar to the "symbols" in our brains and their triggering patterns, which are responsible for the complex notion of meaning. The fact of having this kind of internal structure would endow the program with properties which would make us feel comfortable in identifying with it, to some extent. But until then, I will not feel comfortable in saying "this piece was composed by a computer".

In some sense all problems are abstract versions of the dog-and-bone problem. Many problems are not in physical space but in some sort of conceptual space. When you realize that direct motion towards the goal in that space runs you into some sort of abstract "fence", you can do one of two things: (1) try moving away from the goal in some sort of random way, hoping that you may come upon a hidden "gate" through which you can pass and then reach your bone; or (2) try to find a new "space" in which you can represent the problem, and in which there is no abstract fence separating you from your goal—then you can proceed straight towards the goal in this new space. The first method may seem like the lazy way to go, and the second method may seem like a difficult and complicated way to go. And yet, solutions which involve restructuring the problem space more often than not come as sudden flashes of insight rather than as products of a series of slow, deliberate thought processes. Probably these intuitive flashes come from the extreme core of intelligence—and, needless to say, their source is a closely protected secret of our jealous brains.

In any case, the trouble is not that problem reduction per se leads to failures; it is quite a sound technique. The problem is a deeper one: how do you choose a good internal representation for a problem? What kind of "space" do you see it in? What kinds of action reduce the "distance" between you and your goal in the space you have chosen? This can be expressed in mathematical language as the problem of hunting for an approprate metric (distance function) between states. You want to find a metric in which the distance between you and your goal is very small. What AI sorely lacks is programs which can "step back" and take a look at what is going on, and with this perspective, reorient themselves to the task at hand. It is one thing to write a program which excels at a single task which, when done by a human being, seems to require intelligence—and it is another thing altogether to write an intelligent program! It is the difference between the Sphex wasp, whose wired-in routine gives the deceptive appearance of great intelligence, and a human being observing a Sphex wasp.

An intelligent program would presumably be one which is versatile enough to solve problems of many different sorts. It would learn to do each different one and would accumulate experience in doing so. It would be able to work within a set of rules and yet also, at appropriate moments, to step back and make a judgment about whether working within that set of rules is likely to be profitable in terms of some overall set of goals which it has. It would be able to choose to stop working within a given framework, if need be, and to create a new framework of rules within which to work for a while.

Much of this discussion may remind you of the Mechanical mode and the Intelligent mode. In the former, you are embedded within some fixed framework; in the latter, you can always step back and gain an overview of things. Having an overview is tantamount to choosing a representation within which to work; and working within the rules of the system is tantamount to trying the technique of problem reduction within that selected framework.

The Sphex wasp operates excellently in the M-mode, but it has absolutely no ability to choose its framework or even to alter its M-mode in the slightest. It has no ability to notice when the same thing occurs over and over and over again in its system, for to notice such a thing would be to jump out of the system, even if only ever so slightly. It simply does not notice the sameness of the repetitions. This idea (of not noticing the identity of certain repetitive events) is interesting when we apply it to ourselves. Are there highly repetitious situations which occur in our lives time and time again, and which we handle in the identical stupid way each time, because we don't have enough of an overview to perceive their sameness? This leads back to that recurrent issue, "What is sameness?" It will soon come up as an AI theme, when we discuss pattern recognition.

When you realize that knowledge representation is an altogether different ball game than mere storage of numbers, then the idea that "a computer has the memory of an elephant" is an easy myth to explode. What is stored in memory is not necessarily synonymous with what a program knows; for even if a given piece of knowledge is encoded somewhere inside a complex system, there may be no procedure, or rule, or other type of handler of data, which can get at it—it may be inaccessible. In such a case, you can say that the piece of knowledge has been "forgotten" because access to it has been temporarily or permanently lost. Thus a computer program may "forget" something on a high level which it "remembers" on a low level. This is another one of those ever-recurring level distinctions, from which we can probably learn much about our own selves. When a human forgets, it most likely means that a high-level pointer has been lost—not that any information has been deleted or destroyed. This highlights the extreme importance of keeping track of the ways in which you store incoming experiences, for you never know in advance under what circumstances, or from what angle, you will want to pull something out of storage.

What kind of program would it take to make human beings admit that it had some "understanding", even if begrudgingly? What would it take before you wouldn't feel intuitively that there is "nothing there"?

CHAPTER XIX: Artificial Intelligence: Prospects

In everyday thought, we are constantly manufacturing mental variants on situations we face, ideas we have, or events that happen, and we let some features stay exactly the same while others "slip". What features do we let slip? What ones do we not even consider letting slip? What events are perceived on some deep intuitive level as being close relatives of ones which really happened? What do we think "almost" happened or "could have" happened, even though it unambiguously did not? What alternative versions of events pop without any conscious thought into our minds when we hear a story? Why do some counterfactuals strike us as "less counterfactual" than other counterfactuals? After all, it is obvious that anything that didn't happen didn't happen. There aren't degrees of "didn't-happen-ness". And the same goes for "almost" situations. There are times when one plaintively says, "It almost happened", and other times when one says the same thing, full of relief. But the "almost" lies in the mind, not in the external facts.

Consider how natural it feels to slip from the valueless declarative "I don't know Russian" to the more charged conditional "I would like to know Russian" to the emotional subjunctive "I wish I knew Russian" and finally to the rich counterfactual "If I knew Russian, I would read Chekhov and Lermontov in the original". How flat and dead would be a mind that saw nothing in a negation but an opaque barrier! A live mind can see a window onto a world of possibilities. I believe that "almost" situations and unconsciously manufactured subjunctives represent some of the richest potential sources of insight into how human beings organize and categorize their perceptions of the world.

The manufacture of "subjunctive worlds" happens so casually, so naturally, that we hardly notice what we are doing. We select from our fantasy a world which is close, in some internal mental sense, to the real world. We compare what is real with what we perceive as almost real. In so doing, what we gain is some intangible kind of perspective on reality. Think how immeasurably poorer our mental lives would be if we didn't have this creative capacity for slipping out of the midst of reality into soft "what if"'s! And from the point of view of studying human thought processes, this slippage is very interesting, for most of the time it happens completely without conscious direction, which means that observation of what kinds of things slip, versus what kinds don't, affords a good window on the unconscious mind.

It seems to me that the slippability of a feature of some event (or circumstance) depends on a set of nested contexts in which the event (or circumstance) is perceived to occur. We build up our mental representation of a situation layer by layer. The lowest layer establishes the deepest aspect of the context—sometimes being so low that it cannot vary at all. For instance, the three-dimensionality of our world is so ingrained that most of us never would imagine letting it slip mentally. It is a constant constant. Then there are layers which establish temporarily, though not permanently, fixed aspects of situations, which could be called background assumptions—things which, in the back of your mind, you know can vary, but which most of the time you unquestioningly accept as unchanging aspects. These could still be called "constants". For instance, when you go to a football game, the rules of the game are constants of that sort. Then there are "parameters": you think of them as more variable, but you temporarily hold them constant. At a football game, parameters might include the weather, the opposing team, and so forth. There could be—and probably are—several layers of parameters. Finally, we reach the "shakiest" aspects of your mental representation of the situation—the variables. These are things such as a player's stepping out of bounds, which are mentally "loose" and which you don't mind letting slip away from their real values, for a short moment.

The word frame is in vogue in AI currently, and it could be defined as a computational instantiation of a context. The term is due to Marvin Minsky, as are many ideas about frames, though the general concept has been floating around for a good number of years. In frame language, one could say that mental representations of situations involve frames nested within each other. Each of the various ingredients of a situation has its own frame.

One of the main ideas about frames is that each frame comes with its own set of expectations. The corresponding image is that each chest of drawers comes with a built-in, but loosely bound, drawer in each of its drawer slots, called a default. If I tell you, "Picture a river bank", you will invoke a visual image which has various features, most of which you could override if I added extra phrases such as "in a drought" or "in Brazil" or "without a merry-go-round". The existence of default values for slots allows the recursive process of filling slots to come to an end. In effect, you say, "I will fill in the slots myself as far as three layers down; beyond that I will take the default options." Together with its default expectations, a frame contains knowledge of its limits of applicability, and heuristics for switching to other frames in case it has been stretched beyond its limits of tolerance. The nested structure of a frame gives you a way of "zooming in" and looking at small details from as close up as you wish: you just zoom in on the proper subframe, and then on one of its subframes, etc., until you have the desired amount of detail.

One way that has been suggested for handling the complexities of pattern recognition and other challenges to AI programs is the so-called "actor" formalism of Carl Hewitt (similar to the language "Smalltalk", developed by Alan Kay and others), in which a program is written as a collection of interacting actors, which can pass elaborate messages back and forth among themselves. In a way, this resembles a heterarchical collection of procedures which can call each other. The major difference is that where procedures usually only pass a rather small number of arguments back and forth, the messages exchanged by actors can be arbitrarily long and complex. Actors with the ability to exchange messages become somewhat autonomous agents—in fact, even like autonomous computers, with messages being somewhat like programs. Each actor can have its own idiosyncratic way of interpreting any given message; thus a message's meaning will depend on the actor it is intercepted by. This comes about by the actor having within it a piece of program which interprets messages; so there may be as many interpreters as there are actors.

It is interesting to think how one might merge the frame-notion with the actor-notion. Let us call a frame with the capability of generating and interpreting complex messages a symbol:

frame + actor = symbol

A view which has been abstracted from a concept along some dimension is what I call a conceptual skeleton. In effect, we have dealt with conceptual skeletons all along, without often using that name. It is always of interest, and possibly of importance, when two or more ideas are discovered to share a conceptual skeleton.

A conceptual skeleton is like a set of constant features (as distinguished from parameters or variables)—features which should not be slipped in a subjunctive instant replay or mapping-operation. Having no parameters or variables of its own to vary, it can be the invariant core of several different ideas. Each instance of it does have layers of variability and so can be "slipped" in various ways. Although the name "conceptual skeleton" sounds absolute and rigid, actually there is a lot of play in it. There can be conceptual skeletons on several different levels of abstraction.

One of the major characteristics of each idiosyncratic style of thought is how new experiences get classified and stuffed into memory, for that defines the "handles" by which they will later be retrievable. And for events, objects, ideas, and so on—for everything that can be thought about—there is a wide variety of "handles". I am struck by this each time I reach down to turn on my car radio, and find, to my dismay, that it is already on! What has happened is that two independent representations are being used for the radio. One is "music producer", the other is "boredom reliever". I am aware that the music is on, but I am bored anyway, and before the two realizations have a chance to interact, my reflex to reach down has been triggered. The same reaching-down reflex one day occurred just after I'd left the radio at a repair shop and was driving away, wanting to hear some music. Odd.

When two ideas are seen to share conceptual skeletons on some level of abstraction, different things can happen. Usually the first stage is that you zoom in on both ideas, and, using the higher-level match as a guide, you try to identify corresponding subideas. Sometimes the match can be extended recursively downwards several levels, revealing a profound isomorphism. Sometimes it stops earlier, revealing an analogy or similarity. And then there are times when the high-level similarity is so compelling that, even if there is no apparent lower-level continuation of the map, you just go ahead and make one: this is the forced match.

Let me try to tie things together a little. I have presented a number of related ideas connected with the creation, manipulation, and comparison of symbols. Most of them have to do with slippage in some fashion, the idea being that concepts are composed of some tight and some loose elements, coming from different levels of nested contexts (frames). The loose ones can be dislodged and replaced rather easily, which, depending on the circumstances, can create a "subjunctive instant replay", a forced match, or an analogy. A fusion of two symbols may result from a process in which parts of each symbol are dislodged and other parts remain.

It is obvious that we are talking about mechanization of creativity. But is this not a contradiction in terms? Almost, but not really. Creativity is the essence of that which is not mechanical. Yet every creative act is mechanical—it has its explanation no less than a case of the hiccups does. The mechanical substrate of creativity may be hidden from view, but it exists. Conversely, there is something unmechanical in flexible programs, even today. It may not constitute creativity, but when programs cease to be transparent to their creators, then the approach to creativity has begun.

It is a common notion that randomness is an indispensable ingredient of creative acts. This may be true, but it does not have any bearing on the mechanizability—or rather, programmability!—of creativity. The world is a giant heap of randomness; when you mirror some of it inside your head, your head's interior absorbs a little of that randomness. The triggering patterns of symbols, therefore, can lead you down the most random-seeming paths, simply because they came from your interactions with a crazy, random world. So it can be with a computer program, too. Randomness is an intrinsic feature of thought, not something which has to be "artificially inseminated", whether through dice, decaying nuclei, random number tables, or what-have-you. It is an insult to human creativity to imply that it relies on such arbitrary sources.

CHAPTER XX: Strange Loops, Or Tangled Hierarchies

Pause to think where your sense of having a will comes from. Unless you are a soulist, you'll probably say that it comes from your brain—a piece of hardware which you did not design or choose. And yet that doesn't diminish your sense that you want certain things, and not others. You aren't a "self-programmed object" (whatever that would be), but you still do have a sense of desires, and it springs from the physical substrate of your mentality. Likewise, machines may someday have wills despite the fact that no magic program spontaneously appears in memory from out of nowhere (a "self-programmed program"). They will have wills for much the same reason as you do—by reason of organization and structure on many levels of hardware and software.

I wrote that a central issue of this book would be: "Do words and thoughts follow formal rules?" One major thrust of the book has been to point out the many-leveledness of the mind/brain, and I have tried to show why the ultimate answer to the question is, "Yes—provided that you go down to the lowest level—the hardware—to find the rules."

When we humans think, we certainly do change our own mental rules, and we change the rules that change the rules, and on and on—but these are, so to speak, "software rules". However, the rules at bottom do not change. Neurons run in the same simple way the whole time. You can't "think" your neurons into running some nonneural way, although you can make your mind change style or subject of thought. You have access to your thoughts but not to your neurons. Software rules on various levels can change; hardware rules cannot—in fact, to their rigidity is due the software's flexibility! Not a paradox at all, but a fundamental, simple fact about the mechanisms of intelligence. This distinction between self-modifiable software and inviolate hardware is what I wish to pursue in this final Chapter.

In any system there is always some "protected" level which is unassailable by the rules on other levels, no matter how tangled their interaction may be among themselves. A classic variation on our theme is the Escher picture of Drawing Hands. Here, a left hand (LH) draws a right hand (RH), while at the same time, RH draws LH. Once again, levels which ordinarily are seen as hierarchical—that which draws, and that which is drawn—turn back on each other, creating a Tangled Hierarchy. But the theme of the Chapter is borne out, of course, since behind it all lurks the undrawn but drawing hand of M. C. Escher, creator of both LH and RH. Escher is outside of the two-hand space.

Now we can relate this to the brain, as well as to AI programs. In our thoughts, symbols activate other symbols, and all interact heterarchically. Furthermore, the symbols may cause each other to change internally, in the fashion of programs acting on other programs. The illusion is created, because of the Tangled Hierarchy of symbols, that there is no inviolate level. One thinks there is no such level because that level is shielded from our view.

If it were possible to schematize this whole image, there would be a gigantic forest of symbols linked to each other by tangly lines like vines in a tropical jungle—this would be the top level, the Tangled Hierarchy where thoughts really flow back and forth. This is the elusive level of mind: the analogue to LH and RH. Far below in the schematic picture, analogous to the invisible "prime mover" Escher, there would be a representation of the myriad neurons—the "inviolate substrate" which lets the tangle above it come into being. Interestingly, this other level is itself a tangle in a literal sense—billions of cells and hundreds of billions of axons, joining them all together.

If we look only at the symbol tangle, and forget the neural tangle, then we seem to see a self-programmed object—in just the same way as we seem to see a self-drawn picture if we look at Drawing Hands and somehow fall for the illusion, by forgetting the existence of Escher. For the picture, this is unlikely—but for humans and the way they look at their minds, this is usually what happens. We feel self-programmed. Indeed, we couldn't feel any other way, for we are shielded from the lower levels, the neural tangle. Our thoughts seem to run about in their own space, creating new thoughts and modifying old ones, and we never notice any neurons helping us out! But that is to be expected. We can't.

My feeling is that the process by which we decide what is valid or what is true is an art; and that it relies as deeply on a sense of beauty and simplicity as it does on rock-solid principles of logic or reasoning or anything else which can be objectively formalized. I am not saying either (1) truth is a chimera, or (2) human intelligence is in principle not programmable. I am saying (1) truth is too elusive for any human or any collection of humans ever to attain fully; and (2) Artificial Intelligence, when it reaches the level of human intelligence—or even if it surpasses it—will still be plagued by the problems of art, beauty, and simplicity, and will run up against these things constantly in its own search for knowledge and understanding.

One of the most severe of all problems of evidence interpretation is that of trying to interpret all the confusing signals from the outside as to who one is. In this case, the potential for intralevel and interlevel conflict is tremendous. The psychic mechanisms have to deal simultaneously with the individual's internal need for self-esteem and the constant flow of evidence from the outside affecting the self-image. The result is that information flows in a complex swirl between different levels of the personality; as it goes round and round, parts of it get magnified, reduced, negated, or otherwise distorted, and then those parts in turn get further subjected to the same sort of swirl, over and over again—all of this in an attempt to reconcile what is, with what we wish were. The upshot is that the total picture of "who I am" is integrated in some enormously complex way inside the entire mental structure, and contains in each one of us a large number of unresolved, possibly unresolvable, inconsistencies. These undoubtedly provide much of the dynamic tension which is so much a part of being human.

I think it can have suggestive value to translate Gödel's Theorem into other domains, provided one specifies in advance that the translations are metaphorical and are not intended to be taken literally. That having been said, I see two major ways of using analogies to connect Gödel's Theorem and human thoughts. One involves the problem of wondering about one's sanity. How can you figure out if you are sane? This is a Strange Loop indeed. Once you begin to question your own sanity, you can get trapped in an ever-tighter vortex of self-fulfilling prophecies, though the process is by no means inevitable. Everyone knows that the insane interpret the world via their own peculiarly consistent logic; how can you tell if your own logic is "peculiar" or not, given that you have only your own logic to judge itself? I don't see any answer. I am just reminded of Gödel's second Theorem, which implies that the only versions of formal number theory which assert their own consistency are inconsistent ...

The other metaphorical analogue to Gödel's Theorem which I find provocative suggests that ultimately, we cannot understand our own minds/brains. All the limitative Theorems of metamathematics and the theory of computation suggest that once the ability to represent your own structure has reached a certain critical point, that is the kiss of death: it guarantees that you can never represent yourself totally. Gödel's Incompleteness Theorem, Church's Undecidability Theorem, Turing's Halting Theorem, Tarski's Truth Theorem—all have the flavor of some ancient fairy tale which warns you that "To seek self-knowledge is to embark on a journey which ... will always be incomplete, cannot be charted on any map, will never halt, cannot be described."

Closely linked with the subject-object dichotomy is the symbol-object dichotomy, which was explored in depth by Ludwig Wittgenstein in the early part of this century. Later the words "use" and "mention" were adopted to make the same distinction. Quine and others have written at length about the connection between signs and what they stand for. But not only philosophers have devoted much thought to this deep and abstract matter. In our century both music and art have gone through crises which reflect a profound concern with this problem. Whereas music and painting, for instance, have traditionally expressed ideas or emotions through a vocabulary of "symbols" (i.e. visual images, chords, rhythms, or whatever), now there is a tendency to explore the capacity of music and art to not express anything—just to be. This means to exist as pure globs of paint, or pure sounds, but in either case drained of all symbolic value.

Provided you do not adopt a totally unreasonable definition of "understanding", I see no Gödelian obstacle in the way of the eventual understanding of our minds. For instance, it seems to me quite reasonable to desire to understand the working principles of brains in general, much the same way as we understand the working principles of car engines in general. It is quite different from trying to understand any single brain in every last detail—let alone trying to do this for one's own brain! I don't see how Gödel's Theorem, even if construed in the sloppiest way, has anything to say about the feasibility of this prospect. I see no reason that Gödel's Theorem imposes any limitations on our ability to formulate and verify the general mechanisms by which thought processes take place in the medium of nerve cells. I see no barrier imposed by Gödel's Theorem to the implementation on computers (or their successors) of types of symbol manipulation that achieve roughly the same results as brains do. It is entirely another question to try and duplicate in a program some particular human's mind—but to produce an intelligent program at all is a more limited goal. Gödel's Theorem doesn't ban our reproducing our own level of intelligence via programs any more than it bans our reproducing our own level of intelligence via transmission of hereditary information in DNA, followed by education.

Does Gödel's Theorem, then, have absolutely nothing to offer us in thinking about our own minds? I think it does, although not in the mystical and limitative way which some people think it ought to. I think that the process of coming to understand Gödel's proof, with its construction involving arbitrary codes, complex isomorphisms, high and low levels of interpretation, and the capacity for self-mirroring, may inject some rich undercurrents and flavors into one's set of images about symbols and symbol processing, which may deepen one's intuition for the relationship between mental structures on different levels.

I would like to bring up the idea of "accidental inexplicability" of intelligence. Here is what that involves. It could be that our brains, unlike car engines, are stubborn and intractable systems which we cannot neatly decompose in any way. At present, we have no idea whether our brains will yield to repeated attempts to cleave them into clean layers, each of which can be explained in terms of lower layers—or whether our brains will foil all our attempts at decomposition. But even if we do fail to understand ourselves, there need not be any Gödelian "twist" behind it; it could be simply an accident of fate that our brains are too weak to understand themselves. Think of the lowly giraffe, for instance, whose brain is obviously far below the level required for self-understanding—yet it is remarkably similar to our own brain. In fact, the brains of giraffes, elephants, baboons—even the brains of tortoises or unknown beings who are far smarter than we are—probably all operate on basically the same set of principles. Giraffes may lie far below the threshold of intelligence necessary to understand how those principles fit together to produce the qualities of mind; humans may lie closer to that threshold—perhaps just barely below it, perhaps even above it. The point is that there may be no fundamental (i.e., Gödelian) reason why those qualities are incomprehensible; they may be completely clear to more intelligent beings.

Barring this pessimistic notion of the accidental inexplicability of the brain, what insights might Gödel's proof offer us about explanations of our minds/brains? Gödel's proof offers the notion that a high-level view of a system may contain explanatory power which simply is absent on the lower levels. Looked at this way, Gödel's proof suggests—though by no means does it prove!—that there could be some high-level way of viewing the mind/brain, involving concepts which do not appear on lower levels, and that this level might have explanatory power that does not exist—not even in principle—on lower levels. It would mean that some facts could be explained on the high level quite easily, but not on lower levels at all. No matter how long and cumbersome a low-level statement were made, it would not explain the phenomena in question. It is the analogue to the fact that, if you make derivation after derivation in TNT, no matter how long and cumbersome you make them, you will never come up with one for G—despite the fact that on a higher level, you can see that G is true.

What might such high-level concepts be? It has been proposed for eons, by various holistically or "soulistically" inclined scientists and humanists, that consciousness is a phenomenon that escapes explanation in terms of brain-components; so here is a candidate, at least. There is also the ever-puzzling notion of free will. So perhaps these qualities could be "emergent" in the sense of requiring explanations which cannot be furnished by the physiology alone. But it is important to realize that if we are being guided by Gödel's proof in making such bold hypotheses, we must carry the analogy through thoroughly. In particular, it is vital to recall that G's nontheoremhood does have an explanation—it is not a total mystery! The explanation hinges on understanding not just one level at a time, but the way in which one level mirrors its metalevel, and the consequences of this mirroring. If our analogy is to hold, then, "emergent" phenomena would become explicable in terms of a relationship between different levels in mental systems.

My belief is that the explanations of "emergent" phenomena in our brains—for instance, ideas, hopes, images, analogies, and finally consciousness and free will—are based on a kind of Strange Loop, an interaction between levels in which the top level reaches back down towards the bottom level and influences it, while at the same time being itself determined by the bottom level. In other words, a self-reinforcing "resonance" between different levels—quite like the Henkin sentence which, by merely asserting its own provability, actually becomes provable. The self comes into being at the moment it has the power to reflect itself.

This should not be taken as an antireductionist position. It just implies that a reductionistic explanation of a mind, in order to be comprehensible, must bring in "soft" concepts such as levels, mappings, and meanings. In principle, I have no doubt that a totally reductionistic but incomprehensible explanation of the brain exists; the problem is how to translate it into a language we ourselves can fathom. Surely we don't want a description in terms of positions and momenta of particles; we want a description which relates neural activity to "signals" (intermediate-level phenomena)—and which relates signals, in turn, to "symbols" and "subsystems", including the presumed-to-exist "self-symbol". At the crux, then, of our understanding ourselves will come an understanding of the Tangled Hierarchy of levels inside our minds.

There is a famous breach between two languages of discourse: the subjective language and the objective language. For instance, the "subjective" sensation of redness, and the "objective" wavelength of red light. To many people, these seem to be forever irreconcilable. I don't think so. No more than the two views of Escher's Drawing Hands are irreconcilable—from "in the system", where the hands draw each other, and from outside, where Escher draws it all. The subjective feeling of redness comes from the vortex of self-perception in the brain; the objective wavelength is how you see things when you step back, outside of the system. Though no one of us will ever be able to step back far enough to see the "big picture", we shouldn't forget that it exists. We should remember that physical law is what makes it all happen—way, way down in neural nooks and crannies which are too remote for us to reach with our high-level introspective probes.

One way to gain some perspective on the free-will question is to replace it by what I believe is an equivalent question, but one which involves less loaded terms. Instead of asking, "Does system X have free will?" we ask, "Does system X make choices?"

Unlike a standard chess program, which does not monitor itself and consequently has no ideas about where its moves come from, imagine a program which does monitor itself and does have ideas about its ideas—but it cannot monitor its own processes in complete detail, and therefore has a sort of intuitive sense of its workings, without full understanding. From this balance between self-knowledge and self-ignorance comes the feeling of free will.

Think, for instance, of a writer who is trying to convey certain ideas which to him are contained in mental images. He isn't quite sure how those images fit together in his mind, and he experiments around, expressing things first one way and then another, and finally settles on some version. But does he know where it all came from? Only in a vague sense. Much of the source, like an iceberg, is deep underwater, unseen—and he knows that. Or think of a music composition program, something we discussed earlier, asking when we would feel comfortable in calling it the composer rather than the tool of a human composer. Probably we would feel comfortable when self-knowledge in terms of symbols exists inside the program, and when the program has this delicate balance between self-knowledge and self-ignorance. It is irrelevant whether the system is running deterministically; what makes us call it a "choice maker" is whether we can identify with a high-level description of the process which takes place when the program runs. On a low (machine language) level, the program looks like any other program; on a high (chunked) level, qualities such as "will", "intuition", "creativity", and "consciousness" can emerge.

The important idea is that this "vortex" of self is responsible for the tangledness, for the Gödelian-ness, of the mental processes. People have said to me on occasion, "This stuff with self-reference and so on is very amusing and enjoyable, but do you really think there is anything serious to it?" I certainly do. I think it will eventually turn out to be at the core of AI, and the focus of all attempts to understand how human minds work. And that is why Gödel is so deeply woven into the fabric of my book.

Escher has given a pictorial parable for Gödel's Incompleteness Theorem. And that is why the strands of Gödel and Escher are so deeply interwoven in my book.

The Musical Offering is a fugue of fugues, a Tangled Hierarchy like those of Escher and Gödel, an intellectual construction which reminds me, in ways I cannot express, of the beautiful many-voiced fugue of the human mind. And that is why in my book the three strands of Gödel, Escher, and Bach are woven into an Eternal Golden Braid.