Before answering any question, Anna pauses for a tangible moment, a half-second or so; I’m fairly sure that this is a learned behaviour, an attempt to vet each statement to make sure it’s something she thinks, rather than simply something she’s saying.

There is a thing Yudkowsky came up with, called Pascal’s Mugging, related to the famous wager which says that if you simply multiply risk by reward, you’re vulnerable to absurd situations. In Bostrom’s rather whimsical version of it, the example is that a mugger comes along and demands Pascal’s wallet. Pascal points out: ‘You have no weapon.’ ‘Oh good point,’ says the mugger. ‘But how about if you give me the wallet, I come back tomorrow and give you 10 times the value of the money in it?’ ‘Well’, says Pascal, ‘that’s not a very good bet, is it. It’s hugely likely that you’ll just not come back.’ But the mugger then says: ‘Actually, I’m a wizard from the seventh dimension. I can give you any amount of money you like. I can give you, in fact, any amount of happiness you like. Let’s say that the money in your wallet could buy you one happy day. [Assume for the sake of argument that money can buy happiness.] And let’s say that you think there’s only a 1 in 10100 chance that I’m telling the truth. Well, in that case, I’ll offer you 101,000 happy days.’ By a utilitarian calculus – the idea that you should multiply the chance of something happening by the reward it would bring if it does, exactly the sort of reasoning that Bostrom uses to think about the cosmic endowment, or for that matter that investors and gamblers use to determine where to put their money – this is a good bet. If Pascal took it, on average, he’d expect a 10990-fold return on his investment. But it is, also, pretty obviously ridiculous. The wizard-mugger can just keep upping the numbers he offers until it becomes a good bet. So it’s OK to be wary; you should be, when someone comes up and mouths a lot of maths and numbers and technical talk that you can’t follow but which they say supports their point. The Rationalists have a term for that, in fact: ‘getting Eulered’, blinded by numbers. But that doesn’t mean you should simply dismiss it. If you can’t follow the maths, you should be wary, but you should try to follow the maths. One of the founding principles of the Rationalist movement is that, as Scott Alexander puts it, ‘when math tells you something weird, you at least consider trusting the math. If you’re allowed to just add on as many zeroes as it takes to justify your original intuition, you miss out on the entire movement.’ A weird-seeming answer is a warning flag, rather than a stop sign: a thing to investigate rather than reject.

‘A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome,’ say Russell and Norvig. The ‘best outcome’, of course, depends on what goals the agent has – my goals, and therefore my ‘best outcome’, are likely to be different in some respects from your goals; your goals may be selfishly attaining material riches, while mine are the noble pursuit of knowledge and the betterment of mankind, etc. But an agent is rational insofar as it is good at achieving whatever goals it has. This has the advantage, say Russell and Norvig, of being ‘mathematically well defined and completely general’. And again, importantly, we don’t care at all about how a given agent achieves rationality. An AI that carefully mimics the human brain, to the point of having simulations of individual neurons, could be rational; an AI that runs entirely along the lines of a Turing machine, or Charles Babbage’s Difference Engine, metal gears and all, could be rational too. The mathematically defined concept of ‘rationality’ does not care what engine is used to run it. And, again, it doesn’t care whether or not your AI is conscious, or has emotions, or knows what love is. It’s purely a question of whether it achieves its goals, whatever they are.

Bostrom phrases ‘the orthogonality thesis’ like this: ‘Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.’ What he means is: you can plot a graph, with ‘intelligence’ up the Y axis and ‘goals’ along the X. Any point on the graph, with a couple of minor constraints (you couldn’t have a really dumb computer with really complex goals that it couldn’t fit in its memory, for instance), represents a possible AI. Even the cleverest AI could have what seem to us spectacularly stupid goals.

For Yudkowsky, intelligence/rationality is about matching your mental model of the world to the real world as closely as possible, and about making decisions that achieve what you want them to as often as possible. Both of these processes, he says, can be described using a simple equation called ‘Bayes’ theorem’. First, we should discuss what Yudkowsky means by ‘rational’. There are two fundamental ideas underpinning ‘rationality’ as defined by the Rationalists. They are ‘epistemic rationality’ and ‘instrumental rationality’. ‘Epistemic rationality’ is achieving true beliefs. Or, as Yudkowsky puts it, ‘systematically improving the accuracy of your beliefs’. The Rationalists have a phrase for this: ‘The map is not the territory.’ Your mind contains thousands of models, which it uses to predict reality. For instance, I have a working model of gravity and air resistance and things which allows me (sometimes) to catch a ball that is thrown to me. Even more prosaically, I have a model which says, ‘The lamp is over there’ and ‘The door is behind me’ and ‘The window is in front of me.’ The degree to which I have an accurate model, the degree to which I can walk to where I think the door is and actually find a door there, is the degree to which my model corresponds with the world, or my ‘map’ corresponds with the ‘territory’. ‘This correspondence between belief and reality is commonly called “truth”,’ says Yudkowsky, ‘and I’m happy to call it that.’

Instrumental rationality, by contrast, is about your actions. ‘Rationalists’, says Yudkowsky, ‘should win.’ The idea is the same as the definition of ‘behaving rationally’ in the textbook Artificial Intelligence: A Modern Approach. It is choosing that course of action which is most likely, given what you know now, to achieve the goal you want to achieve. It doesn’t mean, he says, selfish domination, or money, or anything specific. It means ‘steering reality – sending the future where you want it to go’. That could mean to your own selfish ends, or it could mean towards preventing climate change, or turning the universe into paperclips. It is about successfully doing what you wanted to do.

Instrumental rationality doesn’t, necessarily, mean behaving in a ‘rational’ way, as defined by Hollywood and especially Mr Spock. Yudkowsky really doesn’t like Spock. ‘Consider Mr Spock of Star Trek, a naive archetype of rationality,’ he grumbles at one point. ‘Spock’s emotional state is always set to “calm”, even when wildly inappropriate.’ If you are about to be blown up by a Klingon torpedo, then being afraid might be rational. Worse than that, Spock’s ‘rational’ predictions, given in spuriously precise percentages, are usually wrong. ‘He often gives many significant digits for probabilities that are grossly uncalibrated,’ says Yudkowsky. ‘E.g.: “Captain, if you steer the Enterprise directly into that black hole, our probability of surviving is only 2.234 per cent.” Yet nine times out of ten the Enterprise is not destroyed. What kind of tragic fool gives four significant digits for a figure that is off by two orders of magnitude?’ Instead it means winning.

That’s the Rationalist rationality at its most basic, then: trying to believe things that are true, and trying to take decisions that lead to the outcomes you want.

For Yudkowsky, the heart of rational behaviour is the simple mathematical equation known as Bayes’ theorem. When he talks about rationality, he is talking about Bayes; the project of improving human rationality is a project of making humans better Bayesians. The theorem is (he says, and decision theory agrees) absolutely central to what good decision-making involves. When you have evidence for something, that evidence allows you to shift your beliefs only as far – no more, no less – as the distance dictated by Bayes.

Bayes’ theorem is extremely useful from a philosophical point of view. I studied philosophy at university, and there were endless arguments about the ‘problem of induction’. The idea was that you could see a million white swans, but you would never be able to prove the statement ‘all swans are white’, because it would take seeing just one swan which was black – which Western explorers did when they first reached Australia – to disprove it. No amount of ‘inductive reasoning’ – coming to conclusions from evidence – could ever prove anything. But Bayesian thinking lets you sidestep this altogether. You simply learn to think probabilistically. Having never seen a swan, you might assign a prior probability to the hypothesis ‘all swans are white’ of, say, 1 per cent. (All swans could be green, for all you know.) You see your first swan, you update your prior probability in the light of new evidence: you might think that it’s now 15 per cent likely that all swans are white. (You’ve only seen one swan. They could come in all sorts of colours.) That is now your new prior. But after wandering around Renaissance Europe for 40 years, only ever seeing white swans, and constantly updating your priors, you are now much more confident in the statement. As a good Bayesian, you’re never certain, but you’ve seen thousands of swans, each one adding a small dollop of evidence to support your hypothesis, so you push your confidence up to a very solid 95 per cent. Then you get on a boat to Van Diemen’s Land, and you see a black swan. Your confidence immediately plummets to 0.01 per cent. The problem of induction isn’t a problem any more, as long as you’re willing to think in terms of likelihoods and probabilities, rather than certainties. You’re never certain – someone might be painting all those black swans black, or you might be hallucinating – but the more swans you see, the more you can update your priors and increase your confidence. The Rationalists think of all knowledge in these terms: how confident you can be in your beliefs, how much ‘probability mass’ you should assign to some proposition, and how much you can ‘update’ your beliefs in the light of new evidence.

Essentially, he (and the Rationalists) are thoroughgoing utilitarians. Do the thing that (you reasonably expect will) kill the fewest people/make the most people happy/cause the least pain. You can think about it in more detail than that, they would say; but if your thinking pushes you away from doing that, then your thinking has probably gone wrong.

The trolley problem is intended to divide people down deontological or utilitarian lines: if you see a railway trolley heading towards five workers on a track, and you can pull a switch so it goes the other way, but there is one person working on that line, should you do it? A utilitarian should, in theory, say ‘yes’, but a deontologist (someone who follows strict moral rules rather than considering consequences) should say that you never actively kill someone, so you shouldn’t pull the switch even though you would save lives. (This is an enormous oversimplification of both deontology and utilitarianism.) Yudkowsky approaches the trolley problem like this. Sure, he says, it might be the case that you think you can save five lives by killing one. (Or that you can help the poor by robbing a bank, or that you can improve society by staging a military coup and taking over, or any one of 100 versions of ‘I can justify Bad Thing X by promising Good Consequence Y’.) But knowing humans, it is very unlikely that you are right – or that you are likely enough to be right that, if you did it a million times, you’d overall prevent more harm than you caused. In the trolley problem, the philosopher stipulates that you know with certainty that your action will save five and kill one and there’s no other way around it. But in reality, your inadequate human brain can’t ever be certain enough that that’s the case. You’ve evolved, as a human, a whole range of systems for creating moral-sounding reasons for doing self-interested things. You are more likely to do good, overall, by implementing the rule ‘Never kill anyone’ than by trying to work out the maths of utilitarianism on the fly in sudden, stressful situations. And that ends up creating odd-sounding meta-rules, such as ‘For the good of the tribe, do not murder even for the good of the tribe.’ It is more likely that the thing you think of as being for the good of the tribe is in fact for the good of you.

‘People have trouble applying the notion of a good or bad consequence to all the actual consequences that are good or bad,’ he said. ‘Instead they see a small subset of consequences, the immediate local consequences, and think those are the “consequences”.’ For that reason, ‘most people should not immediately try to be “utilitarians”... They are better off continuing to debate which rules are good or bad and then following those rules.’ For utilitarian reasons, don’t try to be a utilitarian! Again, it would amaze me if an internet guy in California had solved all the problems of moral philosophy. But I do find this approach refreshingly direct. There really is a moral law, of improving the world for the greatest number of people. It really does lead to some weird outcomes, like the torture/dust specks thing. However, it is a complex and difficult law to implement and we are usually best off implementing simpler, local laws, such as ‘Do the thing that kills the fewest people.’ You can contrive thought-experiment situations with trolleys or torture that end up forcing you into difficult situations, but in real life, ‘Do the thing that kills the fewest people’ is a solid position to take, and anything that steers you to a different answer should raise lots of red flags. This is the basic moral position for the Rationalists: ‘When human lives are at stake, we have a duty to maximise, not satisfice; and this duty has the same strength as the original duty to save lives. Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.’

Biases

A cognitive bias ‘is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our beliefs so that they less accurately represent the facts, and they skew our decision-making so that it less reliably achieves our goals.’

There are various psychological reasons behind the individual biases, but the fundamental one appears to be that they worked for our ancestors. They were shortcuts. We didn’t need to work out the value of 20,000 things compared to 2,000 things when we were tribal hunter-gatherers; we didn’t need to work out probabilities. We could get pretty good estimates of values and risks from simple rules of thumb, or ‘heuristics’. But now they often misfire.

The availability heuristic: When we are asked how likely something is, we could go and add up all the examples of it, divide this figure by the number of times it could possibly have happened, and get the answer. But that’s difficult and takes a long time. What we tend to do, in reality, is to judge how likely something is by how easily we can think of an example; and how easily we can think of one is only loosely related to how often it happens. More dramatic things, which get disproportionate amounts of coverage in the media, are easier to remember. We can easily think of examples of terrorism, because every single one around the world gets reported, with dramatic images of smoke and fire and blood. We can’t easily think of examples of drowning in the bath, because even though they happen far more frequently they don’t make the news, and even when they do they’re unspectacular.

The conjunction fallacy: What’s more likely: that the climate will stop warming, or that a new technology will be developed which allows fuel to be economically harvested from atmospheric CO2, and the ensuing reduction in greenhouse-gas levels stops the climate from warming? The ‘conjunction fallacy’ is that adding details makes a story seem more plausible, even though they must – by the workings of mathematics – make it less probable. We see the extra details as corroborative, says Yudkowsky (and Kahneman, and modern psychological science). But we should see them as burdensome. They don’t make a story more likely, they make it less. People who want to avoid this ‘need to notice the word “and”,’ says Yudkowsky. ‘They would need to be wary of it – not just wary, but leap back from it... They would need to notice the conjunction of two entire details, and be shocked by the audacity of anyone asking them to endorse such an insanely complicated prediction. And they would need to penalise the probability substantially.’ Again, humans don’t do this; a perfect Bayesian AI would.

The planning fallacy: How long will it take you to do something? Something big, some project that might require a few weeks or months? A good rule of thumb: however long you think it will take, it’ll probably take longer. There is a well-documented way around the planning fallacy, though. Don’t just look at the specifics of what your project involves – look at how long other, similar projects have taken in the past. This is called taking the ‘outside view’ instead of the ‘inside view’. The ‘inside view’ is what you can see when you’re looking at it from your own perspective. ‘So there is a fairly reliable way to fix the planning fallacy,’ says Yudkowsky. ‘Just ask how long similar projects have taken in the past, without considering any of the special properties of this project. Better yet, ask an experienced outsider how long similar projects have taken. You’ll get back an answer that sounds hideously long, and clearly reflects no understanding of the special reasons why this particular task will take less time. This answer is true. Deal with it.’

Scope insensitivity: How much – in US dollars, or pounds sterling, or whatever – is a human life worth? And how much are a million human lives worth? Whatever answer you give to the first question, the answer to the second – surely – should be a million times greater. That at least is the Rationalist response. This may seem obvious, but in fact it is not. There is plenty of evidence to show that we are extremely inconsistent in our approaches to these things. Exactly what’s going on in our brains we don’t know, obviously, but it appears that we make these judgements according to our emotional response, rather than any kind of numbers-based assessment.

Motivated scepticism, motivated stopping and motivated continuation: Jonathan Haidt, the social psychologist, says in his (excellent) book The Righteous Mind: Why Good People Are Divided by Politics and Religion that when we are presented with evidence for or against a hypothesis, we ask ourselves one of two questions. When we want to believe something, ‘we ask ourselves, “Can I believe it?” Then... we search for supporting evidence, and if we find even a single piece of pseudo-evidence, we can stop thinking. We now have permission to believe.’ But when we don’t want to believe something, ‘we ask ourselves, “Must I believe it?” Then we search for contrary evidence, and if we find a single reason to doubt the claim, we can dismiss it.’

The technical terms for the ‘can I believe it/must I believe it’ phenomena are ‘motivated credulity’ and ‘motivated scepticism’. Yudkowsky: ‘A motivated sceptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion.’ Yudkowsky adds another layer to this, which is the idea of motivated stopping and motivated continuation. When we’re looking for something in real life, we aren’t usually given a set of things to choose from: ‘You have to gather evidence, which may be costly, and at some point decide that you have enough evidence to stop and choose. When you’re buying a house, you don’t get exactly 10 houses to choose from... You look at one house, and another, and compare them to each other and at some point you decide that you’ve seen enough houses, and choose.’

‘When we have a hidden motive for choosing the “best” current option, we have a hidden motive to stop, and choose, and reject consideration of any more options,’ says Yudkowsky. ‘When we have a hidden motive to reject the current best option, we have a hidden motive to suspend judgement pending additional evidence, to generate more options – to find something, anything, to do instead of coming to a conclusion.’

The most important one: The most important bias to be aware of is this, which is a sort of collection of several: knowing about biases can make you more biased. Various biases can actually mean that even as you get more information, you become more wrong. Confirmation bias and disconfirmation bias, and related phenomena, for instance. New information comes in, but your brilliant mind finds brilliant ways in which to ignore the stuff it doesn’t like and promote the stuff it does. There’s a particularly pernicious one, the ‘sophistication effect’: ‘Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to these biases.’ So new information like ‘We are all biased and the things we believe are frequently wrong’ can easily become ‘These arguments that are being deployed against me are flawed, and I can point out why because I have this in-depth knowledge of human biases.’ Yudkowsky calls this a ‘fully general counter-argument’. Anybody with a partisan axe to grind can deploy ‘confirmation bias’ to undermine an argument they don’t like. Most of the things we call ‘human biases’ are extremely convenient labels to attach to opinions with which we disagree. But the key is to accuse your own of them. You are biased. (I am biased.) You are probably systematically overconfident in your beliefs.

Raising the Sanity Waterline

Making beliefs pay rent: Another key way of checking your own beliefs is to think about what they actually imply. Yudkowsky calls this ‘Making beliefs pay rent in anticipated experiences.’ For instance: if a tree falls in the forest, does it make a sound? Answer that question in your mind before you go any further. If you thought ‘no’, is that because, to you, ‘sound’ means the sensation, the qualia, of someone hearing something? And if you thought ‘yes’, is that because ‘sound’ means the pressure waves in air that are made when something loud happens? This is one of the longest-running arguments in philosophical history, to the point that it’s a cliché of philosophy alongside angels dancing on the head of a pin. But, assuming that you agree that the physical world still exists when we are not looking at it (which some philosophers dispute, but I am content to ignore them), then – what are people actually arguing about? Yudkowsky imagines an argument between two people, Albert and Barry:

Albert: ‘What do you mean, there’s no sound? The tree’s roots snap, the trunk comes crashing down and hits the ground. This generates vibrations that travel through the ground and the air. That’s where the energy of the fall goes, into heat and sound. Are you saying that if people leave the forest, the tree violates Conservation of Energy?’
Barry: ‘But no one hears anything. If there are no humans in the forest, or, for the sake of argument, anything else with a complex nervous system capable of “hearing”, then no one hears a sound.’

But, points out Yudkowsky – who imagines the argument spiralling out of control somewhat – Albert and Barry actually agree on everything that is happening. They both think that the tree hits the ground and sends waves of energy through the forest. They both agree that no auditory sensations are being experienced. All they disagree about is whether or not that combination of things should be called a ‘sound’ or not. If you had two words – Yudkowsky suggests ‘albergle’ for acoustic vibrations, ‘bargulum’ for auditory experiences – then the argument would disappear; they’d just say ‘OK, it makes an albergle but not a bargulum’.

A surprising number of arguments seem to fall into this form. (About 40 per cent of those on the contemporary British internet seem to revolve around whether or not Person A or Group B is Marxist/socialist/Nazi/alt-right/misogynistic/racist/transphobic/a TERF etc., with people on each side marshalling reasons for and against their inclusion in one definition or another.) But these debates are sterile, for Yudkowsky and the Rationalists, because they don’t constrain your expectations. If your model can explain every outcome, then it can’t explain any outcome. If I argue that we should define ‘sound’ as ‘acoustic vibrations’ rather than ‘auditory experiences’, it won’t change what I expect to find when I walk into the forest to see where the tree has fallen. If I argue that we should define Jeremy Corbyn as a ‘Marxist’ rather than a ‘socialist’, it won’t change what I expect him to do if his Labour Party is elected to power. If I say, ‘I believe that the tree’s trunk broke, rather than that the roots came out of the ground,’ that is a belief that constrains my experiences; if I turn up and see that the roots are out, then I know that my belief was wrong. ‘I believe that Jeremy Corbyn will renationalise the British railway system within a year of coming to power’ constrains my experiences; if he does not, then I know I was wrong. But ‘Jeremy Corbyn is a Marxist’ does not constrain my beliefs and cannot be used to predict anything: if Corbyn does not nationalise the railways, he could still be a Marxist, and vice versa.

Noticing confusion: When something happens that your beliefs don’t anticipate, you should be confused. And you should pay attention to your confusion, because either your belief model is wrong, or something else is going on that you’re not aware of. I should have paid more attention to that sensation of still feels a little forced. It’s one of the most important feelings a truth can have, a part of your strength as a Rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading: Either Your Model Is False Or This Story Is Wrong.’ If you’re trying to become a more rational being – a better Rationalist – then you need to listen to those little moments when something doesn’t quite seem to add up.

Dark Sides

The online group known as the ‘Neoreactionaries’, which is a sort of strange medievalist subset of the alt-right, grew out of the Rationalist movement to some extent. They even left LessWrong and founded their own website, named (spot the reference) ‘More Right’. Mencius Moldbug, the founder of Neoreaction, wrote a few blogs on Robin Hanson’s Overcoming Bias before LessWrong split from it. Michael Anissimov, another prominent Neoreactionary, was until 2013 MIRI’s media director. The pseudonymous ‘Konkvistador’ is a regular Slate Star Codex commenter.

According to Neoreactionaries, the world has been moving steadily to the left for several hundred years, and that it has correspondingly become less safe, less happy and less clever, and that it is impossible to speak your mind freely unless you toe certain leftist lines. They also believe that some ethnic minorities’ poorer life outcomes – in education, income, crime, mental health, etc. – are due to biological and/or cultural factors within those minorities; that women are happier in more traditional, ‘sexist’ societies; that immigration from some developing-world countries actively worsens America by bringing in people with different, and worse, values. And, most notably, they think that democracy should be replaced by an omnipotent and unelected king.

The Rationalists have a particular problem which is that their whole thing is taking opposing arguments seriously – what Alexander calls the ‘principle of charity’. It is part of SSC’s ethos that, ‘if you don’t understand how someone could possibly believe something as stupid as they do’, then you should be prepared to find that that’s your failing, rather than theirs. And what that means is that if you’re the sort of person who wants to go and talk about ‘race science’, for example, you’ll find that going to a Rationalist website and doing so means that you aren’t immediately blocked. Instead, you find people will talk to you seriously and engage with you. And this is a noble and brilliant thing, in many ways! If you want to shout about how terrible The Enemy is, so that you get cheered on by Your Side, then you’ve got the whole internet in which to do that. But the Rationalist community is where you can speak, calmly and collaboratively, with people with whom you profoundly disagree, and try to change minds (and admit to the possibility that you will have your own mind changed).

Doing Good Better

It's not simply that giving money to aid charities is a nice thing to do, an over-and-above-the-call-of-duty moral bonus – ‘supererogatory’, in the language of moral philosophy – it’s that it is a duty. We should all, in wealthy, developed countries, give some non-negligible percentage of our earnings, either as charitable donations or as foreign aid via tax, to improving the lot of people in developing countries.

The idea of the Effective Altruism movement is that we are not, generally, dealing in subtle distinctions: if you want to do the most good with your money, rather than just purchase warm feelings, then some charities are very obviously better than others.

There are three key elements that make a cause worth donating to, according to the tenets of Effective Altruism. One is its importance: the scale of the problem, how much better the world would be if the problem were solved. A second is tractability: how easy it is to solve those problems. And a third is neglectedness: if lots of people are already working on the problem, then the amount of good you can do on the margin is less. So malaria is an excellent target, because it has a huge impact (importance), is easily and cheaply prevented (with bed-nets), and yet receives far less global spend than diseases like cancer, which disproportionately affect those rich countries where people tend to live long enough to get them.

There’s a concept in the Rationalsphere called ‘weirdness points’. It’s the idea that society will let you be only so weird before it stops taking you seriously: you only have a certain number of weirdness points to spend, and so you should spend them on things that you really care about. That’s why, says Mike Story, Rationalists don’t talk all that much about their polyamory – ‘more than just not evangelise, they keep quiet about it. Scott Alexander is pretty open about it, but generally they think if we seem normal it’s better for our ideas.’ If you spend your weirdness points on polyamory, you don’t have them left to spend on Effective Altruism or the importance of Bayes’ theorem.

The Double Crux

The standard ‘double crux’ is a method Rationalists use for examining why two people disagree. ‘It’s about how to figure out what cruxes a disagreement you’re having with someone rest on. The crux of the argument is the thing that, if you knock it over, their conclusion falls down, and they have a different conclusion.’ The example given on LessWrong is an argument between two people about school uniforms. Person A thinks schoolchildren should wear uniforms; person B thinks they shouldn’t. To find the crux, you look at what those beliefs entail, what the more specific implications of them are. So person A might think that school uniforms reduce bullying, by making it less obvious which children are rich and which are poor; person B might think that’s ridiculous. But if you could show that school uniforms do reduce bullying, by some given amount, then person B would change her mind on the uniforms question; likewise, if you could show that they don’t, then person A would change his mind. The technique involves slowly bringing the conversation away from top-level, shouty arguments and towards detailed, specific disagreements.