Capitalism works only when institutions are forced to absorb the consequences of the risks that they take on. When banks can pocket the upside while spreading the cost of their failures, failure is almost certain. If they are serious about learning from the 2007–2009 crisis, policy makers need to restrain financial supermarkets with confused and overlapping objectives, encouraging focused boutiques that live or die according to the soundness of their risk management. They need to shift capital out of institutions underwritten by taxpayers and into ones that stand on their own feet. They need to shrink institutions that are too big to fail and favor ones that are small enough to go under.

The hedged fund does better in a bull market despite the lesser risk it has assumed; and the hedged fund does better in a bear market because of the lesser risk it has assumed. Of course, the calculations work only if the investors pick good stocks; a poor stock picker could have his incompetence magnified under Jones's arrangement. Still, given the advantages of the hedged format, the question was why other fund managers failed to emulate it. The answer began with short selling, which, as Jones observed in his report to investors, was "a little known procedure that scares away users for no good reason." A stigma had attached to short selling ever since the crash and was to survive years into the future; amid the panic of 2008, regulators slapped restrictions on the practice. But as Jones patiently explained, the successful short seller performs a socially useful contrarian function: By selling stocks that rise higher than seems justified, he can dampen bubbles as they emerge; by repurchasing the same stocks later as they fall, he can provide a soft landing. Far from fueling wild speculation, short sellers could moderate the market's gyrations. It was a point that hedge-fund managers were to make repeatedly in future years. The stigma nonetheless persisted.

Jones's next innovation was to distinguish between the money that his fund made through stock picking and the money that it made through its exposure to the market. Years later, this distinction became commonplace: Investors called skill-driven stock-picking returns "alpha" and passive market exposure "beta."

The art of investment is not merely to maximize return but to maximize risk-adjusted return, and the amount of risk that an investor takes depends not just on the stocks he owns but on the correlations among them.

The separation theorem holds that an investor's choice of stocks should be separate from the question of his risk appetite. Most investment advisers in the 1950s assumed that certain types of stocks suited certain types of investor: A widow should not own a go-go stock such as Xerox, whereas a successful business executive should have no interest in a stodgy utility such as AT&T.; Tobin's insight was to see why this was wrong: An investor's choice of stocks could be separated from the amount of risk he wanted. If an investor was risk averse, he should buy the best stocks available but commit only part of his savings. If an investor was risk hungry, he should buy exactly the same stocks but borrow money to buy more of them.

In the 1950s and 1960s, the combination of Darwinist individualism and top-down risk control was almost unique to Jones, and this gave him a powerful advantage. The market may be efficient, in the sense that information is reflected in prices to the extent that existing institutional arrangements allow. But Jones blew up those institutional arrangements, scrapping staid committee meetings and paying people to perform. Thus did he create the edge that brought in serious money.

Block traders had figured out a new approach: They weren't engaged in the overcrowded business of analyzing company data and picking the stocks that would do well; instead, they aimed to make money by supplying something that other investors needed--liquidity.

Innovation is often ascribed to big theories fomented in universities and research parks: Thus Stanford's engineering school stands at the center of Silicon Valley's creativity, and the National Institutes of Health underpin innovation in the pharmaceutical industry. But the truth is that innovation frequently depends less on grand academic breakthroughs than on humble trial and error--on a willingness to go with what works, and never mind the theory that may underlie it. Even in finance, a field in which research findings can be translated directly into business plans, trial and error turns out to be key.

The LSE luminary who inspired Soros the most was Karl Popper. Popper's central contention was that human beings cannot know the truth; the best they can do is to grope at it through trial and error. Soros had melded Karl Popper's ideas with his own knowledge of finance, arriving at a synthesis that he called "reflexivity." As Popper's writings suggested, the details of a listed company were too complex for the human mind to understand, so investors relied on guesses and shortcuts that approximated reality. But Soros was also conscious that those shortcuts had the power to change reality as well, since bullish guesses would drive a stock price up, allowing the company to raise capital cheaply and boosting its performance. Because of this feedback loop, certainty was doubly elusive: To begin with, people are incapable of perceiving reality clearly; but on top of that, reality itself is affected by these unclear perceptions, which themselves shift constantly. Soros had arrived at a conclusion that was at odds with the efficient-market view. Academic finance assumes, as a starting point, that rational investors can arrive at an objective valuation of a stock and that when all information is priced in, the market can be said to have attained an efficient equilibrium. To a disciple of Popper, this premise ignored the most elementary limits to cognition.

Starting in August 1985, Soros kept a diary of his investment thinking, hoping that the discipline of recording his thoughts would sharpen his judgments. The resulting "real-time experiment" is dense, repetitive, and filled with complex ruminations about scenarios that never in the end materialize. But because it is free of the biases that afflict retrospective explanations of success, it is a true portrait of the speculator at work. Soros saw no point in knowing everything about a few stocks in the hope of anticipating small moves; the game was to know a little about a lot of things, so that you could spot the places where the big wave might be coming.

The truth is that markets are at least somewhat efficient, so most information is already in the price; the art of speculation is to develop one insight that others have overlooked and then trade big on that small advantage.

The triple attack on efficient-market theory--statistical, institutional, and psychological--was in some ways a vindication for the hedge-fund industry. It helped to explain how Michael Steinhardt's block trading or Helmut Weymar's commodity speculators could have done so well, and it showed that market practitioners had often been ahead of academic theorists.

When investment managers are viewed merely as sets of performance numbers, the handful of success stories can be dismissed as products of chance--the equivalent of ten-heads-in-a-row coin flippers. But if investment managers are understood as belonging to distinct intellectual "villages" or styles, their success may be concentrated in a way that is not random. Julian Hart Robertson: tall, confident, and athletic of build, he was a guy's guy, a jock's jock, and he hired in his own image. To thrive at Robertson's Tiger Management, you almost needed the physique; otherwise you would be hard-pressed to survive the Tiger retreats. A Tiger should manage the portfolio aggressively, removing good companies to make room for better ones; he should avoid risking more than 5 percent of capital on one bet; and he should keep swinging through bad times until his luck returned to him. The simple truth was that the big jock and his lieutenants--a handful at the outset, perhaps a dozen later on--just analyzed companies, currencies, and commodities and bet on their prospects. This was exactly what the efficient marketers believed to be impossible.

Tiger's defiance of efficient-market presumptions cannot be explained entirely by short selling. In most years Robertson would have beaten the market even without the profits from his shorts, suggesting that he had an edge precisely where the theory said no edge was possible: in traditional stock buying. Moreover, Robertson's record, like Buffett's record, was not an isolated phenomenon. Just as Buffett was part of an investment "village"--the cluster of superstars who had been schooled in Ben Graham's value-investing style--so Robertson was a village headman. On one count in 2008, thirty-six former Tiger employees had set up "Tiger cub" funds, which collectively managed $100 billion; and Robertson had seeded a further twenty-nine funds after restructuring his firm in 2000.20 These Robertson protégés did well: A test of Tiger cubs' performance, presented in the first appendix to this book, shows that they beat not only the market but also other hedge funds. Moreover, the test covers the years 2000 to 2008, a period in which the profusion of long/short equity hedge funds had long since ended whatever easy profits might have existed in short selling during the 1980s. If Robertson's achievement had stood by itself, it might have been possible to dismiss him as a lucky coin flipper. But the success of Tiger's numerous offshoots puts paid to that thesis. Whatever the source of Robertson's investment edge, it was profitable--and transferable.

Value investors generally buy stocks using little or no leverage, and they hold them for the long term; if the investment moves against them, they typically buy more, because a stock that was a bargain at $25 is even more of a bargain at $20. But macro investors take leveraged positions, which make such trend bucking impossibly risky; they have to be ready to jump out of the market if a bet moves against them. Similarly, value investors pride themselves on rock-solid convictions. They have torn apart a company balance sheet and figured out what it is worth; they know they have found value. Macro investors have no method of generating comparable conviction. There is no reliable way to determine the objective "value" of a currency.

Because markets are not perfectly efficient, hedge funds and other creatures of the markets raise difficult issues: They are part of an unstable game that can wreak havoc on the world economy. But by the same token, the inefficiency of the markets allowed hedge funds to do well. Investors would line up to get into them.

Meriwether and his partners scoured the world for this sort of opportunity. They spotted probable convergences in all kinds of settings: between different bonds of the same maturity, between a bond and the futures contract that was based on it, between Treasury and mortgage-backed bonds or between bonds in different currencies. The common theme was that market anomalies occur when the behavior of investors is distorted--whether by tax rules, government regulation, or the idiosyncratic needs of large financial institutions. By being the flexible player with the freedom to mirror the quirks of the inflexible ones, Long-Term provided liquidity to the markets. The real lesson of LTCM's failure was not that its approach to risk was too simple. It was that all attempts to be precise about risk are unavoidably brittle. Long-Term had been too leveraged. It had overlooked the danger that its trades could implode spectacularly if other arbitrageurs were forced to dump copycat positions suddenly; it had misjudged the precision with which financial risk can be measured.

"The market can stay irrational longer than you can stay solvent," Keynes famously declared. Being early and right is the same as being wrong, as investors have repeatedly discovered.

In ordinary liquid markets, prices are fairly efficient and second-guessing them is hard. In illiquid markets, by contrast, there are bargains aplenty--but mistakes can be extremely costly. The biggest danger for buyers of illiquid assets is that, in a crisis, these assets will collapse the hardest. In moments of panic, investors crave securities that can be easily sold, and the rest are shunned ruthlessly.

It's clear that the way Brown and Mercer approached programming was fundamentally different from the way other hedge-fund programmers thought about it. At Tudor, for example, Sushil Wadhwani trained a machine to approach markets in a manner that made sense for human traders. By contrast, Brown and Mercer trained themselves to approach problems in a manner that made sense for a computer. At D. E. Shaw, the approach was frequently to start with theories about the market and to test them against the data. By contrast, Brown and Mercer fed the data into the computer first and let it come up with the answers. D. E. Shaw's approach recalls the programmers who taught computers French grammar. The Brown-Mercer approach resembles that of code crackers, who don't have the option of starting with a grammar book. Presented with apparently random data and no further clues, they sift it repeatedly for patterns, exploiting the power of computers to hunt for ghosts that to the human eye would be invisible.

Hedge funds are paranoid outfits, constantly in fear that margin calls from brokers or redemptions from clients could put them out of business. They live and die by their investment returns, so they focus on them obsessively. They are generally run by a charismatic founder, not by a committee of executives: If they see a threat to their portfolio, they can flip their positions aggressively. Banks are complacent by comparison. They have multiple streams of revenue and their funding seems secure: Deposit-taking banks have sticky capital that enjoys a government guarantee, while investment banks felt (wrongly, as it turned out) that their access to funding from the equity and bond markets made them all but impregnable.

Jones was looking at an asymmetrical bet, and he understood this intuitively. A leveraged financial system in a credit crisis is like a high-wire artist in a storm. The wire is going to wobble, and the artist may lose his balance and tumble a long way. But he is definitely not going to levitate upward. The Paulson team was walking into a version of the trap that had snared the Bank of England in 1992: It looked at the odds of various outcomes in the way that policy makers do, but it failed to ask the trader's question--what is the payout in each instance? From a policy maker's perspective, Lehman's failure might engender chaos or it might not; if you thought there was a fifty-fifty chance of calm, you might choose to take the risk, especially if you were anxious to teach banks a lesson in responsibility. But from a trader's perspective, this calculation was naive; a fifty-fifty chance of calm meant that chaos was virtually certain in practice. Hedge funds from London to Wall Street would conduct a thought experiment: In the calm world, markets would be flat; in the chaotic world, markets would crater; if traders shorted everything in sight, they would lose nothing in the first instance but make a killing in the second one. Faced with this asymmetrical payout, every rational hedge fund would bet aggressively on a collapse. And because they were going to make those bets, collapse would be inevitable. Jones adds, "From a trading perspective, fear is a much stronger emotion than greed, which is why things go down twice as fast as they go up. And that's also just the law of nature. How long does it take for a tree to grow, and how quickly can you burn it down? It's much easier to destroy things than to build them up. So from a trading perspective, the short side is always a beautiful place to be because quite often when you get paid, you get paid in vertical no-pain type of moves."

Hedge funds should not be judged against some benchmark of perfection. The case for believing in the industry is not that it is populated with saints but that its incentives and culture are ultimately less flawed than those of other financial companies. The heart of the case for hedge funds can be summed up in a single phrase. Whereas large parts of the financial system have proved too big to fail, hedge funds are generally small enough to fail. When they blow up, they cost taxpayers nothing.

Put simply, government actions have decreased the cost of risk for too-big-to-fail players; the result will be more risk taking. The vicious cycle will go on until governments are bankrupt.

If financial behemoths cannot be left to go under, and if regulation is both essential and fallible, policy makers should pay more attention to a third option. They should make a concerted effort to drive financial risk into institutions that impose fewer costs on taxpayers. That means encouraging the proliferation of firms that are not too big to fail, so reducing the share of risk taking in the financial system that must be backstopped by the government. It also means favoring institutions where the incentives to control risk are relatively strong and therefore where regulatory scrutiny assumes less of the burden. How can governments promote small-enough-to-fail institutions that manage risk well? This is the key question about the future of finance; and one part of the answer is hiding in plain sight. Governments must encourage hedge funds.

Hedge funds are clearly not the answer to all of the financial system's problems. They will not collect deposits, underwrite securities, or make loans to small companies. But when it comes to managing money without jeopardizing the financial system, hedge funds have proved their mettle. They are nearly always small enough to fail: Between 2000 and 2009, a total of about five thousand hedge funds went out of business, and not a single one required a taxpayer bailout. Because they mark all their assets to market and live in constant fear of margin calls from their brokers, hedge funds generally monitor risk better and recognize setbacks faster than rivals: If they take a severe hit, they tend to liquidate and close shop before there are secondary effects for the financial system. So rather than reining in risk taking by hedge funds, governments should encourage them to thrive and multiply and absorb more risk, shifting the job of high-stakes asset management from too-big-to-fail rivals. And since the goal is to have more hedge funds, burdening them with oversight is counterproductive. The chief policy prescription suggested by the history of the industry can be boiled down to two words: Don't regulate. And yet there is one persuasive argument for regulating hedge funds--or rather, regulating some of them. The persuasive argument is that hedge funds are growing. The case in favor of hedge funds is a case for entrepreneurial boutiques; when hedge funds cease to be small enough to fail, regulation is warranted. Equally, when hedge funds become public companies, they give up the private-partnership structure that has proved so effective in controlling risk: Again, the case for regulation becomes stronger.

The tentative bottom line on hedge-fund performance is surprisingly positive. Hedge funds do seem to generate profits beyond what they get from exposure to the market benchmarks. And despite much griping about excessive hedge-fund fees, there is alpha left over for clients. Of course, hedge funds are not a substitute for other investment vehicles. For ordinary savers, mutual funds that cheaply mimic an index remain the best option. But from the point of view of large investors, hedge funds compare well with most of their rivals. They are not more prone to insider trading or fraud, and they deliver real value for their clients.