Why Indexing Fails

The recent paper “Why Indexing Works” [1] gives a probabilistic explanation of the futility of the Active process and why Passive Indexing is hard to beat. For every 1,000 people who read The Wall Street Journal, maybe 10 read the Bloomberg Markets and for every 10 who read the latest issue of Bloomberg Markets maybe 1 read the research paper titled Another Reason for Active’s Decline [2]. And you don’t need a geologist to tell you that the chances to dig and find are small. This is why making a mathematical case against the underperformance of the $11 trillion plus actively managed fund markets using hypothetical probabilities is not easy.

Probabilities don’t lie, but the assumptions that generate those probabilities might be flimsy indicating expected outperformance or underperformance, but they cannot claim to own the basis for poor Active underperformance. Manufacturing hypothetical probabilities are an easy and rather effective way to put inefficiency into the system. The more quants research, the more manufacturing of hypothetical probabilities occurs and hence more the inefficiency. The seemingly farcical aspect of all of this is that the probabilities cited by the authors of the paper in discussion assume efficiency.

The paper in question makes more than a few assumptions. First, it makes the assumption of Geometric Brownian Motion (GBM) [3] as a realistic statistical distribution to understand markets. Random walk, efficiency, normality is stone age thinking in today’s quantum world. There is enough literature out there on power law and its natural and stock market expression. Despite such prominence, the power law has an army of detractors. In such a context, the GBM hypothesis has limited relevance because it isn’t as universal as the power law in its expression. And even if there was a perfect statistical distribution, thought leaders (thinkers if you will) like Eugene Stanley [4] and his global academic followers would have determined it and Mandelbrot [5] would have attained Wall Street cult.

Second, the authors assume that Active managers are still working with redundant statistical distributions to generate alpha.

The third assumption is a Trojan horse that completely turns the argument against Indexing itself. Think about it, is selection just for the Active manager? No. Can the Passive manager do without selection? An impossible feat. Are the Indexing companies building baskets insulated from the selection process? Hardly. Selection is everywhere, and it blurs the line between what’s Active and what’s Passive. Simply stated, Passive should not be considered different from active because even passive undergoes changes in composition periodically, monthly, quarterly, yearly or when some committee makes an executive decision.

Indexing is not just about a certain Market Capitalized weighted basket [6]. Indexing is about factors today and few claim to have discovered 458 factors [7], most of them claiming to drive alpha. This means that the probability which the authors cite as working against stock selecting Active managers also works against the Beta and Smart Beta fund which are at the mercy of the same probabilities that grace the various Indexing methodologies, giving them a year of success (outperformance) followed by perhaps a few years of failure (underperformance). So just like Active managers, Indexing as an idea is destined to fail periodically not only because of its construction (which requires selection – ok call if active if you want), but also because investors and asset managers have thousands of investment choices from which to “Select”.

This means that the positive skew problem highlighted by the authors is not just the bane for the Active manager but even disastrous for the Passive industry. Hence, the probabilistic argument made by the authors in favor of Passive and against Active falls apart. The surviving Active managers’ searches for ‘gold in garbage’ (a data science expression) has advanced beyond the use of generic statistical distributions. It’s time we saw some new age mathematical research that makes a real case to solve the alpha problems rather than the same banal smarter Passive, dumb Active overused themes.


[1] Heaton J. B, Polson N. G, Witte J. H, “Why Indexing Works”, October 2015, SSRN.
[2] Another Reason for Active’s Decline, April/May 2017, Bloomberg Markets.
[3] Saichev A. I, Malevergne Y, Sornette, D, “Theories of Zipf’s Law and beyond”, 2009, Springer.
[4] Matia K, Pal M, Stanley E, Salunkay H, “EPL – Scale dependent price fluctuations for the Indian stock market”, 2004, Europhysics Letters.
[5] Mandelbrot B, “A Multifractal Walk down Wall Street”, February 1999, Scientific American.
[6] Pal M, “Is Smart Beta Dumb”, March 2017, SSRN.
[7] Authors J, “A clinical test of the 5 most popular Smart Beta factors”, March 2017, Financial Times