EFF/KRF: We read Ms. Li's paper after getting your question. We don't think she addresses the central statistical issue in fund performance evaluation raised in our paper "Luck versus skill in the cross section of mutual fund returns," summarized in our Forum post "Luck versus skill in mutual fund performance." 

Like Ms. Li's paper, our paper also studies the performance of the population of active mutual funds over the entire history of each fund. The critical question we raise (which Ms. Li does not address) is: What distribution (cross section) of alpha estimates for active funds would we expect to see if in fact true alpha is zero for every fund. The answer is that the distribution of alpha estimates generated by luck alone is impressively wide with lots of extreme positive and negative values.

How does the actual distribution of alpha estimates for funds compare to the "luck" distribution? The short answer is, badly. The luck distribution says that if true alpha is zero for every active fund, alpha estimates should be symmetric about zero. But in fact 60%-70% of funds produce negative alpha estimates. What about the funds that produce positive alpha estimates for their entire history? The problem (not addressed by Ms. Li) is that with so many funds, lots of positive performance is expected by chance (luck) even if the true alpha for every fund is zero. We find that about 97% of the alphas in actual fund returns–including most positive alphas–are too low relative to what we expect if true alpha is zero.

Only the top 3% of active funds perform about as well as (but no better than) we expect if their true alphas are zero. The measured alphas of these funds for their entire histories are outrageously positive. These are the funds that get top ratings and are deluged with inflows. But their historical performance is only about as good as expected for the top 3% of funds when all performance is just luck and true alpha is zero. This means that despite the extremely good past performance of the top 3% of active funds, going forward we expect them to do about as well as comparable efficiently managed (that is, low cost highly diversified) passive funds, which produce zero alpha with high reliability. Going forward, we expect the remaining 97% of the active fund universe to perform worse than comparable efficiently managed passive funds.

How can active funds in aggregate do worse than expected if their true alphas are zero? Our results say that the true aggregate alpha for the active fund universe is negative by the amount of fees and expenses. The aggregate portfolio of active funds is close to the market portfolio, and it delivers the market return–before fees and expenses. Investors in active funds get returns after fees and expenses, and for them alpha is negative. 

Our results say that even the top 3% of active funds are only as good as we would expect if their true alphas are zero. Ms. Li concludes that large fractions of funds have positive true alphas. Why is she so much more optimistic about active management? From a statistical perspective, she evaluates each fund as if it is the only fund she considers. But like us, she measures the performance of every fund, and to draw the right conclusions, she must take this into account when evaluating individual funds. Failure to do so is not at all special to Ms. Li. The problem is rather universal among investors, amateur and professional, and it is reinforced by the media and many in the money management industry. 

Here's a simple way to illustrate the problem. Suppose we observe a person who correctly calls a coin flip five times in a row. The probability of such success if the flipper relies only on luck is about 3%, so we might conclude that there is something about the coin known only to the flipper or that he is prescient about coin flips. In either case, it looks likely that our guy is a talented coin flipper (read active manager). This inference is appropriate–if this is the only sequence of five coin flips we examine (the only active manager we evaluate). But if we examine larger and larger samples of five coin flips we are more and more likely to find at least one lucky active manager who is right all five times. In fact, in a sample of 100 managers we expect to find 3, in a sample of 1000 we expect 30, and in 10,000 we expect 300 to succeed just by chance.

In short, we identify the funds with the best performance by examining the performance of all funds, and this has rather dramatic implications for correct inferences about luck versus skill in the historical performance of the winners.