gameguru (32.49)

Gauging the Usefulness of Star Ratings

December 06, 2006 – Comments (2)

I was looking over the performance of the CAPS1StarIndex and CAPS5StarIndex today and I noticed that both of them are substantially outperforming the market. That is, if you had made outperform picks on all 200 of the stocks in either index, you would have a positive score in the CAPS system. (Note that CAPS1StarIndex currently has a negative score, as it makes underperform calls.)

The following then occurred to me:

The “market” benchmark is defined by CAPS as the S&P500, and therefore consists of 500 stocks, while the CAPS universe currently consists of 8348 stocks, 2858 of which have a star rating. This means there are currently about 572 stocks at each star level. Star levels have no set relationship to market capitalization. (However, one might expect larger-cap companies to be more “popular” and perhaps have higher star ratings. For instance, of the 100 highest stocks by market cap, 18 were 5-Star, 28 4-Star, 25 3-Star, 21 2-Star, and just 4 1-Star, while 4 were unrated. That’s nearly 2x as many 5- and 4-Star stocks as 1- and 2-Star stocks in the group.)

It is therefore possible that every star-rating level (1 through 5) could simultaneously outperform the market (defined as the S&P500).

Although the S&P500 is not simply the 500 largest stocks by market cap, it is limited to larger-cap stocks and primarily domestic U.S. stocks. The necessary conditions for this odd situation, then, are simply that small-to-mid cap and international stocks outperform large-cap stocks.

Of course, the converse is also true – all star rating levels could simultaneously underperform the market (defined as the S&P500) if large-cap stocks are doing better than other market classes.

I’m certainly not suggesting that star levels should have anything to do with market cap.

Rather, I’m wondering how well we can really gauge the usefulness of the star ratings generated by CAPS when comparing against just the S&P500. Should we be gauging against a broader benchmark – the Wilshire 5000 maybe? Or against multiple benchmarks – including international, small-cap, large-cap, and broad-market indices? Ultimately, we will need very specific figures – by how much does each star level outperform each benchmark – to determine whether the star ratings provide any useful information.

When the CAPS administration starts judging the system’s performance (and likely making marketing claims on that basis), I hope they’ll take these points under consideration.