Use access key #2 to skip to page content.

lrwilliams84 (< 20)

Does CAPS score correlate with performance?



August 02, 2009 – Comments (8)

I've been wondering if CAPS score (number of stars) is as predictive of performance as TMF likes to claim.  I looked at the stock advisor recommendations since April 2007 and sorted them by CAPS rating at the time of the recommendation.

5*: average return -5%, range -70% to +73% (total of 22)

4*: average return -9%, range -41% to +18% (total of 12)

3 and 2*: average return -1%, range -38% to +123 % (total of 10)

For comparison, S&P average return -11%, range -36% to +31%

Hmmm, looks like 2 and 3 star stocks outperformed 4 and 5 star stocks, but it's probably mostly due to that one outlier (+123%).  So, small number statistics are an issue.  But, how many stocks would I want to own?  Probably not more than this. 

And look at the range of returns, e.g., -70% to +73% for the 5* stocks.  That's pretty scary.  If I bought every 5* stock they recommended when they recommended it, I would be 6% ahead of the S&P.  But, if I missed one or two (e.g., the +73% one), my returns could be much worse (or much better).  That seems pretty random.

I guess I was hoping for some way to filter the stock advisor recommendations since I can't buy everything they recommend (too many stocks), but CAPS stars is not going to help.  Maybe I will just go buy a mutual fund and stop spending time on this.




8 Comments – Post Your Own

#1) On August 02, 2009 at 9:54 AM, TMFJake (77.11) wrote:

LRWilliams, sounds as if you've seen some of the performance studies that measure the CAPS star rating.  The Harvard/Yale study is probably the most demonstrative in measuring CAPS' excess returns persist even when you risk adjust performance.  This study is being updated with complete 2008 and early 2009 data by the way..

But you do touch on a fundamental problem.  As your results show, you can increase your odds of finding winning stocks by screening on highly-rated CAPS stocks, but there's no way to guarantee you'll find a winner short or long term.

Even with the limitations of your random study, it might be interesting to look in more detail at the 5-star winners vs. losers.  Are there any common traits of the losing stocks that might be instructive?  Were these richly valued growth stocks (which were disproportionately punished in the 2008 collapse)??  If so, you might conclude that you'll give up some upside under different market conditions to maintain more downside protection.

I think you can use The Motley Fool Stock Advisor Scorecard and CAPS rating in this way to achieve a profitable edge.  

Happy Weekend!



Report this comment
#2) On August 02, 2009 at 4:46 PM, portefeuille (98.84) wrote:

The star rating system is really not very good but at least it is not entirely worthless.

Using the star rating system as a contra-indicator my even be a winning strategy if you base your calls on value investing as this observation might suggest.

Again one of the main problems with the star rating system are the three curses of the "caps" game. The $1.50, $100 million cutoffs and the "accuracy". On the "accuracy" thing you might want to have a look at this post and this post. For those cutoffs see this post and comments #8-10 here. For some aspect of the "accuracy", random walk (and buy and hold strategy) aspects see comments #30,35,40-47 here.

Now how do these 3 curses (again, they are the 1.5 and 100 rules and the way "accuracy" is calculated) affect the value of the star ranking system?

Well, this has been explained quite a few times so maybe I just safe you (and me!) from links and quotes. The "short" version goes something like this. Stocks that drop under one of those cutoffs can no longer be objects of new calls. What aggravates this situation is that those that have given it an "outperform" rating usually are unlikely to close their calls because most of them are "hopeful" that the stock has a reasonable chance of "coming back" and making their calls right "someday" and thus improving their accuracy. And why do they care about this "accuracy"? Because it is a major part of their overall rank. And why do they care about their rank? camistocks would say because they are Americans (at least most of them are) and Americans love their top tens.

I could elaborate, but I guess all of this is utterly obvious, numerous players have "admitted" that they try to enhance their accuracy (usually they call it "gaming the system"), new players are warned that the worst thing to do is to close a call that is not "in the green by 5 points) and so on and so on.

This is really a plague by now, everybody knows that and I guess that 99% agree that no one in his right mind would ever reinvent any of those 3 curses but since they are there laziness and indifference rule I guess.

A friend of mind in these situations likes to tell me that is a similar thing with the motorcycle. If it had not been invented decades ago it would not have the slightest chance of being allowed on public roads because today not even cars without seatbelts are allowed. But for historic reasons there are still motorcycles on public roads and for analogous reasons these 3 curses will be "with us" for quite some time I guess ...



Report this comment
#3) On August 02, 2009 at 5:33 PM, streetflame (29.24) wrote:

Port, you are right. I think if TMF removed those three conditions, the predictive strength of CAPS would improve enormously (probably increase 5* vs 1* returns by over 10%/year).  One more simple change I would add - make the rating system more continuous (1-100 instead of 1-5).  Unfortunately, successful systems often breed complacency.

Report this comment
#4) On August 02, 2009 at 5:50 PM, streetflame (29.24) wrote:

One other thing - if instead of binary rating there was 'strong buy', 'buy', 'hold', 'sell', and 'strong sell' which could be switched between without penalty - that would give another strong injection to predictive strength.  And if CAPS was real time that would certainly help a small amount.  All told between these 6 improvements, CAPS is very likely leaving over 20% top rated to bottom rated differential returns on the table.

Report this comment
#5) On August 02, 2009 at 6:24 PM, streetflame (29.24) wrote:

"Using the star rating system as a contra-indicator my even be a winning strategy if you base your calls on value investing as this observation might suggest."

My alter-ego UltraContrarian, who only picks 1 and 2 star stocks to outperform and 4 and 5 star stocks to underperform on a value basis, has been flat for almost 3 months.

I think I still have an edge over the market going against the ratings, but it has decreased tremendously.  I'd like to think a substantial part of this is due to improvements in CAPS rating quality and not just luck, structural market conditions (although surely a large part of my outperformance could be chalked up to beta or non-sustainable alpha) or degradation of my analysis.

Report this comment
#6) On August 02, 2009 at 6:59 PM, portefeuille (98.84) wrote:

real time, free to choose any rating, no 1.50, 100 rules, no "accuracy", rate any stock/fund in the world. those are some of the advantages of my list here. it's just great, hehe ...

Report this comment
#7) On August 03, 2009 at 8:22 AM, lrwilliams84 (< 20) wrote:


Thanks for the comments, everyone, and thanks for the pointer to the Harvard/Yale study, TMFJake.  That's an interesting paper, and I'll be curious to see the results of their statistical analysis including 2008 and early 2009 data.  Bottom line, CAPS score is a good predictor of excess returns if you average over lots of data, but for a small number of stocks, the results fall prey to small number statistics.  I did try to look at the differences between my set of 5* stocks that outperformed and ones that underperformed, but I can't see any particular pattern.  Again, I think the number of stocks is just too small.



Report this comment
#8) On August 03, 2009 at 8:26 AM, portefeuille (98.84) wrote:

Again, I think the number of stocks is just too small.

That is not the main problem. Have a look at the post I linked to comment #2 above ...

Report this comment

Blog Archive

August (1)

Featured Broker Partners