TMF Should Reform the CAPS Rating System
September 22, 2009
– Comments (38)
Today, we've seen more proof that TMF should consider reforming the CAPS rating system (or create an "alternative ratings system"). I've seen at least four people as the "Top Fool" in the past two hours. The accuraracy component does not greatly skew results in what you might call the "normal levels", but on the periphery, slight changes have a radical impact on the rating system. This is especially true at the top level and one stock drifting from a -0.2 score to a +0.1 score can have a major change on one's CAPS rating.
There's also the issue that some of the top fools have made it to the top via cheapo strategies. I'm particularly annoyed by all the accuracy and points gaming from the low-volume .OB stocks. IRL, this would be very difficult to accomplish because (a) you wouldn't necessarily be able to buy at the price stated on CAPS, (b) there's a good likelihood that even if you could buy, you could only buy in minute amounts at the advertised price, and (c) some of these low-volume .OB stocks are nearly impossible to short IRL.
Another cheapo strategy has been shorting the ultras. GMX has spoken out about this strategy before and I originally did not oppose it, but at some point, I started seeing a lot of top Fools use absolutely nothing but this strategy. TMFEldrehad, in particular, has been extremely egregious in using it. (Hey, I love Eldrehad's blogs, but it's time to call him out on the junk strategy!)
That said, there are still a lot of people at the top who have gotten there through great stockpicking and what I'd consider more "realistic" calls. BullishBabo, Portefeuille and his eleven "friends", Tenmiles, and Vanamonde, just to name a few, are some of the CAPS players who I see as having a much less "tainted" rating than some others. I won't mention any of the names of the profiles I consider less "legitimate" (and actually, Eldrehad's record was pretty good until recently.)
How to Reform the System
Of course, I'm not just here to "complain." Rather, I'd like to see CAPS implement an alternative rating system that I believe would yield better results. I also believe it would lead to fewer scoring anomolies.
The first element to this system is to eliminate "scoring" based on a stock's percentage gains vs. the S&P's percentage's gain. It yields nonsensical results at times. Rather, it would make more sense if CAPS assigned a dollar value to each pick. For instance, if I green thumb MSFT, CAPS assigns me a $1000 investment in it. The S&P is also assigned a dollar value of $1000.
There are a few reasons why I prefer this method. The first reason is that it would make dividend payments simpler. Right now, CAPS has no good way to account for dividends. Currently, your initial price point is reduced when a dividend is paid out. However, this yields bizarre results because the initial price point is always decreasing, which skews the returns. Instead, CAPS should keep track of the stock price (via the $1000 investment) in one column and the total dividends accumulated in another.
For instance, I buy MSFT at $20. Two years later, it has increased to $30 and it paid out $5 in dividends. Meanwhile, the S&P started at 1000 and increased to 1200. Under the current system, MSFT would be credited with a 100% gain vs. the S&P's 20% gain, for a score of +80. Under my system, the value of my MSFT investment would be credited as $1500 and my second colum would record the $250 in dividends. This would mean a 75% gain for MSFT vs. the S&P's 20%, for a score of +55. This is a pretty major difference and I believe my system more accurately accounts for the dividends. I'd be curious to know if there's a way to keep track of the S&P's total dividends, as well.
The Alternative Ratings System
Step #2 in my reform process would be change the rating system. Returns over time become unrealistic on CAPS because it's scored via initial price point. Hence, if I buy a stock at $2 and it jumps to $20 within a year; then I hold onto it for 4 more years and it hits $22 at the end of the timeframe, my investment was really smart for Year 1, but really poor for Years 2-5. CAPS actually would credit the stock with a 900% gain after Y1 and a 1000% gain after Y5. Under the CAPS scoring system, that means I gained +100 points, despite the fact that my investment performed absolutely dismally from Y2-Y5. In fact, it only yielded a 10% return in those years after yielding 900% in Y1.
Short of turning CAPS into a portfolio management game, there's no easy solution to this problem. However, one way to account for it might be to weight the ratings based on recent returns. For instance, CAPS could measure 6 month return, 1-year return, 2-year return, 5-year return, and total return.
Instead of making score 2/3 of the rating and accuracy 1/3, how about making total return 50%, 6-month return 10%, 1-year return 10%, 2-year return 10%, 5-year return 10%, and accuracy 10%. This system would not be perfect by any stretch of the imagination, but at least it would create an incentive to close out picks with a lower initial-price point. I also think that even if the powers that be don't have any designs on my "alternative system", they should at least consider adding columns for these returns, and allowing people to sort by these returns.
Microcaps and .OBs/.PKs
The third part of my reform is one we're already debating --- changing what is ratable and what is not. I want to see more legitimate microcap stocks ratable and I'd also like to see some of these low-volume .OB stocks that people use to game the system eliminated. I actually wouldn't mind if people could still rate these .OB stocks --- I just don't think they should count for CAPS scoring.
My suggestion has been to allow any stock trading on a major American exchange (NYSE, Nasdaq, AMEX) to be ratable so long as its market cap is above $10 million. I also think that .OB and .PK stocks should be required to meet a very strict threshold to be ratable. I'd prefer a $1 billion market cap and some sort of minimum volume threshold. I don't see these as magic numbers --- I just don't see much of a point in allowing people to "game" the scoring and accuracy components by making picks on stocks that often have the same price for over a week.
My other suggestion would be that, if we are not going to implement stricter requirements for ratability on .OB/.PK stocks, we should require longer holding periods. Instead of a 7-day minimum, it should be increased to a 6-month minimum. That would at least make it very difficult to play off the volatility.
Minimizing Accuracy
There are a lot of problems with the CAPS rating system, but I view accuracy as the single biggest problem. I understand why the accuracy component was implemented, but it does not serve its function particularly well. In my above rating system suggestion, I said it should be lowered to 10% of the rating, as opposed to 33.33%. This would at least minimize it. However, maybe there are other ways to deal with it, as well.
Instead of scoring accuracy based on microscopic changes, perhaps there should be general ranges. For instance, everyone above 70% gets the same score for accuracy. Then, everyone within the 60% - 70% range, obtains the same score. Everyone in the 50% - 60% range also gets the same score. This is not without its problems, because there are no major differences between someone with 59.7% accuracy and a person with 60.2% accuracy, but as I said, the accuracy component becomes more nonsensical at the periphery, and this would eliminate that problem because nearly everyone in the top 100 has over 70% accuracy. This would force people to start picking stocks for potential investment returns rather than trying to mindlessly game the accuracy component.
Some of you might think there are absolutely no legitimate reasons for the accuracy component. I'd argue that there are a few. Most notably, it prevents people from ending and re-initiating all their picks simply to get a lower price point. One notable example --- EPS100Momentum. Sure he has a score of 5,600+, but his accuracy is 35%!!!! So, I can actually see some examples where the accuracy component makes some sense. In a real portfolio, it would not be realistic to do what EPS100 did, because his portfolio value would have been so low after a certain point, that he wouldn't have had cash left over to make those big returns.
So, in any case, those are my basic ideas. I realize that the odds of them being accepted are low, but even if they spark discussion and some elements get adopted, that's not an altogether bad thing. At the current time, however, I view the CAPS rating system as a poor indicator of investing acumen. I agree with portefeuille in viewing the scoring component as a better metric, but it has its flaws, as well.
Feel free to treat this as an idea blog. I might add some more over time myself.
RSS Headlines
Fool UK