Use access key #2 to skip to page content.

JakilaTheHun (99.93)

TMF Should Reform the CAPS Rating System

Recs

34

September 22, 2009 – Comments (38)

Today, we've seen more proof that TMF should consider reforming the CAPS rating system (or create an "alternative ratings system").  I've seen at least four people as the "Top Fool" in the past two hours.  The accuraracy component does not greatly skew results in what you might call the "normal levels", but on the periphery, slight changes have a radical impact on the rating system.  This is especially true at the top level and one stock drifting from a -0.2 score to a +0.1 score can have a major change on one's CAPS rating. 

There's also the issue that some of the top fools have made it to the top via cheapo strategies.  I'm particularly annoyed by all the accuracy and points gaming from the low-volume .OB stocks.  IRL, this would be very difficult to accomplish because (a) you wouldn't necessarily be able to buy at the price stated on CAPS, (b) there's a good likelihood that even if you could buy, you could only buy in minute amounts at the advertised price, and (c)  some of these low-volume .OB stocks are nearly impossible to short IRL.  

Another cheapo strategy has been shorting the ultras.  GMX has spoken out about this strategy before and I originally did not oppose it, but at some point, I started seeing a lot of top Fools use absolutely nothing but this strategy.  TMFEldrehad, in particular, has been extremely egregious in using it.  (Hey, I love Eldrehad's blogs, but it's time to call him out on the junk strategy!)

That said, there are still a lot of people at the top who have gotten there through great stockpicking and what I'd consider more "realistic" calls.  BullishBabo, Portefeuille and his eleven "friends", Tenmiles, and Vanamonde, just to name a few, are some of the CAPS players who I see as having a much less "tainted" rating than some others.  I won't mention any of the names of the profiles I consider less "legitimate" (and actually, Eldrehad's record was pretty good until recently.)  

How to Reform the System

Of course, I'm not just here to "complain."  Rather, I'd like to see CAPS implement an alternative rating system that I believe would yield better results.  I also believe it would lead to fewer scoring anomolies.  

The first element to this system is to eliminate "scoring" based on a stock's percentage gains vs. the S&P's percentage's gain.  It yields nonsensical results at times.  Rather, it would make more sense if CAPS assigned a dollar value to each pick.  For instance, if I green thumb MSFT, CAPS assigns me a $1000 investment in it.  The S&P is also assigned a dollar value of $1000. 

There are a few reasons why I prefer this method.  The first reason is that it would make dividend payments simpler.  Right now, CAPS has no good way to account for dividends.  Currently, your initial price point is reduced when a dividend is paid out.  However, this yields bizarre results because the initial price point is always decreasing, which skews the returns.  Instead, CAPS should keep track of the stock price (via the $1000 investment) in one column and the total dividends accumulated in another. 

For instance, I buy MSFT at $20.  Two years later, it has increased to $30 and it paid out $5 in dividends.  Meanwhile, the S&P started at 1000 and increased to 1200.  Under the current system, MSFT would be credited with a 100% gain vs. the S&P's 20% gain, for a score of +80.  Under my system, the value of my MSFT investment would be credited as $1500 and my second colum would record the $250 in dividends.  This would mean a 75% gain for MSFT vs. the S&P's 20%, for a score of +55.  This is a pretty major difference and I believe my system more accurately accounts for the dividends.  I'd be curious to know if there's a way to keep track of the S&P's total dividends, as well.  

The Alternative Ratings System

Step #2 in my reform process would be change the rating system.  Returns over time become unrealistic on CAPS because it's scored via initial price point.  Hence, if I buy a stock at $2 and it jumps to $20 within a year; then I hold onto it for 4 more years and it hits $22 at the end of the timeframe, my investment was really smart for Year 1, but really poor for Years 2-5.  CAPS actually would credit the stock with a 900% gain after Y1 and a 1000% gain after Y5.  Under the CAPS scoring system, that means I gained +100 points, despite the fact that my investment performed absolutely dismally from Y2-Y5.  In fact, it only yielded a 10% return in those years after yielding 900% in Y1.  

Short of turning CAPS into a portfolio management game, there's no easy solution to this problem.  However, one way to account for it might be to weight the ratings based on recent returns.  For instance, CAPS could measure 6 month return, 1-year return, 2-year return, 5-year return, and total return. 

Instead of making score 2/3 of the rating and accuracy 1/3, how about making total return 50%, 6-month return 10%, 1-year return 10%, 2-year return 10%, 5-year return 10%, and accuracy 10%.  This system would not be perfect by any stretch of the imagination, but at least it would create an incentive to close out picks with a lower initial-price point.  I also think that even if the powers that be don't have any designs on my "alternative system", they should at least consider adding columns for these returns, and allowing people to sort by these returns.  

Microcaps and .OBs/.PKs

The third part of my reform is one we're already debating --- changing what is ratable and what is not.  I want to see more legitimate microcap stocks ratable and I'd also like to see some of these low-volume .OB stocks that people use to game the system eliminated.  I actually wouldn't mind if people could still rate these .OB stocks --- I just don't think they should count for CAPS scoring.  

My suggestion has been to allow any stock trading on a major American exchange (NYSE, Nasdaq, AMEX) to be ratable so long as its market cap is above $10 million.  I also think that .OB and .PK stocks should be required to meet a very strict threshold to be ratable.  I'd prefer a $1 billion market cap and some sort of minimum volume threshold.  I don't see these as magic numbers --- I just don't see much of a point in allowing people to "game" the scoring and accuracy components by making picks on stocks that often have the same price for over a week.  

My other suggestion would be that, if we are not going to implement stricter requirements for ratability on .OB/.PK stocks, we should require longer holding periods.  Instead of a 7-day minimum, it should be increased to a 6-month minimum.  That would at least make it very difficult to play off the volatility.  

Minimizing Accuracy

There are a lot of problems with the CAPS rating system, but I view accuracy as the single biggest problem.  I understand why the accuracy component was implemented, but it does not serve its function particularly well.  In my above rating system suggestion, I said it should be lowered to 10% of the rating, as opposed to 33.33%.  This would at least minimize it.  However, maybe there are other ways to deal with it, as well.

Instead of scoring accuracy based on microscopic changes, perhaps there should be general ranges.  For instance, everyone above 70% gets the same score for accuracy.  Then, everyone within the 60% - 70% range, obtains the same score.  Everyone in the 50% - 60% range also gets the same score.  This is not without its problems, because there are no major differences between someone with 59.7% accuracy and a person with 60.2% accuracy, but as I said, the accuracy component becomes more nonsensical at the periphery, and this would eliminate that problem because nearly everyone in the top 100 has over 70% accuracy. This would force people to start picking stocks for potential investment returns rather than trying to mindlessly game the accuracy component.  

Some of you might think there are absolutely no legitimate reasons for the accuracy component.  I'd argue that there are a few.  Most notably, it prevents people from ending and re-initiating all their picks simply to get a lower price point.  One notable example --- EPS100Momentum.  Sure he has a score of 5,600+, but his accuracy is 35%!!!! So, I can actually see some examples where the accuracy component makes some sense.  In a real portfolio, it would not be realistic to do what EPS100 did, because his portfolio value would have been so low after a certain point, that he wouldn't have had cash left over to make those big returns. 

 

 

So, in any case, those are my basic ideas.  I realize that the odds of them being accepted are low, but even if they spark discussion and some elements get adopted, that's not an altogether bad thing.  At the current time, however, I view the CAPS rating system as a poor indicator of investing acumen.  I agree with portefeuille in viewing the scoring component as a better metric, but it has its flaws, as well. 

 

Feel free to treat this as an idea blog.  I might add some more over time myself. 

38 Comments – Post Your Own

#1) On September 22, 2009 at 12:06 PM, anticitrade (99.66) wrote:

I agree with everything unconditionally.  (for whatever that is worth)

Report this comment
#2) On September 22, 2009 at 12:08 PM, portefeuille (99.60) wrote:

Currently, your initial price point is reduced when a dividend is paid out.  However, this yields bizarre results because the initial price point is always decreasing, which skews the returns.

The way they "treat" dividends is almost correct. This is from comment #24 here.

-------------------------

What you want is that the "performance", i.e. the relative change, is not affected by the dividend payment.

The relative change before the dividend payment is

(b - a)/a,

where b is the price before the dividend is paid and a is the old cost basis.

The relative change after the dividend payment is

(b - d - x * a)/(x * a),

where d is the dividend and the new cost basis is a * x, i.e.

x = old cost basis / new cost basis.

So all you have to do is solve

(b - a)/a = (b - d - x * a)/(x * a)

for x, giving you x = (1 - d/b).

Another way to arrive at this:

Before the dividend you have n shares with a cost basis of a/share. You get n * d/(b - d) new shares ("dividends reinvested"). So the new cost basis is

n * a/(n + n * d/(b - d)) = a * (1 - d/b).

So the only problem that could arise is the case where d > b, making x = 1 - d/b negative. But cases in which the stock price right before the dividend payment b is smaller than the dividend d are "rare" and should probably be treated individually since the reason for this "strange" situation might have to be taken into account (tax laws for example).

------------------------- 

Report this comment
#3) On September 22, 2009 at 12:15 PM, SkepticalOx (99.45) wrote:

Well, if you consider CAPS to be sort of like an analyst game, then accuracy matters. Would you listen to an analyst who is right 35% of the time?

From a portfolio manager's standpoint, it matters much less. You could be right 5% of the time as long as your right calls made up for all your bad calls in terms of performance.

With Fools making outperform and underperform calls, CAPS seems much more like a game where we're playing analysts and not a portfolio simulation game. Accuracy matters a whole lot then.

Report this comment
#4) On September 22, 2009 at 12:20 PM, portefeuille (99.60) wrote:

-------------------------

Don't pay too much attention to CAPS rankings

June 19, 2009

Yesterday I was perusing the blogs and came across a comment I've heard too many times. An individual was citing another individual's CAPS score and accuracy, implying that the values in some way lent credibility to the opinions they were voicing. It's quite distressing to read things like this, not because I care what people do on CAPS, but because I worry that people may make decisions with their hard earned money based only on recommendations from Top Fools. I'm going to explore in detail the accuracy component of the CAPS ranking, but first a note on score.
Browse through the profiles of the leader board and you'll see two tickers on almost every single one. GMGMQ.PK (yes, that's GM equity), and PXCE.OB. It is near certain that GM equity is worthless, and PXCE is a shell of a company that has most likely had it's price manipulated by "pump and dumpers." So, to rack up points, you simply red thumb these (in some cases more than once) and it's an easy 100 points plus an additional "correct call." This sort of thing is clearly not actionable in practice and so is irrelevant from an investing stand point, yet in some people's mind, the points it generates lends credibility to one's arguments. More important than a person's score is how they have achieved such a score. Have they made picks that are actionable in practice, and supported their pick with thoughtful commentary, or have they gamed the system?
Accuracy
Accuracy is critically important to CAPS scores, but is it really important? I'll often hear comments like "individual X has been right on Y% of their calls, therefore they should be listened to." Oh really? I will submit that it is not at all relevant, and in fact, a computer could beat anybody's accuracy. Let's explore.
It is well known (well enough that I'm not going to source this statement) that stock prices typically follow a random walk about some longer term trend. With a margin of "success" of only 5%, how well would you do if you simply made picks at random and closed them once the score exceeded 5? To do this we simply need to run a random walk simulator and compare the results to what human players have done.
Methodology:
Lognormal daily stock prices projected for 252 days, or approximately one year (not attempting to model intra-day). The price is not in absolute terms, but in terms of price relative to the S&P 500, so it begins at 1, and a price of 1.05 gives a success on an outperform pick. The pick, either outperform or underperform is chosen at random (this part isn't actually relevant). Volatility is fixed throughout the projection. I projected 10,000 picks, and closed each as soon as it generated a score greater than 5. No pick is closed with a negative score. Esentially, I'm trying to replicate with a simulator the kind of strategy most of the top CAPS players use.
Clearly the volatility assumption is critical. The higher the volatility, the more likely it will at some point be successful. So what is our assumption? I looked at three widely followed companies, and took a standard deviation of the difference in their daily returns and that of the S&P 500. Since prices in our model are relative, so too should volatility. The period of time is 1st quarter 2009.
BAC - 11.4%
XOM - 1.3%
AAPL - 1.9%
Results
A 2% volatility during this period seemed like the most reasonable assumption to me for a real life (not CAPS) portfolio. Clearly in CAPS you can take on more risk than you would with your brokerage account. The results may surprise some people. Fully 85% of the picks were recorded as a success. That means, if you simply picked stocks at random, at some point, 85% of them would have beaten the S&P by 5%. Increase volatility to 5%, and the success ratio is 92%. If we use BAC's volatility, which in truth is the kind of pick a lot of CAPS players have been using over the past 9 months, (think triple levered ETFS like FAZ and FAS) the success ratio is a whopping 95%. So, in actuality, CAPS players, even the best, are underachieving the accuracy a computer could generate. Chalk it up to the picks they actually believe in and hold for a longer period of time.
I'll conclude by saying what I think everyone already knows, but perhaps at times forgets. When it comes to investing, it's not what you do, but why and how you do it. The Fool with a rating of 99 point something, score over 10K, 85% accuracy and average score of 5 is doing nothing more than playing the CAPS game, and should not be taken any more seriously than the Fool with a score <20. The strength of a person's research and argument is what's really important, and is the true value of this site.

-------------------------

Report this comment
#5) On September 22, 2009 at 12:23 PM, Melaschasm (52.63) wrote:

I agree with most of what you have said.  Unless there is a massive overhaul for CAPS scoring, I think the most important issue is to reduce the impact of accuracy on a player's rating.

It seems like an adjustment to the formula to reduce the importance of accuracy would be a somewhat simple adjustment.  I do not know the exact formula, but the example below at least gives and idea of what direction CAPS should go, in my opinion.

Example if a 1% difference in accuracy is equal to a 20% difference in score, then a change so that a 1% difference in accuracy should only equal a 10% difference in score.  Thus if one person has a score of 100 and the other has a score of 110, but 1% more accuracy, the person with a score of 110 should be tied, rather than have a significantly higher rating.

Report this comment
#6) On September 22, 2009 at 12:38 PM, UltraContrarian (31.74) wrote:

So is this right port? 

Buy MSFT @ $20
      Cost basis = $20
MSFT @ $25 - pays out $2.50 in dividends
      Adjusted cost basis = 20 * (1 - 2.50/25) = $18
MSFT @ $30 - pays out $2.50 in dividends
      Adjusted cost basis = 18 * (1 - 2.50/30) = $16.50
Sell MSFT @ $30
Total return = 30 / 16.50 = +82%

Jakila, note that the return is higher than you calculated because dividends are paid out along the way and reinvested, instead of one lump sum at the end.

"I'd be curious to know if there's a way to keep track of the S&P's total dividends, as well."

There is, the total return for SPY is computed exactly the same way as the stocks it is compared against.

Report this comment
#7) On September 22, 2009 at 12:49 PM, JakilaTheHun (99.93) wrote:

Jakila, note that the return is higher than you calculated because dividends are paid out along the way and reinvested, instead of one lump sum at the end.

I realize that, but tacking them on at the end still makes infinitely more sense than lowering the cost-basis.   I wouldn't be opposed to CAPS "automatically re-investing" the dividends, either.  This would also work better under my $1000 position idea than under the current system.  Perhaps, the "accumulated dividends" column could be presumed to be re-invested.  

Report this comment
#8) On September 22, 2009 at 12:53 PM, portefeuille (99.60) wrote:

So is this right port?

Yes, the total return would be 27/33 * 100% = ca. 82%.

Report this comment
#9) On September 22, 2009 at 12:55 PM, portefeuille (99.60) wrote:

I wouldn't be opposed to CAPS "automatically re-investing" the dividends, either.

Re-investing leads to the same result as I have shown in comment #2 above. Great post, by the way!

Report this comment
#10) On September 22, 2009 at 1:14 PM, gembree (99.91) wrote:

I think you would find most of us agree, but just can't help making lucrative picks to fill the space in that 200-pick portfolio.

Port is correct on dividends.

Really the system is very broken and I would prefer a clean wipe and reboot with better yardsticks, but I don't think that's what TMF has in mind.

Report this comment
#11) On September 22, 2009 at 1:15 PM, UltraContrarian (31.74) wrote:

Sorry Jakila, you are wrong and TMF and port are right.  The cost basis is adjusted by division in order to make it equivalent to reinvesting dividends.Maybe you are thinking of adjusting the cost basis by subtraction (like we use when filling out income taxes) - that would not be right for calculating investment returns and that's why TMF doesn't use it.

Other than that however, great post.

Report this comment
#12) On September 22, 2009 at 2:20 PM, bigpeach (30.55) wrote:

The other way to think of it, is that CAPS is not meant to rate players, rather it is a stock rating system using "group think." Sure it's got a player rating system, but it's rather meaningless and I think everyone knows that. If people wanted to rate themselves against others as investors, there are lots of portfolio simulators out there. CAPS is not it, and wasn't meant to be.

If I had a gripe, it is not with how players are rated, I couldn't care less about that, rather it is that a bogus ranking system impacts an individual stock rating, via high rated players having disproportionate impact on star rating. Especially considering (someone else's words) many of the "top players" couldn't pick an outperformer if it smacked them in the face.

Report this comment
#13) On September 22, 2009 at 2:37 PM, JakilaTheHun (99.93) wrote:

If I had a gripe, it is not with how players are rated, I couldn't care less about that, rather it is that a bogus ranking system impacts an individual stock rating, via high rated players having disproportionate impact on star rating.

That's true, bigpeach, but you might also consider the fact that the reason why the star ratings have issues to begin with is because the underlying CAPS rating system has flaws.  For instance, there are many stocks that will remain 5 star stocks or 1 star stocks for a very long time, solely because people don't want to close out their old picks and lose accuracy points.  

The functioning of the star rating system is directly related to the underlying player rating system, since it is what drives incentives in the first place.

Report this comment
#14) On September 22, 2009 at 2:43 PM, TMFJake (52.53) wrote:

Jakila, great post. 

I think you've participated in some of the recent discussions about ways to improve CAPS scoring.  In a recent blog post, I highlighted the consensus ideas that I'd like to move forward with later this year:

1.  An alternative ranking based solely on your "top picks."

2.  Implementation of a volume filter to replace the market cap and/or share price minimums.

3.  An alternative ranking system based on rolling 12-month returns.

4.  An improved ranking system for Sector/Industry leaders.

These changes address some, but not all, of the concerns you list above. My inclination is still to start with this list, but, since this list is for work later this year, I'm open to feedback that we should change our priorities. :)

Report this comment
#15) On September 22, 2009 at 3:05 PM, kaskoosek (74.14) wrote:

TMFJake 

 

Can we remove ultras from caps all together? 

Report this comment
#16) On September 22, 2009 at 3:59 PM, AllStarPortfolio (20.64) wrote:

I think accuracy is very important, because this is a stock rating game. But i think that maybe stocks should have a short term rating, and a long term rating?
   What if, instead of having 5 years come up automatically when you rate outperform/underperform it came up blank and you had to pick one? that would bring the timeframe into play more.
   With that adjustment it would be fairly easy to rate stocks as 'long term 5 star' and short term 2 star' at the same time.
   It would also allow us to actually rate on accuracy better. A player who closed picks all the time without reaching their timeframe would not rate as high as those that reached their timeframe.

 As far as dividends, i don't really have a suggestion, but i can see where there could be real problems in the future.
   What if a micro cap IPOs at 1.00, and then in 5 or 10 years shoots to 100,00, and starts paying a divivdend of 1.00 quarterly. I don't think this would be all too rare in a long term game. Soon your buy point would be far in the negative.
   Does anybody know of a stock that has a similar profile to this scenario?

-solaris

Report this comment
#17) On September 22, 2009 at 4:12 PM, portefeuille (99.60) wrote:

   What if a micro cap IPOs at 1.00, and then in 5 or 10 years shoots to 100,00, and starts paying a divivdend of 1.00 quarterly. I don't think this would be all too rare in a long term game. Soon your buy point would be far in the negative.

The new cost basis would be $1 * x = $1 * (1 - d/b) = $1 * (1 - 1/100) = $0.99.

Report this comment
#18) On September 22, 2009 at 4:13 PM, TMFJake (52.53) wrote:

kaskoosek:  We could, although I still wonder whether the current market conditions are exaggerating the the impact of Ultras on our rating system.  I personally don't have a hard time filtering out any noise that I see with scorecards loaded with Ultras.  But if the commmunity really thinks they should be eliminated, I'm not dead set against it. I just hate excluding them when we consistently get feedback that we should allow for some form of weighting our doubling down.  Ultras are a form of doubling down, albeit a screwy way to obtain leverage from my perspective...

Report this comment
#19) On September 22, 2009 at 4:16 PM, mikecart1 (99.02) wrote:

Who cares about the rating system.  I have yet to pick a single stock the top Fool on this site has made.  That shows how much I respect or care about the system.  I use it more for narrowing down my own real life picks.

Report this comment
#20) On September 22, 2009 at 4:38 PM, bigpeach (30.55) wrote:

By the way Jake, I like the new screens you added. The highest stock returns screen I would consider the most useful for finding people who actually know what they're doing, but it doesn't seem to work as intended. Plus, even if it did work, it should really include a minimum number of picks for obvious reasons.

See user gsmetz who shows up in the screen with an average return of 160% for an example of what I mean about it not working.

Report this comment
#21) On September 22, 2009 at 4:52 PM, bigpeach (30.55) wrote:

Plus, returns are a skewed distribution, so it should use median as a measure of center rather than mean. Despite my criticism, I do like the new top ten lists.

Report this comment
#22) On September 22, 2009 at 5:02 PM, TMFJake (52.53) wrote:

bigpeach:  Thanks!  I filed a bug on the list you referenced.

Report this comment
#23) On September 22, 2009 at 5:03 PM, mustbepatient (29.48) wrote:

I vote to keep the ultras.  Face it, buying and shorting them are both options for generating returns in real life.

Report this comment
#24) On September 22, 2009 at 5:19 PM, outoffocus (23.20) wrote:

Well since everyone is giving suggestions I'll give mine.  I think any stock that doesnt meet the share price minimum should not be allowed to be a 5 star stock.  It's frustrating looking at the CAPS from page at "Today's Hottest Five Star Stocks" only to see that the stock is up 200% on a 20 cent move in the share price.  In other words if a stock is selling for less than $1 per share (an has been for some time) then doesnt that automatically imply that its not a 5 star stock?   The only reason those stocks are holding a 5 star rating is because some inactive CAPS account greenthumbed it in 2007 and hasn't checked up on it since. And yet the stock remains 5 stars because it doesnt meet the share price/market cap minimum for us to red thumb it down.

I say if a stock falls below the share price/market cap minimum it automatically loses its 5 star rating.

Report this comment
#25) On September 22, 2009 at 5:30 PM, Melaschasm (52.63) wrote:

I agree that accuracy is important, but the current system seems to reward those who grind accuracy.

Maybe a change so that you can only get accuracy points for a stock once a month, unless you use an opposit pick as your last call.  For example:

Green thumb stock X and close it 5 times in one month = 1 accuracy point.

Green thumb stock X once a month for 5 months = 5 accuracy points

Green thumb stock X then red thumb stock X, then green thumb stock X in one month = 3 accuracy points.

 Or have it be set by price so that if you keep picking the same stock in the same direction, you only get the accuracy points once a month, unless your start price is lower than the end price for the next green thumb, or higher for the next red thumb.  For example:

Green thumb ETFC at $1.50, close at $1.75, then green thumb again during same month.  If your second green thumb is at a price $1.75 or more, you do not get more accuracy.  If your second green thumb is at $1.70 or less, you get additional accuracy.

Note that these limitaions on accuracy ratings should only be preventing positive accuracy points.  If the pick goes into the red, it should hurt your accuracy, even if it does not qualify to help.

Report this comment
#26) On September 22, 2009 at 6:33 PM, TMFJake (52.53) wrote:

outoffocus:  the Hottest 5-Star Stock List problem is being corrected soon.  No more currently unratable stocks will appear on this list.

Report this comment
#27) On September 22, 2009 at 8:00 PM, TMFJake (52.53) wrote:

bigpeach: the Highest Stock Returns List on the Top Tens page should be corrected now.  Thanks!

Report this comment
#28) On September 22, 2009 at 11:09 PM, BigFatBEAR (29.24) wrote:

Jakila,

Great post, and I agree with you. It was weird seeing so much shake-up on the top 10 today. I also aggreed with your shout-outs and call-outs...

When I started this portfolio, I had been playing caps for around 8 months, and saw that the best investors were heavily bearish at the time. So, I basically just did some debt screens and red-thumbed every .ob and .pk I could find. This somehow earned me the Penny Lucky Charm, but it should be noted that I've never traded a penny stock with real money! Now BravoBevo, that guy deserves a penny charm...

It's only because I happened to short everything in June of last year that I'm highly rated. In fact, at one point I was rated 99.70, which I thought definitely over-stated my investing accumen. I've only been at this stuff for 1.5 years, and have learned 90% of my knowledge through TMF, and the other 10% through my friendship with GMX. I have yet to crack the cover of one of these classics.

Even with a healthy dose of cheesy activity, I'm now rapidly losing traction to a herd of bulls that are clamoring for the upper echelons, which I feel is fair.

The social investing sit over at kaching.com isn't without its faults, but it does have some more realistic ways of managing money and rating players...  sounds like TMFJake is upping the game, a little bit though!

Anyway, I always appreciate your thoughts and your untainted rating, both here and on kaching.

Report this comment
#29) On September 23, 2009 at 12:22 AM, Tastylunch (29.33) wrote:

Well I guess if we are going to get this radical it pays to strip the question to its core elements

1) What is the goal of CAPS?

- The Fool seems to intend to be a wisdom of the crowds-ish system that ranks stocks in single component. Unlike say Value Line there's no seperate timeliness, safety components. The star ratings are fairly arbitrary and yet also intended to be a comprehensive aggregate rating.

The Fool presumably can use proprietary internal metrics generated by CAPS for their research for their newsletters and mutual fund 

2) Is CAPS successful at it's intended purpose?

According that Harvard study it looks like it may be successful at beating the amrket (I'd say the sample size is still too small to be conclusive)but there also looks to be room for substantial improvement

3) What kind of game should CAPS be?

- The Fool has basically made it an analyst game. I agree with that. This makes sense given CAPS purpose, the Fool wants to remove unrelated factors to the star ratings such as portfolio weightin.  They also want to make ratings easy and binary so as to gather as much information ("intelligence" if you will)as possible.

The goal is a high volume of good quality decisions.

4) what's a players motivation for playing CAPS?

Perhaps it's greatest weakness is this factor.The only real reward stated reward is the player rating.

I think the majority of players think CAPS is a portfolio game and come here to find other good investors.The pitchine can lead one to think it is "investors helping investors" CAPS is really not about the players though but the ratings of stocks.

This means the scoring system becomes doubly important as it only affects the star ratings but is the power user's primary motivator in many cases.

But also becasue new players seem to use a player's score as a proxy for investor acumen however accurate/inaccurate reflection that may be.

5) How should a player be scored?

An interesting problem, because the way to measure the best analyst isn't as clearcut as measuring the "best investor". I'm not sure I have the answer here,

Since it's analyst game I do think you have to prize accuracy.That's how sell side analysts are judged on the street. It's not an investing game and it probbaly shouldnt be if it's about generating ratings. for stocks. Investors may not care what your accuracy is (just your returns) but analysts sure do. An analysts doens't care if a stock is shortable or illquid either as they aren't investing.

But also perhaps gaming ETFs and .OBs aren't super useful to star ratings due to accu farming (my personal belief is that 1 star ratings tend to more reliable on here than 5 since holding open green thumbs hurts less than a red gone bad) . So some restrictions there might be useful (how about making the minimum hold time  on .obs be 3 months? That ought to really cut down the gaming while still letting the community rate them)

Here's an idea, perhaps CAPS ought go full blown analyst and require players to pick price targets and hard time targets besides making a call. One of the weaknesses of the crrent system is that CAPS weights 5yr calls stronger than the others in start ratings but a lot of players don't pay attention to time frame on calls since there is no personal score incentive to do so.

In any event I do think there needs to be more "pain" on players who hold open losing picks. I have a green thumb open on FAZ  based off Trade that worked for three days (but not seven :(  )I'm getting roasted on but I have reason to close it othern than space.

I think changing CAPS player ratings to one year rolling ratings will keep players more honest and actively engaged with their picks.  As It will really cause the player to drop in the standings if all their current holdings blow.

anyway that's my rambling 2 cents.

Report this comment
#30) On September 23, 2009 at 12:31 AM, topsecret09 (37.78) wrote:

#14) On September 22, 2009 at 2:43 PM, TMFJake (96.49) wrote:

1.  An alternative ranking based solely on your "top picks."

2.  Implementation of a volume filter to replace the market cap and/or share price minimums.      Great Ideas.....  TS

Report this comment
#31) On September 23, 2009 at 9:27 AM, portefeuille (99.60) wrote:

I think that the score points a player receives for his "buy"/"sell" call on a certain stock/fund should be based on the price of the call/put option having that stock/fund as underlying and with maturity and strike price corresponding to the "time frame" and target price. The player receives s score points for his call where s = (p2 - p1) / p1 * 100, where p1 is the option price at the time the call is made and p2 is the option price at the time the call is ended ("times 100" simply because most people like "percentages"). The price of the options is determined by an improved Black-Scholes formula. Once the corresponding option is "in the money" the call is ended and the player is free to make a new call on that stock/fund. This simple system should take care of most of the "issues" of the current system.

The only "major" problem of this option based system is, that a Black-Scholes formula (even an improved one) has its weaknesses. One of them is that increased volatility at certain dates (earnings releases, clinical trial results, etc.) is not "incorporated properly". And for many stocks/funds there are no corresponding "real" options or the existing ones are too "illiquid" to be meaningful. A solution would be to give other players the chance to make a "counter-bid". So let's say I want to make a "buy" call on stock ABC. Then the system should give other players a chance to do a quick research (that would actually be a "20-minute delay" that would make some sense, hehe). The system would print something like "ABC,buy,2y,$100,option price $1 (implied volatility: 50)". Now if you think the implied volatility is "too low" since ABC is about to announce earnings the next day you can make a counter-bid, say something like "option price $1.5 (implied volatility: 70)". At the end of the "quick research&counter-bid" period the "highest bid" wins and that player gets to make the call. All other bidders are free to "join" at that "winning" option price. 

This bidding process needs some more thinking I guess, but it should not be too difficult to turn this into something "reasonable". It should be prevented that players sabotage the actions of other players but since the winner must make the call and the bidding process is anonymous the system should be pretty "sabotage resistant" ...

Report this comment
#32) On September 23, 2009 at 9:36 AM, portefeuille (99.60) wrote:

I think it is pretty obvious that the proposed system would give "reasonable" point scores for calls. The player rating system should of course be based on those score points. I think sorting the players by the sum of their score points should do. The stock rating system should be based on the calls and the player ratings. Also of interest might be the "raw data", say a figure showing the "strike/maturity combinations". Each of those points might be labeled by the player(s) who made that call, their player rating and the time(s) those calls were made.

Report this comment
#33) On September 23, 2009 at 9:46 AM, portefeuille (99.60) wrote:

... labeled by the player(s) who made that call, their player rating and the time(s) those calls were made.

... and showing the type of call ("buy" or "sell") of course. green and red maybe?-

Report this comment
#34) On September 23, 2009 at 9:47 AM, portefeuille (99.60) wrote:

There should be a "counter-bidding process" when the calls are ended as well.

Report this comment
#35) On September 23, 2009 at 9:56 AM, portefeuille (99.60) wrote:

Not sure how the suggestion made in comment #34 would work.

Report this comment
#36) On September 23, 2009 at 10:02 AM, portefeuille (99.60) wrote:

do a quick research

sorry for the lack of "sophistication" in my comments.

Report this comment
#37) On September 23, 2009 at 1:11 PM, Tastylunch (29.33) wrote:

portefeuille

The major weakness I see in what you propose Re; options scoring is that I doubt the majority of CAPS players would understand how that works.

Whatever system the Fool goews with needs to be readily accessible to the masses in order to attract and retain the number ofactive players they want.

 

Report this comment
#38) On September 23, 2009 at 1:29 PM, degaston (99.58) wrote:

What would I change .....

1. Make accuracy be just 10-20% of the score. 

2. Use dollar volume for the past 100 trading days instead of market cap as the cutoff for whether or not a stock can be picked or not. If a stock isn't averaging 1M+ dollars of volume daily then its not liquid enough to be meaningful. 

3. In calculating 1* to 5* ratings stop using picks of players who haven't made any picks' changes in the last 100 trading days. 

 4. Change the 1* to 5* rating system to be 1.0 to 5.9 with each of the 5 star levels divided into 10 deciles. 

 5. Retroactively calculate new player rankings and stock ratings based on the new rules. But continue to publish the legacy ratings for the next 12 months too. 

 6. It would be nice to see the player score chart go back to the beginning of CAPS in 2006. That's because I want to see how a player does during the bull cycles and bear cycles. IMHO the ones with steady growth through up/down are the ones who have the best prospects on doing well in the future. Just showing the last 6-9 months isn't very useful, especially right now.

7. If a player outperforms by 5% on closing a pick then they get a plus on their accuracy score. If its by 0.1% or 4.9% they get nothing. I'd like to suggest making a sliding scale between -5% and +5% on calculating accuracy scores using the following formula:  A = (Points+5)/10. Thus if a player closes at at -2 points they get an accuracy score of 30% for that pick. If they close at 4 points they get an accuracy score of 90%. If they close out below -5 points they get 0%. 

8. Plan to tweak/improve the rating system each 12 months so you can respond to feedback and system's effectiveness.

9. Most important of all - keep the system HONEST. If TMF ever tries to manipulate the CAPS scoring system for any hidden agenda it'll self-destruct. It's ok for TMF to make money on any models they learn by mining the data. Its TMF data and they have the right to mine it all they want to find patterns that actually do make money. But there can never be any excuse to go fudge/alter historical data for some hidden agenda. When a player opens/closes a pick then make sure the prices of that stock and the SP500 are accurate and never change this historical data EVER. 

 

 

Report this comment

Featured Broker Partners


Advertisement