Use access key #2 to skip to page content.

Scoring System is screwing up the 5-star stock ratings



March 20, 2009 – Comments (9)

Because of the current scoring system which gives an important weighting to the accuracy of ones picks, many allstar players have found that it is in their interest to never close their incorrect picks.

 Meanwhile players who have correctly guessed whether or not a stock will outperform close their picks quickly to farm accuracy points.

This results in star ratings for stocks which do not accurately reflect players perception of how they will perform in the future.

Caps should change the scoring system to a purely score based system so that there is no longer an incentive for players to keep incorrect picks open.

9 Comments – Post Your Own

#1) On March 20, 2009 at 12:24 PM, EverydayInvestor (< 20) wrote:

I prefer a metric that we can call the "Goode Ratio" which is points per pick per year. It would do everything you want while not penalizing newer players who haven't had the time to bank 10k points.

Report this comment
#2) On March 20, 2009 at 12:52 PM, RonChapmanJr (30.14) wrote:

Life is tough.

Report this comment
#3) On March 20, 2009 at 1:30 PM, chk999 (99.96) wrote:

When I coatailed on DRYS, I ended with a score of -1000 on it at one point. But my opinion on the company never changed and I was able to close it out for plus 13 and change. Just because a pick goes against you initially doesn't mean it was wrong.

Report this comment
#4) On March 20, 2009 at 1:37 PM, TDRH (96.52) wrote:

"Goode Ratio" is a good idea.

As far as the accuracy though, your picks on effect the stock rating if they are within the timeframe.    Most of mine are entered with 3 weeks.  Once the 3 weeks expires it does not influence the rating.  

One factor that is given no weight is the average pick score.    I would propose that the Fool divide the 25% weight of the accuracy between accuracy and average pick score ranking.   This would knock me out of the top 20,  but it would reward those who do not repeatedly close stocks with +5 gains and bank accuracy.  

Report this comment
#5) On March 20, 2009 at 1:51 PM, CMFEldrehad (99.99) wrote:

"Goodie Ratio" is actually a bad idea in my opinion.

Imagine, for a minute, what the CAPS pick lists of the Top Ten would look like under such a formula.

They'd all be very, very small - which, in my view, would run directly counter to the overarching goal of creating the best possible stock rating system based on community intelligence.

Average pick score as a metric carries with it the same inherent problem - namely that it encourages smaller pick lists.

After all, wouldn't we want the system to extract more infomation from the participants, especially the best and brightest ones, not less?

I actually participated in a lot of very lengthy discussions with the CAPS team here at the Fool on these very issues.  Make no mistake, I think there are good arguments to be made on both sides - and while we may disagree in the end, I can tell you that the CAPS team very seriously considered these very issues in developing/implementing the current player ranking system.

-Russell (a.k.a. TMFEldrehad)

Report this comment
#6) On March 20, 2009 at 2:43 PM, portefeuille (98.93) wrote:

The scoring system should take into account the volatility of the stock/fund a player rates.

It is only a matter of time until the top10 list features some players using by far more dubious strategies than the current top 10.

Consider the following strategy:

Of the stocks/(leveraged) ETFs ratable in the caps game give an outperformance rating to the 200 with the highest volatility.

Do this every day creating new caps players.

Wait for a "sharp" stock market rally. The "rising tide lifts all boats" effect will do the trick.

A sharp rise in the S&P 500 (of the not entirely unreasonable 30% in 4 weeks kind) should give you somewhere around 90% accuracy and 10000 score points leaving you in the top 10 of the caps players.

Report this comment
#7) On March 20, 2009 at 2:48 PM, portefeuille (98.93) wrote:

... another scenario is given here (comment #11)

Report this comment
#8) On March 20, 2009 at 3:16 PM, EverydayInvestor (< 20) wrote:

"points per pick per year", or as I humbly call it, the "Goode Ratio", would give high scores to many CAPS allstars. It would not punish the current practice of banking scores of 5+ but would not reward it either (a lower score would be compensated for by a shorter time period). So if I score 10 points by shorting AIG in one week, I would receive 521 Goode Points, as I have made a lot of points in a short period of time. A call on VTV to outperform that garners me 5 points in three years would give me only 1.7 Goode Points. 

Hmm, actually this system would reward short-term picks too much. So using it alone wouldn't work. Maybe combine it with total points. 

The problem with accuracy is that whether a pick is -10 points or -500 does not matter, same with whether it is +0.1 or + 4.9. Using some form of average per pick would be better and more logical. Really, is a -1 really significantly different from a +1? To me small losses and small gains are irrelevant.

Says the guy who is best at gaming accuracy.

Report this comment
#9) On March 25, 2009 at 12:06 PM, CMFEldrehad (99.99) wrote:

In a nutshell, the "Goodie Ratio" or anything that looks like points per pick per day is, in effect, a portfolio simulation and really no different from all of the other portfolio simulation games that I've seen elsewhere on the net.

And I think CAPS is vastly superior to them.

The players that 'win' in these types of portfolio simulation games are invariably the ones that have least diversified, highest risk portfolios that they can find.  While CAPS isn't necessarily immune to this kind of approach, the additive nature of the scoring system (more correct picks = a better score) and, believe it or not, the way the accuracy system works (being 70% accurate over 50 picks counts more than being 70% accurate over 10), actually work to lessen the impact of this kind of 'gaming'.

Regarding average pick as a metric:  imagine a participant sitting on a current average pick score of 10 who is deciding whether or not to make a new pick.  That new pick will, by definition, start out with a score of zero and immediately drop that players average pick score and standing in the rankings.

A system designed to gather community intelligence cannot have, in my view, a scoring system that has a built-in disincentive that discourages participants from providing the system new information and intelligence.  Using average pick score as a metric would carry with it this very disincentive.

I openly admit that the current CAPS player ranking system isn't perfect and that there is plenty of room for honest disagreement here.  I will also say, though, that in my opinion, of all of the different player ranking methodologies that have been suggested, while not perfect, I like the current one the best.


Russell (a.k.a. TMFEldrehad)

Report this comment

Featured Broker Partners