Use access key #2 to skip to page content.

ikkyu2 (99.52)

CAPS 'accuracy', how calculated

Recs

1

May 12, 2007 – Comments (7)

I sent the following to the good Fools™ as a feature request:

With regard to 'accuracy', a number of players (wcwhiner prominent among them) have been pointing out that by sitting on picks like a hawk and 'harvesting' them as soon as they are worth +5.01, then re-picking them, the accuracy stats can be artificially inflated.

If I pick MGM to outperform once and the pick is worth +65, I get an accuracy point for 1 accurate pick.  The guy who picks MGM to outperform 13 times over the same period can get credited with 13 accurate picks, but he's added no more information to the CAPS universe than I have.

There's an easy - and statistically valid - way to fix this.  For every stock in the CAPS universe that a player has ever made a pick on, he is eligible for 1 "accuracy point."  If the total sum of all his picks' scores on that stock (current and ended) is greater than zero, he gets the point, as he was "right" (weighted) more than he was "wrong."  If they are not, he does not get it.

The number of accuracy points is divided by the total number of stocks the player has ever picked to get an accuracy percentage; that percentage can then form the basis for the current accuracy percentile calculation, same as now.

This would eliminate gaming the system, while not fundamentally changing the incentives to make accurate picks and to contribute valid information to the CAPS statisical universe.

7 Comments – Post Your Own

#1) On May 12, 2007 at 5:55 PM, ikkyu2 (99.52) wrote:

Or, to prevent gaming the system, the breakpoint could be the same as it is now:

Sum of all pick scores on a given stock, current and ended:

< 5.00 : no accuracy point for this stock
>= 5.00 : 1 accuracy point for this stock

Report this comment
#2) On May 12, 2007 at 6:02 PM, drnerdo2k2 (35.27) wrote:

ikkyu2,

While I think that's a better method for calculating accuracy, I think the real problem is the fact that accuracy is taken into account at all.  I think it's sort of reinforcing a bad habit that the vast majority of independent investors have, which is that they'd much rather deal with a portfolio that goes up in small, consistent increments than one that might not be as consistent on a day-to-day basis but generates the same amount or more money.  The scoring should reflect the real world, where no one cares about accuracy, just how much $$$ they made.  I know options traders who routinely average 10-20% accuracy, but due to the huge magnitude of their bets that pay off have absolutely thrashed the S&P since they began trading.  While I think very few people can actually deal with the level of stress that such a strategy generates (even when it works), I think it's a bad thing for the scoring system to encourage people to look at something that has no real value except as a psychological benchmark.

Anyways, if people let the scoring system from a website affect how they actually invest their real $$$ they have no business investing.  Hopefully most won't take it too seriously.  Personally I'd rather see the accuracy component of the formula replace with points/pick, but they've provided us with handy screeners that I can use to look for those people.

Best of luck,

Doc

Report this comment
#3) On May 14, 2007 at 2:24 PM, ikkyu2 (99.52) wrote:

Doc,

Thanks for your comments.

I'm not sure I agree with you that accuracy is worthless here in CAPS.  I take your larger point, which is that alpha (outperformance) is what matters and that the magnitude of overall alpha in terms of $$$ is what investors strive to accomplish.

The CAPS system, on the other hand, is designed to provide rankings to individual stocks.  This is a little bit at odds with the desires of the investor, in fact.  The investor, or the fund manager, desires in most cases to accumulate a basket of stocks that is going to outperform the broad market.  But CAPS desires to rate individual stocks.  If it's going to be successful at doing so, there's got to be some kind of incentive to be right, not only about the overall makeup of your basket, but about your individual picks.  The incentive in CAPS is 'accuracy.'  Without it, your 'individual' rankings are polluted by 'this stock shares characteristics of ZZZ, XYZ, and YYY corporations, and so belongs in the high-scoring basket along with them, even though it's not going to outperform.'

The problem with accuracy, as people are pointing out, is how gameable it is.  One wants accuracy in discovering alpha - both the investor and the CAPS designers want that - but the way it's set up now, you can game it by seeking beta, as wcwhiner has done.

I'm beginning to wonder if maybe the breakpoint should be set individually for every stock, with regard to that stock's beta, which is freely available via Yahoo's database.  That would be a way of weighting some of the beta out of the accuracy-determining system.  The problem even with this is that historical beta is no guarantee of future beta, and rewarding people who are able to predict future beta isn't particularly useful either to the investor or to the goals of CAPS.

Report this comment
#4) On May 16, 2007 at 3:06 PM, wcwhiner (97.17) wrote:

Per-stock "accuracy" was one of my first suggestions, once I realized management was hostile to changing its metrics.  I figured an incremental suggestion might go over better (it didn't).

Not only can I game the current system with beta, but the evidence suggests that CAPS players in the aggregate have responded to existing incentives the same way.  My estimate is that "CAPS beta" now is around 1.5, with an alpha statistically indistinguishable from 0.

Alpha is what we're after, whether as analysts or portfolio managers.  There are a lot of ways to estimate historical alphas, most of them pretty well-established by now.  Redesigning the system better to tease out and reward alpha should be pretty easy, given its current failings.

Please keep plugging away with your suggestions, and please encourage others to do the same.  The more players let TMF decisionmakers know how they feel, the better. 

Report this comment
#5) On May 17, 2007 at 9:00 PM, ikkyu2 (99.52) wrote:

wcwhiner, thanks for your comments.

To be fair, I do see the other side of the coin, and I suspect you see it too.  The CAPS system's scoring system is already somewhat mathematically sophisticated for the average reader, requiring an understanding of concepts such as percentile ranks.

It is an open secret, for example, that a CAPS All-Star is someone with a percentile rank greater than 80; it's perhaps a bit more obscure that only All-Stars' picks are used to figure a given stock's CAPS 5-star rating.  The bottom 80% of CAPS players' input is discarded as worthless for the purpose of establishing a CAPS rating.  Those of us who are old hands at statistics can perceive this at a glance but I don't think the average CAPS user can; I think they're challenged just to understand how the current scoring system works.

Basing a scoring system on Jensen's alpha or setting volatility-based individual breakpoints for individual stocks would make it even more difficult for CAPS players to figure out the system and I suspect it would probably drive a lot of them off.

And finally, having a system that's gameable by those who think more about it make it easier for a few TMF employees to stay in the top 10, which is probably important for TMF from a P.R. standpoint.  I mean, how would it look if it turned out that a bunch of random users were consistently smarter than TMF's best and brightest?  I respect TMF too much to want to see them take this kind of a hit to their business model.  (Give Web 2.0 another 10 years to mature and this won't be looked at any longer as something shameful, I predict.)

Report this comment
#6) On May 17, 2007 at 9:03 PM, ikkyu2 (99.52) wrote:

Incidentally, I've been pretty careful not to re-up any of my picks, because I don't want to distort my results that way.
I would like to believe that my average pick score of +3 is all Jensen's alpha!  That's certainly not so; but I try to maintain my portfolio as if I were seeking alpha.

Report this comment
#7) On May 22, 2007 at 8:00 PM, wcwhiner (97.17) wrote:

I have my quarrels with TMF management, but I don't believe they care that the current system helps keep a couple of them esconced in the top 10.  My guess, and it's always dangerous to guess motivations of people you don't know, is that they have a bad case of Not Invented Here.  TMF's name itself, after all, was the founder's response to less-than-complimentary input on his efforts.  I appreciate the ego it takes to fight the good fight against sustained negative input.  It's hard both to do that and to have the flexibility to question your own conclusions.  I know I usually don't do a good enough job at it myself, and I suspect TMF is making the same mistake here.

On the issue of complexity, I disagree, but perhaps not the way you think.  I agree that the vast bulk of players doesn't understand the current scoring system, or the likely intersections of market behavior, the law of large numbers and game strategy.  That's never going to change.  What I believe they can all understand is the headline: doing well or doing poorly, player rank, and out- or underperformance.  Replacing the current, facially simplistic system with one the guts of which have more moving parts but the output of which is more intuitive (especially if, you know, said new system actually works) would go over just swimmingly.  What the heck is "score" to someone new to CAPS, after all?  Express player scores as risk-adjusted value added per trading year.  Few will care what happens under the hood, and most newbies would find "+0.5%" a whole lot more approachable than "+100 score, 54.5% accuracy."

On your alpha, congrats.  I wouldn't take +3 as an estimate of alpha, though -- eyeballing your big winners and losers, I see a lot of volatile stocks.  My seat-of-the-pants guess is that overexposure to systematic risk would soak up a lot of what looks like alpha.

Report this comment

Featured Broker Partners


Advertisement