June 02, 2009
– Comments (32)
brand new research by bigpeach in comments #57,64 here!
this is relevant to the "caps" game.
Thank you bigpeach!
#50) On June 02, 2009 at 4:34 PM, portefeuille (99.99) wrote:
How does it make accuracy worthless? Two correct calls will weigh more heavily on the accuracy score than just one.
... because players like goldminingxpert improve their "accuracy" by ending calls that have scored a little over 5 points.
The probability of any call scoring more than 5 points at any time t > "7 days" is around 85%. (I will look that up or calculate it myself, but to get a feeling read this and this).
So a player that randomly chooses his calls and ends them as soon as they reach more than 5 score points (Please do not do that unless you know how to write a script that does it for you.) will have an accuracy of around 85% (again, I will check that) and around 0 score points (since he made random calls).
That is one way to explain why the "caps" game "accuracy" is garbage.
Do a google search to find others described in earlier blogs here.
Suppose we draw a line some distance from the origin of the walk. How many times will the random walk cross the line if permitted to continue walking forever? The following, perhaps surprising theorem is the answer: simple random walk on $\mathbb Z$ (the integers!) will cross every point an infinite number of times. This result has many names: the level-crossing phenomenon, recurrence or the gambler's ruin. The reason for the last name is as follows: if you are a gambler with a finite amount of money playing a fair game against a bank with an infinite amount of money, you will surely lose. The amount of money you have will perform a random walk, and it will almost surely, at some time, reach 0 and the game will be over.
(from the first wikipedia entry mentioned above)
(I know that this is not the way it goes in the stock market but still interesting and, as I said above, it gives you an idea ...)
#57) On June 02, 2009 at 6:02 PM, bigpeach (91.87) wrote:
The probability of any call scoring more than 5 points at any time t > "7 days" is around 85%.
Right in concept, here are some numbers. Using a random walk with no drift (sounds reasonable for a stock's performance relative to an index) and a daily standard deviation of 2%, we get a probability of being right in 7 days of 22%. 21 days (roughly the number of trading days in a month) give a probability of 50%, 63 days (one quarter) is 68%. We can interpret this to mean a person employing a "close as soon as the score is positive" strategy underperforms if their accuracy is less than 68% (under a 2% daily vol assumption).
If we increase daily vol to 5%, which is more likely these days for some of the higher beta picks people like on CAPS, the probability jumps to 81% over a quarter. Obviously if you hold until positive (the strategy of many of the top Fools) this is even higher.
So, if your strategy is to pick at random and close as soon as you have >5 points, your accuracy should be quite high, especially in a high vol environment like the past year. As porte said, you could write a computer script to get an 80% accuracy rating.
#64) On June 02, 2009 at 6:18 PM, bigpeach (91.87) wrote:
No problem, actually enjoyed doing it a bit, but let's not insult. I think I'll save this for reference whenever somebody uses the "I have a high score therefore I'm credible" argument. By the way, just ran my simulator for 300 days (just a big number to model the "hold until positive" strategy that a lot of people use). 91%! Of course that's with a 5% daily vol and one could argue with that assumption.
That's why I look at average score pick before I look at accuracy. Not that it's a definitive number --- but it tends to give one some insight on how much accuracy might be inflated. If I see 89% accuracy, a +7000 score, and a +2.3 average score pick, it's not nearly as impressive to me as 77% accuracy, a +12000 score, and a +31.0 average score pick. Yet, I think it's completely possible under CAPS' rating system that the former player would have a higher rating.
That's why I look at average score pick before I look at accuracy.
So do I. And I try to "annualise" it (in my head, so very preliminary data, but good enough for me), having looked at the "average period from start to end of the calls" (again a casual glance should do).
if you threw "average hold length" back into my avg pick score, I'd have a high one. Most of my big winners were held for the pick-minimum. My CHK pick which scored 60 points was held for all of 10 days. Boy did I make a killing on that option trade...
Are you really saying that you pick stocks at random, or just list the whole sp500, then over time you can achieve 85% accuracy with no further analysis?
Most people's accuracy on CAPS is less than 70% . Why is that? Because they are over-analyzing their picks?
Also, in real life you theory means that if you take 100 random stocks and hedge them with spy short. Then 85 stock should show gain against the spy?
I wish making money in the stock market was really that easy.
Yes, it is probably greater than 85%, see the analysis by bigpeach. There is more to come:
#71) On June 02, 2009 at 9:23 PM, bigpeach (91.87) wrote:
porte, I meant rank, of which accuracy is far too large a component. Score too is misleading because playing the CAPS game by opening and closing repeatedly would in real life generate trading costs and short term capital gains taxes, which would obviously erode that "score". Plus there is no feature for transaction volume. An obvious flaw. Players are rewarded for making as many picks as possible, rather than making quality picks.
In any case, I'm going to get off GMX's blog and write one on this subject. This is worthwhile information to be able to reference, especially considering how many people on this site follow the advice of the "Top Fools".
No, because they do not end all calls once they have scored more than 5 points.
Also, in real life you theory means that if you take 100 random stocks and hedge them with spy short. Then 85 stock should show gain against the spy?
Yes, around 85% maybe closer to 90%.
You make about as much money on the "winners" as you lose on the losers.
Please have a look at the discussion in comments #44-64 here (especially the "random walk" / bigpeach part!)
..So you would have a net score of about 0 with 85% accuracy... You will still need to combine good picks with this method to get high score AND high accuracy..
Another interesting scoring quirk to consider is that CAPS rewards holding onto (originally good) picks once they become mediocre investments...
For example: say I find a stock trading at 10 that should be worth 20, so I pick it.. A month later, the stock has doubled and is at my "fair value" estimate of 20, and my CAPS score is 100 for the pick (the S&P has stayed even for the month). Now, I have no opinion whether the stock will outperform the S&P going forward.. nonetheless, I still have an incentive to hold onto the pick! Assuming both my pick and the S&P goes up by 10% over the next year:
My pick % change = (22 - 10)/10 =+120%
S&P change = +10%
CAPS Pick Score = 120 - 10 = 110
...So I just made 10 points for performance equal to the S&P!!
In the real world of course, when presented with new bargain opportunities, you might take them, but in CAPS, assuming you got a good enough entry price, you might just want to hold onto what youve already got, because its gonna be a point scoring machine if it just matches the S&P
...So this is a scenario where points scored per pick is also misleading! Someone who went long a bunch of high beta stocks right at the bottom can just hold onto them until their business prospects seriously deteriorate
Yes, or luck, or multiple attempts (players) or making "sector bets" or playing "high beta, low beta", "market timing" or any combination of those and quite a few other things ...
You forgot to mention the case of the S&P 500 index and that stock losing 10% "over the next year".
stock: 10 -> 20 (+100%) -> 18 (+80%)
S&P 500 index: 1000 -> 1000 (0 %) -> 900 (-10%)
outperformance (i.e. score points, (what you call "CAPS Pick Score")): 0 -> 100 -> 90
so in the first case (that you described) you gain 10 more score points (100 -> 110).
In the second case (that I described) you lose 10 score points (100 -> 90).
This is true.. (and i had considered it) however, we are in a fiat money system and in the long run, the market will go up even with no real growth and even potentially in a case of negative real growth due to inflation.. In a system with no inherent inflationary pressure, I would agree that it is a good method.
This is the game CAPS created. Admittedly, there is a big flaw. A buddy that I had on here a while back has an accuracy of close to 70%, but if you look at the real return and not use the comparison of the S&P, they would be more or less at 10% accuracy. I don't think she plays here anymore, so I will link to their port. halfwayup
Here is the main argument for CAPS..... at least for people that truly invest and don't just play the game. I know you have issues with the accuracy rating and feel that it is overrated on CAPS and I agree. A fix would be to only count the pick accurate if it beats the S&P and is also a true gain in real life. If it beats the S&P, but is a negative return then it wouldn't count as being accurate. Leave the point system the same, but change the accuracy rating to this and it would be a completely different ball game on here.
The random walk theory works, but again it is all theory. Obviously it doesn't work for every port:)
Now I will be going back to read camistocks reply on that blog. I always enjoy his comments and blogs.
Dude, it was just a link to the article he wrote for a paper. What a let down. I read it already. Oh well. Later.
I quit rolling over my bull picks in the ETFs and Financials a few weeks ago and am holding close to 84% accuracy.
TMFElderhad and Bullishbabo kept rolling into new bull ETF positions (locking in accuracy) the last few weeks and now they are getting their accuracy reamed from the strategy today and with any continued sell-off. Instead of holding onto gains, they have added instant and growing losses.
Moral of the story - it only works IF your picks keep heading the right direction. If goldminingX can continue to make the correct picks, I don't see any problem with that, more power to him.
Your timing will remain of paramount concern to stay above 80% in accuracy.
I believe goldminingX is making a mistake and will run into problems trying to prove he can pick stocks on the upside with his new green thumb player. His temperment/style and stock picking value to me and the rest of CAPS is his successful focus on shorting and red-thumbs.
Motley Fool wants us to rate us many stocks as possible for their 5-star scoring system. They want tons of picks to comb through and create data points. Their goal is not to find the best stock picker, per say, from what I gather, but instead generate profitable stock rankings they can sell to others.
hi, welcome back tigerpack. has your strategy changed as of late? I somehow missed your daily updates.
I should open a "lounge" (no chat of course, just comments) like those "chartists".
Let's call it "chez portefeuille".
People tend to dislike my corrections, but actually I am doing it more for myself than for anything else. I no interest in any kind of "teaching". That said, it is "per se" ...
I no interest in any kind of "teaching".
I have no interest in any kind of "teaching".
Needless to say, I'm a big proponent of accuracy rating and compounding gains =)
portefeuille, thanks for the pdf you provided me on CCPs.
This is an interesting outcome:
Ironically, our model suggests that it is easier to justify the netting benefits of a central clearing counterparty dedicated to a particular class of derivatives after a CCP has already been set up for a different class of derivatives. In this sense, "one mistake justifies another." For example, the threshold size of the CDS market that justifies the netting benefits of a CDS-dedicated CCP is lowered once a significant fraction ofinterest-rate swaps are cleared. 11
The last part of the above comment is vague to me.
But overall I found the paper vague, i'll re-read it later. The notional value of what is being guaranteed must matter, and I didn't see it discussed anywhere.
Thanks for the shout out porte. I did some more work actually, and still intend to write it all up. Been really busy at work the tail end of this week so haven't had a chance, will do this weekend.
* * * REVISED LEADER BOARD STANDINGS * * *
Here is the new leader board equally weighting the three categories of total points, points per pick and accuracy.
I have talked about such a formula as being a better recipe for finding stock pickers--- I am taking ONLY the Top 20 page at the close Friday and reshuffling with my 3-tier formula:
okay, I am #4,5,11.
Did you "annualise" the "points per pick" somehow?
TigerPack: You should try that using score and see what happens, it would be interesting.
sorting by score rather then re shuffle.
There's also something interesting I've done before. I wrote a python script which would calculate a players actual return provided they were allowed to trade and compound those gains.
I assumed everyone started with 200 units of capital. In the beginning a pick was a unit of capital used. If you had 190 picks at start without having closed a single pick, then you would have 10 units of capital remaining. If you sold that pick would either add to or reduce your capital base. So assuming you had one pick that doubled and you sold it, then you would now have 12 units of capital. Your next pick would carry the weight 12/11. 12 is the amount of "capital you can work with" the 11 is the number of picks remaining before hitting the max at 200.
It was very interesting to see the true return for a given player ignoring accuracy.
Also the underperform score is wieghted wrong here. When we talk about the market we invest long in S&P, we never short it and call that long term investing. So the score for calling underperform using the difference between your pick and S&P does not make sense to me. But here we say is it going to underperform S&P not make money for you. Your pick is down 50% and so is the S&P your score contribution 0. Because of this the results in the paper don't quite make sense. But in real money you made a bundle.
now about the paper, the interesting thing about caps is not the score on the stock to create a prediction but instead the network effect of people in the community. That's why beginners are able to do pretty well. Some just copy whats already going on by the more highly ranked people. But the other thing is that caps was better in the beginning because the system was seeded by experts: the TMFer's. They missed the boat on that. So this is some sense is a dephi system too (the team of experts system). They ignored the evolution of the system quality over time. The dummies are just adding noise to the system which means the great players need to be much more highly weighted.
This noise in the decision making process must be deconvolved from the data so to speak.
This is what the paper should discuss. I don't have the time to muck through it or write my own, but perhaps I can help with some ideas if you want to. Also my math is really rusty at that level at this point.
Wow! feel free to publish the results here! how did I do? (half-joking, half-interested, half-scared (my math is not that rusty, I hope!)).
We should write our own paper. I have the feeling that russiangambit (see comment #8 here), anchak, you, I and some others would make good co-authors. And we let russiangambit choose one of his "half-crazy genuises" from the Moscow State University as co-author to give it some extra credibility. Moscow State vs. Harvard/Yale. I love that idea!
An "easy" new calculation that we could put in our rating model that would help estimate real world returns would be:
Days each pick is open!
I would rather follow someone that averages a 10% gain over 30 days vs. one that is created over 90 days, for example.
CAPS already figures the start date and end date for each pick; they easily calculate the average number of days each pick is open for all players with their database, without much effort.
If we add this category to my 3-tier model, to make 4 categories of rankings, UltraLong, portefeuille, bullishbabo & anticitrade would move up considerably on the leader board as they have been around just 6 months or less vs. the old school (and fading) super bears. Most of the bears earned the same number of total points, with a greater number of picks, over a much longer period of time.
This is all going to look so funny in 6 months. Us old school (and fading bears). "Stock prices have reached what looks like a permanently high plateau." -- Irving Fisher 1929.
"Moscow State vs. Harvard/Yale"
That wouldn't even be a fair competition...
tmfjake writes that "the staff" is thinking about changing the way accuracy is calculated in the "caps" game (see comment #42 here) and goldminingxpert has proposed this (see comment #43 of that post):
There's no need whatsoever to change accuracy, but if you do change it, change it to you get one accuracy check per stock up or down... so if my cumulative 12 picks on SKF have scored positively (they have) I am 1 for 1 accurate. If my 12 picks scored net negative, I am unaccurate on 1 out of 1 pick. That solves most of the "non-problem" that the CAPS bull community has.
I have calculated this new accuracy for portefeuille:
ticker "aggregate score points"
DNDN 580.98GMK 196.91HIG 186.98CX 158.99DB 146.75SLM 145.52RMBS 139.39BX 138.47ASH 131.02BCS 125.68CIEN 124.99ACGY 122.78ARNA 119.58FIG 119.55ALD 115.69ACH 112.48F 108.41SA 104.61SNDK 98.37TCK 94.48HUN 91.91CS 91.46JASO 89.58AFFX 88.70IBN 86.92BRCD 86.51NG 85.94AWC 83.31AAUK 83.18S 82.62MQBKY.PK 81.38CENX 80.16PCX 76.40SD 73.64STX 73.54MOT 73.00MCO 72.95MT 72.36NVAX 72.12NILSY.PK 68.36LUKOY.PK 67.93MTL 66.61HGSI 66.34ENZN 66.22ING 66.05ATPG 65.04RSX 64.75UBS 64.56CYPB 64.54IPI 64.35VIGN 64.29CCJ 63.61X 62.96WDC 62.44JAVA 61.93YHOO 61.56AMZN 60.67EXPE 60.63SVA 60.15LDK 60.10SLT 59.74KLAC 59.39AUY 59.31MBT 58.59RIMM 58.19RBS 58.15BK 57.03WYNN 56.97GMGMQ.PK 56.10OGZPY.PK 56.06MIDD 55.20PQ 54.23AIXG 54.13ADM 53.72AYR 52.56SNO 51.94TMRK 51.92CSR 51.85AHBIF.PK 51.74GERN 51.10NDAQ 50.72PKX 50.71LIHR 50.14
NITE 2.84IWC 2.73BRK-A 2.62MPEL 2.53CTCH 2.51DUG 1.73SNY 1.67DXD 1.34QCOR 1.33WRES 1.21SKS 0.51WYE 0.49ALTH 0.28HPQ 0.21ILMN -0.21MTU -0.29PSQ -0.94LXP -1.28AMR -1.34XLF -1.60AIB -1.66NVS -1.84BIIB -1.93GSG -2.40EWJ -2.51BMY -2.64AMGN -2.68PCCWY.PK -2.79COP -2.99DDG -3.00FMCN -3.44FRMSF.PK -3.46FDM -3.59LSR -4.39JNJ -4.58WNR -5.17QLTI -5.63SGP -5.66PZI -5.79LUV -5.90SEF -6.16GE -6.34VPHM -6.46USB -7.39DT -7.63GILD -7.73NOVL -8.28PFE -8.58WRI -9.66GENZ -9.92MSO -9.96APL -10.18LLY -10.35TWM -10.78SKF -10.83JBLU -10.90BAIRY.PK -11.12MIC -11.45MTW -12.09ALNY -12.34KFT -13.01UNG -13.42TWI -13.86MFG -13.92FAZ -14.29EEV -14.40MWN -14.43SRS -14.73PGNX -15.10TZA -15.19TTWO -15.26ADPT -16.03SMN -16.24NTDOY.PK -16.56GHDX -16.89CEPH -18.14FXP -18.62EDZ -20.52KHD -22.84CDE -26.99DAL -33.09AFAM -34.61C -37.14GW.DL -38.20TSCM -40.44ANDS -46.53CREL -50.51ALXA -52.23OMEX -54.58SQNM -67.28EK -88.81
I have positive "aggregate score points" for 397/474 stocks or ETFs that have been objects of the calls I have made so far.
397/474 = ca. 0.838 -> ca. 83.8% accuracy_new.
A small observation: I think of those 77/474 losers (stocks or ETFs that currently "show" negative "aggregate score points") some still have the potential to recover (I would of course need to make a new call on those that are not objects of currently "active" calls ...). Even if all recovered (very unlikely) there will probably be a few moving to the "negative territory" from the "plus side". Actually I am quite "satisfied" with the results stated in comment #29 above.
a feature that produces a list like the one in comment #29 above.
************************************************************************************a guide to my blog posts can be found in the comment section to this post(should be or should be close to the last comment) ************************************************************************************