Mar

12

One of the most valuable things I learned from the Chair is how not to do a study.

Let us summarize how to do a study. First define a pattern or event of some type. Then calculate the expected return subsequent to that event when the event happened. Then compare that return to the returns for all other non-event time periods. Do a t-test to establish significance at the 95% level.

That said the real problem is how can we insure ourselves against the possibility of biasing our study or otherwise completely messing up. the first thing that comes to mind is to never include data in your decision process that was not known at the time. For example Enron went bankrupt and then several years later after an audit the financial results were released showing that the original releases had been fraudulent. You cannot use the adjusted data based on the argument that it is the best data. Only the original data was known at the time so you must use that.

The same thing goes for price data. You have to use the prices that were known at the close if you are doing a buy at the close study. You cannot use retrospectively adjusted prices when the data is adjusted later than the supposed decision was made.

Always use tradeables. For example the S&P 500 index does not trade as an index. The S&P futures do and SPY does as well so one would use either of them as data for your study. The reason is that individual stocks can have stale quotes. Some of the smaller stocks in an index do not trade nearly as often as the larger caps. Thus the index can be behind the true position of the market. The tradeables trade and thus are subject to arbitrage that tends to keep them in line with the real market level.

This is a short list of things not to do. However it is representative of the fact that it is harder to learn what not to do than what to do. Other contributions would be welcome.

Victor Niederhoffer adds:

Always simulate what the chances were that your observed results were due to pure luck and take into account the path that your results would take and what that would have required of money management.

Consider the impact of retrospection on your results. The human mind is capable of ascertaining many regularities that occurred in the past, and is good at uncovering them in a study after the events occurred, but not very good at uncovering predictions based on new data that they are not already privy too. Never use range forecasts as they don't tell you whether you would have made or lost. Be aware of the difference between description and prediction, and statistical significance versus predictive distributions.

Never be overconfident. Do take account of the drift in your data, and the shape of the distributions you are drawing from. Mr. T, is not very good if only 2 or 3 observations removed from your sample would change the results.

To what extent are the regularities you believe you have uncovered been extant in the literature or the knowledge of shrewd fast moving traders. That changes things. What is the extent of regression bias in your results? 

Alston Mabry comments:

Something else, basically another riff on the Chair's comments: I find that statistics like means and correlations are, of course, useful, but they almost always hide important, idiosyncratic structure in the underlying data. In a sense, summary statistics are "intended" to do that, but I find it useful to unpack them and examine the structure in the data series, how the summary stats change over time, etc.

Anton Johnson writes: 

A couple of important things to consider.

Large changes in outcome resulting from small adjustments of a parameter is a sign of over-fitting and usually bodes badly for real-time results. Sometimes eliminating or finding a suitable replacement for the sensitive parameter will result in a more robust and usable model.

As a general rule, the number of parameters used in a study should be FAR fewer than the number of resulting trade signals.

Ken Drees adds:

Coach Bob Knight's new book The Power of Negative Thinking mentions "NO" being safer than yes. You can always more easily change a "no" into a "yes" versus the opposite–deciding to change your mind from positive to negative.

The gist of the book is to tamp down the uber positive thinking crowd–no, you can't do anything you want, no, you can't magically power your way to a fine end. PONT, Power of Negative Thinking is how Knight coached. He explains it that you must limit faults, limit mistakes–if we don't do these things then we have a chance to win. He keys on dealing with negatives to achieve a positive. He must have come across a lot of less disciplined approaches to coaching in order to come up with an against the grain type philosophy (PONT).

A lot of his points are probably already in the quiver of the sharpened spec. His hyper worried routines, careful study of the opponent, downplaying of good fortune and constant moving of yesterday's win into the rearview mirror broadens out into that persona you conjure when you think of him–that brooding face, those searching eyes–never smiling. The idea of "can't do it" was probably the most different from what we hear today–most are afraid to say "can't–that it means "I won't". Knight loves the honesty of a player saying I can't understand that assignment, or I can't push myself any farther. I would not recommend the book to cross over into speculation, but it's a quick read and there are more than some items to enjoy.

During it, I thought about player health in relation to speculating. I am my own coach. It's a luxury to have someone call your number and sit you down for a breather, to know you may need rest over more drill. How do I know that I am playing/ trading fatigued—only after a poor result? Knight seems to have the keen memory still in gear. There are some interesting stories about his games and Big 10 accomplishments.

Coach Knight will definitely tell you "No".  

Leo Jia writes: 

Very interesting, Ken. Thank you for sharing.

There seems to be some rationale in being positive. As I understand it, when one says "yes, I can do it" and envisions the actual doing, he actually plants a seed in his subconscious brain. The subconscious brain can be more powerful in many ways than the conscious. So planting a seed there is to use the additional powers of the brain, which are not accessible by the conscious mind normally, and thus increase one's chance of achieving a goal.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search