May
3
Perfect Foresight Isn’t Always Best, from Bill Rafter
May 3, 2009 |
On February 1st, Cassandra picks up the Wall Street Journal outside her door and discovers that it is the paper for March 1st, one month in the future. She does not know what to do with it, and no one believes her story, so she simply saves the paper. The next day, February 2nd, the paper dated March 2nd appears. This continues unabated. Come March 1st Cassandra checks the actual stock prices and sure enough they all match the prices in her originally-delivered paper. Likewise with all of the subsequent newspapers.
Soon Cassandra puts a PC to use and starts ranking stocks on their average 1-month growth rates based on this perfect knowledge. She then partitions her available capital into 21 tranches and each day purchases in equal values the 200 stocks of the Russell 3000 expected (actually "guaranteed") to do the best over the next 21 market days (1 calendar month). This partitioning eliminates any start-date bias. The returns she achieves are the best possible for a buy-and-hold strategy over that time frame. After all, it is the result of perfect look-ahead bias. Returns for this, the control group from 2001 through 2008, are a not-surprising 25.53 times her capital per year.
John B says, if you give a trader any system, no matter how good, in half an hour he will have modified it in an attempt to improve it. This is true even when perfect foresight is present, and Cassandra succumbs to her curiosity. You see, Cassandra is uncomfortable with the decision to buy-and-ignore. And she is correct to be uncomfortable. She has this constant stream of excellent information, albeit for later periods. It has to be worth something, according to information theory. And Cassandra's intuition is legendary.
Cassandra alters her trading in that she will now only hold her positions for 15 days instead of 21 days. She does this without knowing in advance what the prices will be 15 days after her purchases. When Cassie sells the portfolio on day 15, she will then purchase another 200 stocks based on their next 21-day forecast. Now of course, she is investing with sub-perfect knowledge, because she does not know the price of those stocks on day 15. She is using knowledge farther in the future to trade for a date shorter in the future. Intuitively it would seem unlikely that she could tinker her way to a better return than the perfect portfolio. But she can, and does so repeatedly. Strategy A beats Strategy B, where Strategy A consists of more-frequent forecasting and trading with sub-perfect knowledge, and Strategy B consists of less-frequent activity with perfect knowledge. The 15-day recast produces 39.52 times her capital per annum, about half-again better than the control.
Encouraged by the success of active investment management over a seemingly-unbeatable investment, Cassandra shortens up even more. She still forecasts 21 days ahead on the basis of the newspapers, but she only holds 10 days, or half the forecast period. This produces a whopping 138.53 times her capital per year. Recasting every 5 days produces 56.62 times her capital annually, less than 10 days, but greater than 15 days or the control period of 21 days.
Additionally, these experiences were not limited to monthly forecasting. Trying to further game the system, Cassandra subscribed to Investor's Business Daily, which arrived always a calendar quarter ahead. Her control group 63-day (quarterly) returns were 5 times capital each year. But reducing the holding period to 2 months increased it to 12 times, and reducing it to one month produced returns of 10 times. From 5 days out to the forecasted time in both the monthly and quarterly scenarios, the improvements were universal. Furthermore, extending the holding periods beyond the forecast period was universally worse. This is also true for different rate of return ranking processes. That is, it works identically whether one uses exact rates of change or regression rates of change.
Why?
There are two possible explanations for the increased profitability. The least obvious factor would be portfolio rebalancing of an entire portfolio every time new positions are taken. Indeed, rebalancing tremendously adds profitability and the more frequent the better. However in this study rebalancing was excluded for any unchanged holdings. In this study new positions would have had an equal value allocation, and existing positions were left alone to grow or decline in value, unchanged in size until liquidated. Thus with rebalancing absent, there can only be one explanation - that more frequent forecasting is more profitable, even better than some forms of perfect knowledge.
What About Risk?
Increasing profits without consideration of risk is foolhardy. It is worth noting as an aside that perfect knowledge does not always produce profits in every period. This is a long-only strategy in which Cassandra is purchasing the top-200 ranked stocks. There are some periods when virtually all stocks decline, or at least when the average return of the top-200 is negative. Thus we must consider the dark side of investing.
One of the distinct advantages of more frequent analysis and forecasting is that losses get cut earlier, providing damage control. Given the inclusion of risk into the equation, shortened time horizons become exceedingly obvious in the role of risk reduction. Recasting our 21-day forecast every 15 days produced an annual return of 39 times one's money, but at the risk of a 30 percent decline in capital. At 10 days the returns were 138 times, but with a 47 percent drawdown. At 5 days, the returns were 56 times, but the maximum drawdown was only 14 percent. Thus 5 days had a reward-to-risk ratio of 391, compared to 292 for 10 days and 127 for 15 days. The monotone progression makes the point. And it is confirmed in the quarterly data as well; the risk-adjusted rewards are improved by ever shorter forecasting periods.
What Are the Practical Applications?
Here we see a situation in which our investor has perfect future knowledge and yet she is absolutely advantaged by re-evaluating her decisions with increasing frequency. This is essentially a conflict of two mutually-exclusive ideas. On the one hand, perfect knowledge has to win, and indeed it will over that exact period. However it is also intuitive that more-frequent information would be beneficial, if one is willing to shorten the trading period. And here we have evidence of the latter's success over a perfectly-chosen portfolio. If active management can improve a portfolio chosen with perfect foresight, surely logic dictates its value on a sub-perfect portfolio. That would have to be proven, but any of those results would be dependent upon a particular investment or trading program. The beauty of this particular study is that it is program-independent. The clear lesson from this is that any financial advisor who tells you to buy an investment and ignore interim news or market action is repeating some time-honored advice that is clearly and profoundly wrong. The buy-and-ignore strategy only works if you or the advice provider is unable or unwilling to devote more time to your investment analysis. Clearly lack of perpetual investment diligence applies to many investors and advice providers, but the costs of that ignorance or intransigence are generally dismissed.
The purpose here is not to suggest a specific trading frequency, but only to suggest that the trading frequency should be a shorter number of days than the forecasting horizon. Over time there will certainly be sweet spots, and one should not assume those to be constant. Some traders will argue that they trade without making any forecasts. However, every trading decision is at least an implied forecast.
Technicians of course practice their craft with increasing analysis frequency, and they should gain comfort with these results. But some technicians also look at less frequent data, and this work should provoke caution in that regard.
Some fundamental analysts would argue that it is not possible to increase the frequency of analysis. Assuming the fundamental variable of Earnings (released quarterly), that would be true. However the Price/Earnings ratio is available daily, albeit based partly on a quarterly number. Greenblatt ("The Little Book That Beats the Market") uses only two ratios as inputs: Return on Capital and Earnings Yield, and produces daily recommendations. Likewise Haugen ("The Inefficient Stock Market") uses approximately 70 inputs (several of which are technical) very frequently despite the fact of some of their precursor variables are only available quarterly.
Furthermore, the rationale for frequent use of fundamental information is abetted by the blurring of the definitions as to what constitutes technical versus fundamental information. Something we have learned as a result of the recent market declines is that at least one of the major credit rating agencies (S & P) reevaluates its company ratings partly on the basis of a company's stock price performance. A big drop in the stock price might earn that company a downgraded rating. Thus daily priced-based activities (a technical factor) become part of the fundamentals of that stock.
What Prompted This Research?
With so many things to be studied, why spend time on such a seemingly esoteric idea? Well, it's not so esoteric. In our research we had attempted to look at longer time horizons, as we wanted to hold our positions longer. The investment community has sold its customers on the concept that more frequent analysis/increased trading must be bad for the investor. The government effectively limits more frequent trading for IRA accounts, although it is allowed for qualified pension plans. Consequently we started crippling our research to make it equal to "industry practice". Every time we did that we experienced declining results. That of course suggested that targeting separate time frames for forecasting and trading could be advantageous to any trading program.
Research Notes
To eliminate survivor bias our list of the Russell 3000 stocks from 2001 through 2008 consists of about 4200 stocks, the difference being those deceased, merged or otherwise eliminated. The constituents were obtained directly from Russell.
Dr. Rafter is President of Mathematical Investment Decisions, a quantitative research consultancy
Jim Sogi comments:
Absolutely fascinating. The reduction of risk seems to be a main factor as the rate of variance goes down faster than the decline in profits. Is there a sweet spot for maximum risk/reward, or would it go down to the vig as the timeframe shortens? An exercise for the reader?
Charles Pennington comments:
I know it's problematic to use the word "obvious" in mathematics, but isn't this obvious? The un-updated predictions are partly about the past, not the future, and to that extent they don't help you as much as the updated predictions would.
Bill Rafter responds:
Lots of things are obvious and still ignored. And sometimes seasoned professional traders do the exact opposite of what is obvious.
We learned quite a lot from that research and altered our trading process –with definite benefits. Specifically what we did was to have different time periods for our forecasts and trading, and that was not obvious to us at first.
Dr. Rafter is President of Mathematical Investment Decisions, a quantitative research consultancy
George Parkanyi writes:
I did a lot of research on re-balancing fixed names packaged into groups of six, trading on the relative price movement against each other over fixed time intervals. I settled on 4 months as "pretty good". The other major variable you need to consider besides the time frame is the volatility. The wider and faster the swings, the more you can compact your re-balancing time frame. The challenge is try to catch as many smaller interim fluctuations as possible but not to sell your winners too soon or to average down on your losers too soon when the stocks are in big moves.
In my research I ran a large batch of tests keeping the time frame constant but varying the volatility range of the randomly seeded stock simulator I developed. I ran a whole bunch of tests using my simple re-allocation algorithm at different volatility bands within which I allowed my "stocks" to fluctuate. These tests generated an average annual compounding increment over buy-and-hold, that, as you might expect, increased with volatility.
What was encouraging, is that the outperformance generated by the simulated data used in the periodic reallocations matched testing I did on real stock data. Theory was confirmed by reality. You can seriously beat indexing (over the long run) with this type of active re-allocation methodology. The trick is to re-allocate individual securities (index components) against each other, not whole asset classes or indices, which are already smoothed out and much less volatile because they are, by definition, averages.
I've published the re-balancing methodology on my blog if anyone's interested — it's not a long read, and easy to digest, but too long for an email or single blog post. The links to the 3 segments are the first three in the blogroll column under the My Portfolio heading.
Comments
Archives
- January 2026
- December 2025
- November 2025
- October 2025
- September 2025
- August 2025
- July 2025
- June 2025
- May 2025
- April 2025
- March 2025
- February 2025
- January 2025
- December 2024
- November 2024
- October 2024
- September 2024
- August 2024
- July 2024
- June 2024
- May 2024
- April 2024
- March 2024
- February 2024
- January 2024
- December 2023
- November 2023
- October 2023
- September 2023
- August 2023
- July 2023
- June 2023
- May 2023
- April 2023
- March 2023
- February 2023
- January 2023
- December 2022
- November 2022
- October 2022
- September 2022
- August 2022
- July 2022
- June 2022
- May 2022
- April 2022
- March 2022
- February 2022
- January 2022
- December 2021
- November 2021
- October 2021
- September 2021
- August 2021
- July 2021
- June 2021
- May 2021
- April 2021
- March 2021
- February 2021
- January 2021
- December 2020
- November 2020
- October 2020
- September 2020
- August 2020
- July 2020
- June 2020
- May 2020
- April 2020
- March 2020
- February 2020
- January 2020
- December 2019
- November 2019
- October 2019
- September 2019
- August 2019
- July 2019
- June 2019
- May 2019
- April 2019
- March 2019
- February 2019
- January 2019
- December 2018
- November 2018
- October 2018
- September 2018
- August 2018
- July 2018
- June 2018
- May 2018
- April 2018
- March 2018
- February 2018
- January 2018
- December 2017
- November 2017
- October 2017
- September 2017
- August 2017
- July 2017
- June 2017
- May 2017
- April 2017
- March 2017
- February 2017
- January 2017
- December 2016
- November 2016
- October 2016
- September 2016
- August 2016
- July 2016
- June 2016
- May 2016
- April 2016
- March 2016
- February 2016
- January 2016
- December 2015
- November 2015
- October 2015
- September 2015
- August 2015
- July 2015
- June 2015
- May 2015
- April 2015
- March 2015
- February 2015
- January 2015
- December 2014
- November 2014
- October 2014
- September 2014
- August 2014
- July 2014
- June 2014
- May 2014
- April 2014
- March 2014
- February 2014
- January 2014
- December 2013
- November 2013
- October 2013
- September 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- Older Archives
Resources & Links
- The Letters Prize
- Pre-2007 Victor Niederhoffer Posts
- Vic’s NYC Junto
- Reading List
- Programming in 60 Seconds
- The Objectivist Center
- Foundation for Economic Education
- Tigerchess
- Dick Sears' G.T. Index
- Pre-2007 Daily Speculations
- Laurel & Vics' Worldly Investor Articles