Feb

27

This has long been in my mind, recently put to text and published on my webpage. Happy to have any feedback.

Statistical hypothesis testing in trading strategy development

So What is statistical hypothesis testing? From Wikipedia: “A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis.”

Though the exact procedures are still not without debates, the general idea is: if a hypothesis can be confirmed as true or valid, it has to stand out from the random processes that apply to the same matter of the hypothesis.

So, it sounds very logical. For instance, if you want to prove that you have good skills at the football penalty kicks, you do say 100 kicks (without a goalkeeper) and compare your results with those of a thousand idiots. Say you scored 97 and rank the 11th among the thousand idiots, or the top 1.1%, then the committee confirms your skill, or in other words, they confirm that your claim of having good skills at the football penalty kicks as true or valid. That means that since you rank at the top 1.1% they trust that you truly have the skill and you will score similarly in future kicks.

Steve Ellison comments:

I am one of the "idiots", ha ha, who has found patterns that back-tested with a statistically significant edge, only to find they did not work very well when I actually traded them.

Part of the problem is that, with a threshold of p = 0.05, if you evaluate more than 20 hypotheses, you are likely to find some that show significance just by random chance. And this problem is multiplied in any study that involves multiple comparisons.

Furthermore, in an era of widespread machine learning, some institution is likely to find a pattern before you do, and may either arb the edge away or discover at its own expense there really is not an edge. David Aronson, who was on the Spec List for some years, discussed "data mining bias" in his 2007 book Evidence-Based Technical Analysis, when machine learning capabilities were in their infancy compared to today.

Big Al adds:

That appears to be a big problem with all sorts of research. It's easy to imagine a large, diverse group of researchers forming a sort of "meta-researcher" that is data snooping on multiple levels, even though the individual researchers are not aware of it.

As a trader, one must be skeptical and ideally have enough data to split it into a test dataset while reserving an out-of-sample data set for confirmation.

When I'm feeling more optimistic, I think of the market as layers of players, from very large down to minute (e.g., me), and most of the market bulk is the result of the bigger players making macro moves, which creates effects that smaller players can trade off of. The issue now is that, with AI technology, tens or even hundreds of billions of dollars can be deployed to black-box strategies that constantly search for smaller anomalies and patterns. But then the Palindrome's concept of reflexivity kicks in as all those black boxes create effects of their own.

Zubin Al Genubi writes:

I am looking at what factors causes price change and why and how. Model it to understand its function. Test with Monte Carlo. Its gives you a step ahead of price. Volatility clustering is a classic example. This what modern biologists do.

Jeffery Rollert responds:

My mental model is a sphere of sponge, suspended in space, with rain droplets hitting it everywhere all the time. It’s a variation of Al’s idea yet with more dimensions. One additional dimension is the age of the idea. As ideas are older, they are absorbed and move to the center where they have less impact on the balance. Market moves are represented when the sphere’s center of gravity shifts from the geometric center. Sort of plate tectonics but with a lot of plates.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search