Jun

27

It’s been said that machine learning algorithms have no particular prediction power for the stock market, due to it’s intrinsic randomness and the all pervasive opportunity for curve fitting. (My very first inquiry here on the List, back in 2009, was about how one could avoid overfitting.)

Specially when dealing with short-term trading (day trading), it’s considered that nothing can really beat the random nature of the market. And I agree, to a certain point.

But my experience in recent years, particularly applying machine learning for the day trading of stock index futures, would suggest otherwise. There’s a frontier between stock market analysis and data science that can lead to predictive power using machine learning algorithms.

Perhaps the best advise is to use them for classification, not regression. And here we are in pattern analysis, so to speak. Classification algorithms can identify feature characteristics and recurrent patterns in an nonlinear dimensional space. So, with enough work on the subject, it’s possible to identify tradable patterns, (which in fact remain uncertain, as per the black box characteristic of certain algorithms), and to build a trading strategy around them.

My best experience so far has been with a QDA (Quadratic Discriminant Analysis) algorithm, with real trades for the past 15 months. Here’s the documentation link for the QDA algorithm, which is public knowledge as part of the Scikit-learn python library:

QDA algorithm

Laurel Kenner asks:

Newton, what role does human inspiration play in revising hypotheses in the scientific method that AI seeks to automate? Can an AI replicate out-of-line ideas, lucky errors, the contrarian stubbornness of human thought? Can it know what it is to add value?

Newton Linchen responds:

After 15+ years in the stock market, I went to grad school in computer science, to learn such algorithms. Before that, I had no idea how they worked, and as a trading strategies developer, my main concern was always to avoid curve fitting at all costs.

I’m not the best student, but from what I’ve learned, and applied in my work as an analyst, it became clear that we, who are foreign to this area, have much misconceptions about the nature and the role of AI. In fact, I think AI is a very bad name, as, most of the time, we are dealing just with approximation functions.

What machine learning algorithms excel, is to perceive relationships between the features, mapping those features in a hyperplane, (which is a confusing name to express that each feature is understood as a dimension itself in the dataset). The quest is not for to suppress human insight, knowledge and creativity, au contraire: is to use it in an orderly fashion as features for the algorithms to learn from. I believe Marcos Lopez de Prado has a body of work in this field, and clearly it is him who should be bringing this topic, not me.

In my experience, the algorithms learn patterns (that perhaps we couldn’t identify), and generate predictions. A whole other work begins when we decide what to do with those predictions (as in terms of time length of the trade, profit target, stop loss, stop-the-algorithm policy, etc). So, AI won’t substitute human knowledge and insight in the markets, but I believe it’s a precious tool for research.

Nils Poertner adds:

human intuition is underrated, isn't it - to make sense of the world sometimes? we are all geniuses in a way but don't see it that way.

The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honours the servant and has forgotten the gift.
- Albert Einstein

Paolo Pezzutti comments:

I have not worked with Qda. I am convinced that classifiers are more interesting from the trader's perspective than regressors. I have coded things using xgboost which is also quite popular just to understand the approach and the potential. There is a huge effort to do in order to identify and select features. Overfitting is an easy trap.

Leo Jia writes:

I prefer seeking entry-exit pairs to only predictive conditions. Overfiting is unavoidable. But there are techniques to largely eliminate them. Also, it's not enough to only seek the best performing (however one defines it) conditions based on the given data as paradigm is surely changing, so it's important to find lukewarm or even losing conditions that are to well perform sometimes in the future. In any regard, expecting machine learning to do a great job on timeseries financial data as it can do with, say, image detection is unrealistic due to many factors, notably the lack of training data and the paradigm changes. So other assisting techniques (which turn out essential) have to be used ulteriorly.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search