Apr

13

This is a topic that keeps appearing when people talk about probability. I don't seem to have a good intuition for it. Is the stock market with memory or without memory? Why? What would be your intuitive explanation of what memory is?

From Memorylessness:

In probability and statistics, memorylessness is a property of certain probability distributions. It usually refers to the cases when the distribution of a "waiting time" until a certain event occurs does not depend on how much time has elapsed already. To model memoryless situations accurately, we must constantly 'forget' which state the system is in: the probabilities would not be influenced by the history of the process.

Only two kinds of distributions are memoryless: geometric distributions of non-negative integers and the exponential distributions of non-negative real numbers.

Humbert H. responds:

Of course it's not completely memoryless otherwise there would be no point to any spec of this list trying to beat the market. It's ALMOST memoryless, and that's why it's hard to beat, but there are still some irregularities, like days of the week, month, season, reaction to events, like increased volatility following a big change. It would have a lot more memory if people didn't try to take advantage of the irregularities, because market participants have emotions and also information doesn't spread instantaneously even in this day and age.

Eric Lindell comments:

Blackjack is with memory, provided the number of decks is finite. As you play with more and more decks, the game becomes less memory-dependent. A small player in a huge market makes trades that are less memory-dependent than a big player's trades. The bigger the portion of the total market a trader trades, the more memory-dependent it becomes.

Wikipedia's discussion of a memoryless probability distribution refers to a poisson process. The time before the next car arrives at a toll booth doesn't depend on the time since the last car arrived — provided the cars' arrivals are truly random. This would NOT be the case with a nonrandom distribution, as when more cars arrive per minute during rush hour.

Zubin Al Genubi writes:

A normal distribution of a series of events, indicates that the events are independent of each other, in that the occurrence of one does not affect the probability of another. Of course the market has memory and emotion. We are looking for the regularities to trade that are not random with a high degree of confidence.

Larry Williams agrees:

Amen! People react in similar fashion to events and those reactions create patterns. Plus, there are unique time elements to many markets; jewelry is mostly sold at Christmas, hogs live and die in 18 months etc.

Penny Brown adds:

Investors who suffer a big, sudden decline in a stock remember it. Often they vow to hold on until they are made "whole". This can cause a stock to sell off as it approaches that spot. But if the stock clears this area, the weak hands are gone, and the stock can move up sharply.

Big Al suggests:

For further study, re the quality of "memoryless" and possible applications:

Markov chain

Hidden Markov model

Also, Vic has referred to Markov processes relating to the market calendar at the top of this site.

Mar

30

From The Mind of Bill JamesThe Mind of Bill James by Scott Gray, which I've been reading:

All things in baseball tend to return to their previous form. A team whose record improves one year will tend to decline the following year, and vice versa. In 1980, for example, only five of the twenty-six teams moved in the same direction in which they moved in 1979. It also applies to individual players. Bill found a way to express not merely the statistical principle of regression to the mean, but also what he called the 70/50 rule. Seventy percent of teams that decline in one year will improve the next; 70 percent of teams that improve will decline; and in all cases the amount of rise or fall is about 50 percent, so that a team twenty games over .500 one year would be ten games over .500 the next. (The percentages are much different for very big or very small improvements and declines.) “These were not things that I had expected to find,” Bill wrote. “Weaned on the notion of ‘momentum’ since childhood, I had expected a team which won eighty-three games one year and eighty-seven the next to continue to improve, to move on to ninety; instead, they consistently relapsed. Half-expecting to find that the rich grow richer and the poor grow poorer, I found instead that the rich and the poor converged on a common target at an alarming rate of speed.

It also applies to individual players. Bill found a way to express not merely the statistical principle of regression to the mean, but also what he called the 70/50 rule. Seventy percent of teams that decline in one year will improve the next; 70 percent of teams that improve will decline; and in all cases the amount of rise or fall is about 50.

Steve Ellison writes:

From my experience, I think the S&P 500 is less mean reverting in 1- to 2-week timeframes now than it was in the mid-2010s. On the other hand, the presidential election cycle pattern has been spot on since the beginning of bearish midterm election year 2022.

Eric Lindell comments:

These data pertain to relative performance — eg, a team's record relative to other teams. For absolute gauges — like how much I weigh on a diet, there's no such reversion to mean.

Kim Zussman responds:

Funny stuff! For asset managers everything is relative. For their customers (without yachts), it is absolutely absolute.

Where Are the Customers' Yachts?: or A Good Hard Look at Wall Street

Humbert H. agrees:

And that’s why people should not entrust their assets to asset managers, unless these people suffer from some sort of emotional instability and can’t handle losses without some stranger pretending to care.

Dec

17

is as good/bad for the economy and stock mkts (as leading econ) as being sober is for the alcoholic. credit mkts will not like deflation.

Eric Lindell responds:

deflation poses the same problem as inflation — introducing random noise into the price system. With stable prices, producers know that a price increase signals increased demand for a product. Von Mises compares inflation to running someone over and deflation to running him over backward. January 2014 was the only deflationary month in recent years. Deflation as cure for inflation is bad mathematics.

H. Humbert asks:

But there is no deflation. Other than the Walmart CEO talking about some possible food deflation (which is not deflation overall) who has any original information that deflation is likely?

Eric Lindell responds:

Re MV = PQ, deflation would accompany decreased money supply/velocity or in increased output.

H. Humbert replies:

That's true, but not in itself predictive. Will any of these things occur and be strong enough to matter? All that can be said now is that there seems to be some evidence of disinflation, not deflation. I'll believe deflation overall when I'll see it.

William Huggins writes:

the reason we aren't likely to see it is the totem power of irving fisher who taught generations of american economists that deflation led to an activity-suppressing feedback loop. far be it from me to opine on the well-regarded analysis of a seminal thinker (for several other reasons), but it may be worth noting the source of this deeply held belief. as a result of fisher's expert authority, particularly among other influential economists like friedman, the view is profoundly held by today's cadre of central bankers whose playbook warns them that deflation will lead to the great depression pt 2.

historically, the US had a notable 20 year run of deflation in the late 19th century and the economy at the time was soaring. (very) reasonable arguments can be made for confounding factors like mass land seizures, new tech, reconstruction, immigration, etc at the same time but to bring it back to the basic monetarist framework (assuming stable V during the period), the economy could have simply been expanding faster than the money supply. the big difference with today is that the money supply has been untethered from physical constraint. combining (potentially) limitless quantity with a dread of not having enough pretty much assures the outcome.

interesting question arises when one thinks about palindrome's reflexivity theory, where systematic incorrect beliefs start to create new (unsustainable) realities that seem to defy physics and then burst suddenly. are the CBs doing enough to trim their BS and get the money supply under control? (M2 drifting back to that 6% annual growth since the 90s) but will the fear of deflation drive us too far in the other direction?

H. Humbert comments:

It's an interesting coincidence that the belief that deflation is bad for highly technical economic reasons that have nothing to do with unsustainable money printing, coincided with inflation being useful when said money printing occurred.

What Irving Fisher was evidently saying was that deflation is bad because it suppresses economic activity through some sort of a feedback loop. I think the deranged animals that advocate (or justify or fight any attempts to control) the kind of deficit spending that we have given the debt that we have don't like deflation because it would cause them to have to stop the spending a couple of years sooner than otherwise, hence they would lose their hold on power that much sooner, and that's all that they care about. Irving Fisher is described thusly in his wiki page:

Irving Fisher (February 27, 1867 – April 29, 1947) was an American economist, statistician, inventor, eugenicist and progressive social campaigner. He was one of the earliest American neoclassical economists, though his later work on debt deflation has been embraced by the post-Keynesian school. Joseph Schumpeter described him as "the greatest economist the United States has ever produced", an assessment later repeated by James Tobin and Milton Friedman.

So it's an interesting coincidence that the some progressive social campaigner economist found through his incredibly insightful, but totally politically unmotivated, theoretical work a formula that the animals need to stay in power.

William Huggins responds:

exactly - they will either inflate it away, or at some point engage in a selective default. that said, societies can go on self-financing for a very long time (japan) as much of that interest is being put right back in the pockets of americans. its not like the wealth is being disintegrated, its simply being moved around. i have no idea how to gauge the limit.

i wouldn't be so quick to dismiss Fisher's work simply because you dislike "animals" who are actually your fellow countrymen whom you disagree with (do americans really hate one another so much? is there another reflexive breakpoint that's much more important to watch for?).

my point was that he was very wrong about the 1929 crash and I believe his losses must have set a terrifying fear upon him when the markets didn't bounce back. hence deflation as his bete noir, not some silly "convenience" for politicians who weren't even a dirty twinkle when we wrote. the issue is inherited wisdom being unbalanced, not conspiracy most foul.

H. Humbert replies:

You may very well be right about his motivation, I just found it interesting. I hate inflation because it's unfair to people who are good, who behave according to what I consider to be good moral principles. It also hurts many who are weak, whether through no fault of their own or otherwise. But those I call animals talk about any feeble attempt to restore sanity to the budget as an attempt to simply stop the government from functioning, just because those who attempt it are somehow motivated by evil intent. Lying to keep power while destroying the country is despicable behavior.

Oct

9

For the military guys here- does remote viewing work? friend of mine - a statistician - who was tangentially involved decades ago- said what is striking: "those who didn't believe in it - scored worse than chance". Can imagine that.

I go with the notion it may work in rare cases - but when it comes to forecasting mkts - one may run into many new challenges. probably takes time and would require years of training. not exact science anyway. could help with overall intuition perhaps.

Alex Castaldo is skeptical:

"those who didn't believe in it - scored worse than chance".

Trying to salvage something from a negative experimental result. Reminds me of "Well, our anticancer drug failed in a large sample test, but it seemed to work for left handed women between 65 and 75 years of age. That's very promising". Shifting the analysis to a question other than what was asked.

Nils Poertner responds:

for trading (or life in general) - it is good to be skeptical- and don't believe anything that comes along. on the other hand, one wants to keep the option of some (pleasant) surprises that one does not know everything. Controlled RV was used by the Military to my knowledge. that itself is a hint it may work.

Eric Lindell asks:

were these controlled experiments where either the viewer or viewed were in a faraday cage? Personally, I think there are two possible outcomes statistically: chance and not chance.

I'd like to see a rigorous study of remote viewing by those who don't believe in it — with faraday and standard scientific controls. I'd be surprised if it held up. You would need an objective measure of similarity of appearance between viewed and vision — which itself would be hard to gauge — statistically or even anecdotally. The faraday control especially is key to identifying the question itself — let alone its answer.

Humbert H. writes:

I've seen at least two Sci-Fi type movies where the remote viewer is tortured by all the evil he can see to the point of not being able to live on. I would say there are enough people in this world who wouldn't be troubled by seeing evil if they can become really rich, so I would say there is no real evidence of statistically significant remote viewing.

Steve Ellison comments:

There is a huge problem in academia, where the paradigm is "publish or perish", of research that can't be replicated. A 1940 study by Rhine and Pratt that found evidence of extrasensory perception was the original poster child for this problem. A big part of the problem is the traditional significance cutoff of p = 0.05. That's a reasonable starting point, but when thousands of researchers are working at any moment, 5% of their studies will reject the null hypothesis purely by chance. It adds up to a lot of non-replicability.

I have often thought that an advantage for those of us who are scholars of the market is that we don't have any pressure to publish and hence don't need to force dubious findings into practice. Instead of a pat on the back for being published, we get a cruel but not unusual form of "capital punishment" if our backtests can't be replicated in the market.

Anders Hallen actually finds research for critique:

Stock Market Prediction Using Associative Remote Viewing by Inexperienced Remote Viewers

Sep

18

remember the hype about Chat GPT some weeks /months ago? def for trading /investing - I doubt using that or any other program will help to master time ahead - prob a recipe for disaster at the end.

Peter Ringel writes:

I am still hyped! Hyped for boost in efficiency of the economy via AI. Not hyped for AI-trading systems! So far the training data set seem too small for AI - trading, thankfully. Together with what the Senator and others posted here: humans still beat skynet. Yet, I like to remind myself every day: the bastards are coming.

Hernan Avella responds:

So far the training data set seem too small for AI - trading , thankfully.

How do you figure this? Each trading day probably produces more than 100's million rows between trades and quote updates for all levels and exchanges, if you include futures, equities. I don't think lack of data is the issue here.

Peter Ringel replies:

I know even less about AI-coding, than about trading-coding. So everything is based on perceived experts. Thankfully, so far they are pessimistic.

Hernan Avella continues:

So everything is based on perceived experts.

The set of experts in ML-DL is very small, and the set of experts in trading is also small. I imagine the intersection is even smaller and more importantly, secretive. My suspicion is that the training set is more than enough, but the problem of ergodicity and stationarity (lack of) of the ever evolving competition are the culprit.

Peter Ringel responds:

I hope, you are wrong with this. But at some point you will be not. I speculate, that the "small" existing universe of trading history data + some sort of data - > model on human psychology - will be enough - will make us traders obsolete.

Peter Saint-Andre writes:

In my limited, non-trading experience with LLMs, I've found that their output reflects conventional wisdom. That might leave plenty of room for creative strategies outside the mainstream.

Peter Ringel agrees:

yes, they are regression x1000 on speed. so far feedback loops/ "reflexivity" kill it. As far as I understand.

Hernan Avella warns:

I would abstain from making any statements about the state of the art ML applied to trading, specially from a place of ignorance. Whoever works in this field (which there are only a handful in this list), and interacts with just the basic chat GPT 4.0, realizes immediately the productivity boost and immense potential to improve one's process. Only a moron would expect a good output from just feeding prices to the engine or asking simple questions.

Peter Ringel agrees again:

nooo! especially if you are ignorant in a field , better check if that poses a risk to your systems. I believe AI is a risk to traders. Here is a fact already reality: ChatGPT empowers people to do substantial back-tests.

Big Al adds:

And doing backtests poorly, or being improperly overconfident in backtests, is a threat to one's trading.

Humbert K. wonders:

With reference to the skynet, it is hard to guess if and when fully autonomous weapons will happen. My 2 cents is: Fully autonomous weapons will happen. There are debates as to whether we should let machines make kill decisions. I can say though our adversaries' weapons developments will not be bound in any way by any moral or ethical standards. If the bots can communicate with each other and collaborate to perform. When will they no longer need human inputs or interventions?

Eric Lindell writes:

There's a limit to what computers can do with the massive amounts of data available in countless categories. To find the perfect mix of factors to plug into a formula — if there is such a thing — would require a number of operations that increases exponentially with the data-set size.

Humans are good at intuitively navigating such complex search spaces. Computers using brute force just aren't powerful enough yet — and may (in principle) never be. That said, if a human comes up with a plausible conjecture relating stock picks with subsequent price performance, computers can certainly back-check the theory.

I'm working on one now regarding immediate post-IPO performance of stocks selected by certain criteria — criteria that aren't widely (or even narrowly) recognized for their relevance — pertaining to historical research of a revisionist nature.

Sep

8

More an open question - don't have the answer…To what extent are economic figures released from gov and gov related entities are really representative of the whole eco situation in the US and Canada? Eg have a number of friends in the US who have lost their jobs in recent months in various industries - and find it hard to get back in. Of course these are all anecdotes only.

The thing I noticed about so many analysts now (also traders) is that they take everything for granted- but our world is based (at least to some extent) on smoke and mirrors.

Larry Williams responds:

For years I have heard this argument: the Gummint guys cook the books, yet their data has, indeed, reflected reality. As I see it, the Shadow Stat crowd just seeks something to prove they are right about being wrong.

Humbert H. comments:

This weekend some figures came out with a huge drop in employment of the native-born Americans and a large increase in the employment of the foreign-born. Supposedly, Bureau of Labor statistics show that 1.2 million native-born workers lost their jobs last month while the number of foreign-born workers increased by 668,000 in August. So depending on who your friends are, you can get a vastly different impression of the overall employment situation.

Steve Ellison comments:

The labor market is very much a mixed bag. The Wall Street Journal had a feature article in May about the "white-collar recession", while it appears that job openings for blue-collar and service workers are going begging.

The big tech company layoffs this year included significant numbers of H-1B visa holders. An H-1B visa holder who is laid off must find a new job within 60 days or leave the US. I read a month or so ago that 90% of the laid-off H-1B visa holders had found re-employment. That situation might be exacerbating the white-collar recession for native-born workers as even in good economic times, many companies use H-1Bs as a way to pay below-market salaries. It is easy to imagine that in a tech market glutted with job seekers, most companies choose the cut-rate H-1B holders.

I looked in the latest BLS report:

Comparing apples to apples (in thousands):
first number July - second number August
Foreign-born employed: 29728 - 30396
Foreign-born unemployed: 1142 - 1171
Native employed: 132254 - 131031
Native unemployed: 5230 - 5452

Big Al writes:

When I think of economic data, I think about how the releases affect markets. As has been posted on the list before, the question is: If you knew the number beforehand, could you trade it? How will the market react? And in today's market, there may be many black boxes programmed to trade each release in particular ways, and then adapting to the reactions to previous releases. And then one must wonder whether some players get the number faster than others.

I asked ChatGPT for examples of data breaches, and it provided these:

US Federal Reserve Lockup Breach (2020): In March 2020, it was reported that a former Federal Reserve employee and his contacts had allegedly leaked confidential economic information to a financial analyst, who then provided it to traders. This case raised concerns about the security of the Federal Reserve's data release process and led to a review of its procedures.

UK Pre-Release of Budget Information (2013): In 2013, it was discovered that some traders had gained access to the UK government's budget information a day before its official release. This breach resulted in regulatory investigations and legal actions against those involved.

Australian Bureau of Statistics Data Leak (2016): In 2016, the Australian Bureau of Statistics had to delay the release of its employment data due to concerns about leaks. The incident highlighted the importance of maintaining data integrity and security in the release process.

European Central Bank Data Leak (2016): The European Central Bank had a data leak in 2016 when it accidentally released sensitive market-moving information to a select group of media organizations a day ahead of the official announcement. This breach raised questions about data handling procedures.

Kim Zussman adds:

NGOs too:

Unusual Option Market Activity and the Terrorist Attacks of September 11, 2001

Eric Lindell asks:

Relative to which indicators would you say their data reflects reality? The government misdirects on so many things, why would their data be reliable? Cost projections for scientific or national security projects are not reliable. Remember when they redefined unemployment to make it drop a few points? Didn't they stop reporting M2? Didn't they lose a couple trill in the pentagon budget? Have recently reported CPI numbers reflected actual costs to consumers? From what I've seen in stores, CPI numbers seem low.

Nils Poertner answers:

exactly. Eric, or see this Gell-Mann amnesia effect. People (not just medical doctors) correctly knew about "misreporting" related to some viral infections, but then read the WSJ and think CPIs numbers are all correct.

H. Humbert comments:

My take is the labor market is just fine and doing exactly what we want to see. Labor participation is rising. Demand for workers is falling.

Archives

Resources & Links

Search