Thoughts on Trading and System Design

When designing a trading system or trading methodology, some curve-fitting is unavoidable. Nearly every trader has had the frustrating experience of seeing a "can't lose" trading method fall apart when it is applied in real time. Carefully Designed & Tested System Should Bring Success - The system can be carefully designed with the best of intentions. It seems to incorporate all the elements of a successful system. It controls risk and lets profits run. In testing it shows tremendous profits in nearly every market you intend to trade.

You have back tested, forward tested, and analyzed statistical measures that are supposed to ensure robustness. You have even paper traded for a while, adhering strictly to your trading rules, and the paper trading has been reasonably profitable, although not up to the standards of your test results.

Did Market Conditions Make It Less Successful? You can rationalize this by saying the markets have been tough lately and difficult to trade. You make money for a short time but then, incredibly, you experience a loss that exceeds any of the losses that your system saw in five years of testing and six months of paper trading. The draw down continues, and eventually you stop trading the system.

This often-repeated scenario is almost certainly the most common trading experience among system traders. Ironically, the introduction of personal computers and sophisticated analytical software has not contributed to the solution of the problem but has probably made it more likely to occur.

With Advantage of Hindsight, Markets Appear More Orderly Than They Are - It is extremely easy to sit down in front of a screen and devise a trading system that seems, in hindsight at least, to be unbeatable. Most oscillators, for instance, look like they call market bottoms and tops pretty well. Most trend-following indicators hug major trends very nicely, With the advantage of our hindsight, markets usually appear to be much more orderly than they are, so buying dips in an up trend, for example, seems to be an excellent strategy.

Effective Looking But Overly Curve-Fitted - Any trader with a PC, some data and some analytical software can devise an effective looking trading system with very little effort. The problem is that, more likely than not, the system will be overly curve-fitted and useless for real-time trading.

Defining curve-fitting isn't as easy as it may seem. It's a bit like art appreciation; it's hard to describe what you like, but you know it when you see it. In fact we curve-fit lots of data in our non-trading endeavors and think nothing of it. After all, if you stop and think about it, the line between experience and curve-fitting is a very fine one.

Examples of Curve-Fitting - A couple of examples may help define curve-fitting. Let's say you're a creative sort of person and you devise five new and different technical indicators that you hope will generate profitable trading signals. You program them carefully, and display them one at a time against several favorite markets. Much to your chagrin, you find that four of the indicators seem to have no relationship to the markets at all. The fifth, however, follows prices very nicely and often seems to anticipate major moves. You decide to use the fifth indicator and discard the others.

Curve-Fitting Without Realizing It - Believe it or not, by choosing the study that best agrees with your perception of what a technical indicator should do, you have curve-fitted.

You might reply that this is the method by which virtually every trading system, whether mechanical or discretionary, is invented and you would be correct. After all, there are trading systems that work, and they have been put together by observation, not by luck or guesswork.

Some Curve-Fitting is Appropriate - Some fitting of a trading method to price movement is natural and beneficial. To the extent that we do this and still retain flexibility and the ability to handle future events, curve-fitting is entirely appropriate.

There is a vague line that is all too easily stepped over, however, and to cross that boundary brings certain failure rather than success. We find it difficult to come up with an exact definition and perhaps the line is so fine that there isn't a clear definition of where experience and common sense leave off and unproductive optimization sets in.

Two Mice and a Maze Analogy - Here is an analogy that may help to explain the phenomenon. Construct a maze that is relatively intricate and that requires a fair amount of twisting and turning to get to the center. Train two mice to navigate the maze. The first mouse learns that if it turns left twice, then takes five steps, then turns right three times, takes two steps, turns left once, and then turns right twice it will be at the center of the maze. The second mouse learns that at every junction it first turns left, and if it bumps into a partition it should turn around and go the opposite way.

The first mouse, once it has learned the pattern, negotiates the maze flawlessly every time. The second mouse is slower, although it eventually arrives at the proper destination. Contrary to How it Appears, the First Mouse Does Not Have The Advantage - The first mouse seems to have an obvious advantage. It finds its way to the center of the maze with ease, while the second mouse, handicapped by a much simpler navigation system, struggles and makes many wrong turns before finally reaching the goal. The second mouse clearly seems much less capable than the first mouse.

A Similar Maze but A Difference - Now let's build a new maze...make it similar to the first for a few turns, but different thereafter. The first mouse will proceed confidently for a while, but will soon be hopelessly lost and disoriented, much like some traders we have observed.

The second mouse won't notice the difference between the first maze and the second. The mouse that had previously seemed inferior plods along and treats every twist and turn in the new maze the same simple way. Slowly and inexorably the second mouse will find its way to the center, no matter how the maze is configured.

Rewards Providing the Maze Doesn't Change - The first mouse either performs perfectly or terribly because it has been taught an overly curve-fitted system, while the second mouse has learned a crude system that enables it to get to the center every time.

The first mouse will perform extremely well and will be rewarded as long as any maze the mouse encounters is substantially the same as the original maze that it memorized so capably. The second mouse will be slow but it will eventually be rewarded no matter what changes you make to the maze.

Trade Successfully With Crude Methods - The principles in our analogy hold true when trading systems are applied to the maze of futures prices. If we are going to navigate the markets successfully, we must create relatively crude systems that will work no matter what maze is created by tomorrow's prices. Make Your Trading Less Complicated - We should be very careful about how we use computers to test and modify trading systems. Changes that merely improve the numbers without adding to the logic of the trading plan should be questioned. Try to find ways to make your trading less complicated and more adaptive to changes in market conditions.

In Futures Trading the Correct Answers to Questions seem Contrary - One of the underlying reasons why it is so easy to fall into inadvertent curve-fitting is that in futures trading the correct answers to even the simplest questions seem counter-intuitive.

New Traders Wrongly Want To Trade Contra-Trend - For example, one of the most basic decisions a trader makes when first conceiving a system is whether to be a counter-trend trader or a trend-follower. Almost all beginning traders, in our experience, opt to be counter-trend traders. They won't buy until prices appear "cheap" to them, and they won't enter a position unless there has been a reaction or some form of correction that allows them to buy on a dip or sell on a rally.

They buy only when there is a bargain to be picked off and sell only after prices have reached an apparent peak. They are constantly looking for signals like key reversals, support and resistance levels, and any other pattern that will allow them to get into the market at a major turning point. Traders Want To Buy the Exact Bottom and Sell the Exact Top - It is only natural that such a trader gravitates toward counter-trend indicators like stochastics or RSI, and may become a devotee of Elliott wave theory, percentage retracement calculations, and methods that can forecast or identify tops and bottoms. If buying bottoms and selling tops is good, then buying at the exact time the market turns must be even better.

Picking Tops/Bottoms Easier Providing there's an Underlying Structure - It would be much easier to find tops and bottoms if the market had some form of underlying structure or order that made highs and lows predictable. It seems that trading at tops and bottoms is so impossible using conventional methods that these traders, out of sheer desperation, must eventually fall prey to methods that presume some underlying orderliness to the markets.

Searching For Tops & Bottoms Forecasting Methods - Methods such as Gann angles, Fibonacci ratios, cycles, wave theories, or even such totally absurd approaches as astrology or the "delta phenomenon" offer the only hope of forecasting tops and bottoms on a regular basis.

Take any hare-brained idea and write a book about it or program it into a software package and suddenly it has instant credibility. Desperate top and bottom seekers will flock to it. The Desire To Pick Bottoms & Tops Has Caused More Failure than Anything Else - After many years of observation, I have come to the conclusion that the natural desire to buy low and sell high is more responsible for failure among futures traders than any other behavior.

Successful Traders Are Trend-Followers - Almost every successful trader we are aware of is a trend-follower. This includes both private traders and professional Commodity Trading Advisors who trade billions of dollars worth of public funds. Unfortunately, trend-follower is counter-intuitive. At first it seems to make no sense at all. It is the direct opposite of how we have been taught to succeed as shrewd traders.

Inexperienced Traders Think they Could Have Made the Trade Earlier than Trend Followers - Why buy at or near new highs when we obviously should have bought earlier at much better prices? Trend-following methods are usually scorned by less experienced traders who always assume that they somehow could have bought days ago, at the bottom. The fact is, however, that a market must make new highs and lows continually in order to get anywhere important. When gold went from $200 to over $800 in 1979 it was making new highs all the way. When soybeans broke $5 and went to $12 in the early 70's the same thing was obviously true.

Trend-Following Approach Is Best - We won't go so far as to say that anyone who is successful at counter-trend trading should abandon it, but we firmly believe that the vast majority of traders should concentrate their efforts on the trend-following approach.

Using Too Many Indicators Result in Poorer Results - Another common theme that runs through system design is the search for confirmation of a trading signal. Simply put, this means that a trading signal given by one technical indicator or chart pattern must be confirmed by one or more other indicators in order to be valid.

For example, if a stochastic dips below 20 and then turns up, the trade won't be taken unless an RSI or another oscillator confirms the stochastic signal. This can be taken to absurd ends; we have seen trading systems with as many as thirty elements that all have to fall into line before a trade is taken.

Physiologically You Feel Better With Extra Filters - It is easy to see how that can happen. No one wants to take a loss. It is comforting, after a trading loss, to tinker with your trading system and add another filter or confirmation that, in hindsight, eliminates the loser. Software that emphasizes optimization makes it even easier. Probably the best way to handle this sort of situation is to avoid redundancies. For example, if you are using one oscillator to signal market exits, it is better to decide exactly how the oscillator should be used and stick to your rules rather than adding several more oscillators to your system and requiring that they confirm one another before you exit.

Additional Rules Detract From System Performance - The same is true of any other type of indicator. Remove any indicator that essentially duplicates the information from any other. Remember the mice and the maze. Every new rule you add that makes your trading look better in hindsight detracts from your system's ability to handle future price aberrations.

One of the areas that is most easily abused during system design is data; specifically, which markets and time periods to test over and which markets to trade in a portfolio.

Some Systems are Designed to Only Work in Specific Markets Using Hindsight - It is a favorite technique of system sellers to design systems to fit specific markets, create a track record based on a complicated system with lots of rules that eliminate losers in hindsight, and market the hypothetical track record giving the impression that the results are reproducible in real time.

This is akin to teaching the first mouse how to navigate one maze, and then selling it as a mouse that can navigate any maze.

Some Systems are Designed to Work on Data for a Short Time Period Based On Hindsight - There is a much less obvious but equally dangerous form of curve-fitting that involves curve fitting the data to the system. We are referring to the increasingly popular practice of using a computer to pick out short time periods during which chosen markets have historically acted similarly.

For example, we might be told that over the past ten years buying silver on May 10 and selling it on June 1 has resulted in a profit every time. The obvious inference is that if we do it this year, we have a 100% chance of winning. There are tables and tables of this meaningless coincidental data being offered to traders in books and almanacs.

Seasonal Characteristics Are Highly Questionable - Part of the theory is that there is some sort of very short term seasonal or cyclical basis for the similarities, although this is patently unprovable.

A properly programmed PC will find literally thousands of "trades" like this over any fairly extensive set of data, just as an optimization involving a great number of variables will almost always find a great number of "profitable" combinations.

Data Optimization Can Fit a System to Arrive at a False Impression of a Seasonal Characteristic - The optimization fits the system to the data, and the seasonality testing fits the data to the system. Both practices result in overly curb-fitted trading results that offer no hope of success in real trading.

The Trouble Is . . . The Markets Don't Listen - Here is another example of something that initially seems conceptually wrong. We are continually told that every market has its individual character, and that therefore a trading system must be tailored to each market.

We are also told: "Don't trade too many markets because it is difficult to watch more than a few at a time," and: don't test more than a few markets because it is unreasonable to expect a trading system to work well over a range of markets."

All of these concepts seem logical at first. The trouble is, the markets won't listen. They are not predictable. They will not act tomorrow in the same way that they did today or yesterday, and you are fooling yourself if you expect them to.

Trading Systems Should Operate on a Wide Variety of Markets and Market Conditions - Trading systems should be designed to operate profitably over a wide variety of markets and market conditions. They should be simple and flexible enough that they won't be thrown for a loop by changing conditions.

There Is No Best Indicator While we are reasonably convinced that there is no best technical indicator, some are less likely to lend themselves to unwanted curve-fitting.

First, we can divide indicators into two major categories: static and adaptive. Static indicators are technical studies or other entry or exit methods that do not "flex" with changing market conditions, especially market volatility. Good examples of static indicators are those technical studies, stops, and profit targets that are denominated strictly in dollars or market points.

Systems that Use Changeable Targets and Stops are Likely Less Curve-Fitted - Adaptive indicators change stops and targets as the markets change. When these adaptive indicators generate a trading signal, you can say that the market put you into or took you out of a position. Examples include volatility-based entries and exists, channel breakout systems such as Donchian's weekly rule, entering or exiting on an 'n' day high or low, and using recent swing highs and swing lows as entry, exit or stop points. As a general rule, adaptive indicators are less likely to become overly curve-fitted to the markets than static indicators because the system designer will not feel the need to optimize them. This is not because they are any less amenable to over-optimization than static indicators, but because they adapt to changing market conditions while retaining their integrity.

Changeable Target & Stop Methods are Less Likely to Strictly Limit Losses or Profits - The main disadvantage of adaptive indicators is that they do not strictly limit a loss or accurately lock in a profit.lock in a profit. For example, if your exit to limit a loss is a 10-day low, the 10-day low could be $500 away or $5,000 away. If your account is $20,000 in size, it seems unwise to risk as much as 25% of it in one trade, although 2.5% seems acceptable. The same is true if you are fortunate enough to be locking in a profit. Adaptive indicators expand with volatility, making it easy for a hard-won profit to disappear as quickly as it was created. A reasonable compromise might be to allow the markets to dictate your entries and exits under normal conditions, but if a particular market becomes too volatile, limit your potential loss by using a static dollar stop (perhaps keyed to your account size) or avoid the market altogether.

Some Systems are Designed to Work on Data for a Short Time Period Based On Hindsight - There is a much less obvious but equally dangerous form of curve-fitting that involves curve fitting the data to the system. I am referring to the increasingly popular practice of using a computer to pick out short time periods during which chosen markets have historically acted similarly. For example, we might be told that over the past ten years buying silver on May 10 and selling it on June 1 has resulted in a profit every time. The obvious inference is that if we do it this year, we have a 100% chance of winning. There are tables and tables of this meaningless coincidental data being offered to traders in books and almanacs.

Seasonal Characteristics Are Highly Questionable Part of the theory is that there is some sort of very short term seasonal or cyclical basis for the similarities, although this is patently unprovable. A properly programmed PC will find literally thousands of "trades" like this over any fairly extensive set of data, just as an optimization involving a great number of variables will almost always find a great number of "profitable" combinations.

Data Optimization Can Fit a System to Arrive at False Impression of A Seasonal Characteristic -
The optimization fits the system to the data, and the seasonality testing fits the data to the system. Both practices result in overly curb-fitted trading results