Neural Networks Primer
I would like to present a cursory overview of what neural networks are and how they can be applied to the financial markets. Attention here will focus on neural network paradigms, architectures, and training and testing regimens for a financial forecasting application.

Neural networks "learn" to solve problems by transmitting information between neurons, which are the basic processing units of a neural network. A neural network typically includes several layers of neurons. Network architecture determines how many layers there are, how many neurons are in each layer, how they are connected, and what transfer function is to be used. There are numerous learning paradigms, including two popular in financial analysis. The first popular paradigm is a recurrent back-propagation network that learns temporal information through the order in which the facts are presented. The second paradigm is a feed-forward back-propagation network that trains through back propagation of error, in which temporal information is encoded into the input data itself by taking a preprocessed "snapshot" of the data. A typical back-prop network architecture is shown in Figure 5-7. This paradigm will be used here to illustrate network architecture.

Figure5-7 Simple feed-forward back-propagation network. A back-prop net using technical, fundamental, and intermarket data. The network trains by back propagation of error throughout the network.

The back-propagation network is composed of an input layer, one or more hidden layers, and an output layer. The input layer contains a neuron corresponding to each input (independent) variable. The output layer contains a neuron for each (dependent) variable to be predicted. The hidden layer contains neurons which are connected to both the input and output layers. The layers are typically fully connected, with every neuron in one layer connected to each neuron in an adjacent layer.

The values associated with each input neuron are fed forward into each neuron in the first hidden layer. They are then multiplied by an appropriate weight, summed, and passed through a transfer function to produce an output. The outputs from the first hidden layer are then fed forward into either the next hidden layer or directly into the output layer in networks that have only one hidden layer. The output layer's output is the prediction made by the network.

The number of neurons in the hidden layer is determined through experimentation. For any nonlinear problem such as prediction of stock or futures prices, the network needs at least one hidden layer. In addition, the transfer function should be a nonlinear, continuously differentiable function, such as the sigmoid, which allows the network to perform nonlinear statistical modeling. Figure 5-8 presents an example of a hidden neuron.

Figure5-8 An example of a hidden neuron. Neural networks are composed of individual, interconnected, processing elements called "neurons".

Input Data Selection and Preprocessing Neural network modeling is based upon the developer's understanding of the underlying real-world relationship between the inputs and the output. Decisions must be made about what is to be predicted and what data will be used as inputs in the network. "Garbage-in, garbage-out" applies to neural networks. Knowledge of the financial markets, coupled with the use of various tools such as principal component analysis to find correlations between related markets, is necessary for appropriate input data selection.

Once input data has been selected, it must be preprocessed. By reducing the number of inputs to the network, preprocessing helps it learn more easily. Two widely used preprocessing methods are known as "transformation" and "normalization." Transformation manipulates raw data inputs to create a single input to a net. Normalization transforms a single data input to distribute the data evenly and scale the data to match the range of the input neurons.

In most neural network applications, transformation involves algebraic or statistical manipulation of the input data. In financial forecasting applications, a wide variety of technical indicators, commonly used by traders to interpret market behavior, can be used as transforms. Preprocessed inputs might include differences, ratios, and moving averages of the open, high, low, close, volume, and open interest raw data. Each neuron in the input layer represents one of the preprocessed inputs.

Various transformation and normalization methods can be explored, since some are more appropriate than others for specific applications. Once the network architecture has been selected and the inputs have been chosen and preprocessed, data facts must be selected.

Fact Selection A fact is represented as a row of related numbers where the first i numbers correspond to the i network inputs and the last j numbers correspond to the j network outputs. A group of related facts is known as a "fact set." If two facts have exactly the same input and output values, only one of the facts should be included in the fact set. Once it has been defined, in most financial applications the fact set should be separated into mutually exclusive training and testing subsets.

Back-propagation networks operate in two modes: One is a learning mode in which the network uses facts from the training set to modify its internal representation through weight changes. The other is a recall mode in which the network processes inputs from the test set and utilizes its previously learned representation to generate associated outputs. The relative performance on the test set of various trained networks is used to determine which net should be incorporated into the final application.

Training and Testing Once the facts have been selected, they are presented to the network serially during training. The weights, which allow the network to adapt its internal representation when modeling a problem, are typically initialized with small randomly assigned weights. If the weights are initially set to the same value, the network might never learn since error changes are proportional to the weight values. For each pass through the training set, the network computes a measure of the error between its generated output and the desired output for each output in the output layer. The errors are then propagated backward through the network, layer by layer, altering the connection weights between the neurons in order to minimize the total error associated with each output.

Each time the weights change, the network is taking a step on a multidimensional surface, representing the overall error space. During training the network is traveling along the surface, trying to find the lowest point or minimum error. The weight changes are proportional to a training parameter called the "learning rate." Other training parameters that can be adjusted during the training process include temperature, gain, and noise.

With various training parameters, preprocessing methods, and architectural configurations that can be explored, an automated training and testing regime that integrates testing with training is needed. Tools such as genetic algorithms and simulated annealing can be used to expedite searching these parameter spaces. Genetic algorithms are effective for many parameter optimization tasks. Simulated annealing automates learning rate adjustments during training by including a variable temperature term that affects the learning rate. When the temperature is high, learning is rapid. When the temperature drops, learning slows down as the network settles upon a solution.

Over training Over training, analogous to curve fitting of rule-based trading systems, is one of the major pitfalls that must be avoided when developing neural networks. Over training occurs when a network memorizes the subtleties and idiosyncrasies specific to the training set, lessening its ability to generalize to new data. When over training occurs, a network performs well on the training set, but poorly on an out-of-sample test set and later during actual trading. To avoid over training, network training should be halted periodically at predetermined intervals and then run in recall mode on the test set to evaluate the network's performance on predetermined error criteria. Then the training resumes from the point at which it was halted. This automated process continues iteratively until the performance on the test set begins to degrade, suggesting that the network has begun to over train. All interim results that met the error criteria are evaluated further.

Error Statistics Another important network design decision concerns which error statistics to use for training and testing. One measure might be the difference between the actual calculated statistic such as a moving-average value and the network's output. This difference would be calculated for each fact in the test set, summed, and divided by the number of facts in the test set. This is a standard error measure called "average error." Various error measures include average absolute error, sum-of-squares error, and root- mean-squared (RMS) error. After a neural network model has been selected, it will need to be retrained periodically. Ongoing research should be conducted to modify the inputs, outputs, architecture, and overall implementation of the testing and training process for the purpose of improving network performance and predictive accuracy.

You cannot expect to design an effective neural network financial forecasting application unless you have considerable trading experience, programming and mathematics expertise, and the time to devote to the task. Successful development of a neural network application is a combination of both "art" and "science." This endeavor can be extremely time consuming and labor intensive, even for a group of experts working collectively.

I speak from experience. In 1991 the Predictive Technologies Group, the research and development division of my firm, introduced VantagePoint Intermarket Analysis software, after experimenting with intermarket analysis since the mid-1980s. VantagePoint applies neural networks to intermarket analysis in order to predict market trends, moving averages and next-day prices for various financial futures markets. Currently there are 21 custom-designed VantagePoint programs for currencies, interest rate markets, stock indices, and the energy complex which allow traders to benefit from intermarket analysis, without having to reinvent the wheel or become rocket scientists.

Practical Applications
While used extensively by traders to identify trends, simple moving averages, due to their mathematical construction, are considered a lagging indicator. This limitation has presented an enormous challenge to technical analysts for decades. Continuing research directed at reducing the lag has resulted in a steady stream of modifications to this relatively simple, yet effective, technical indicator. [These are discussed extensively in Chapter 8.]

Moving-Averages Primer A moving average smoothes out fluctuations in prices to demonstrate an underlying directional trend. In a typical moving-average system, a crossover oscillator, involving either two moving averages or a price and a moving average, is developed. Entry and exit points are determined when one indicator crosses above or below the other indicator. For instance, a trader might rely upon Donchian's 5-day versus a 20-day moving-average method to determine entry and exit points.

Still, moving averages are slow to react at turning points. Traditional moving- average systems typically get in and out of trades after a change in market direction has occurred - often by several days - which gives back profits and can turn winning trades into losers. Additionally, crossover systems tend to generate false signals during sideways or nontrending markets, resulting in whipsaws when alternating buy and sell signals are triggered as the moving averages crisscross one another.

Through system testing, the size of the moving averages can be optimized in an effort to tailor them to each market's price behavior and to find the best crossovers which will capture the "turns" in that market and help reduce the lag. Additionally, other smoothing methods can be used in conjunction with moving averages to reduce the lag further, resulting in trading strategies that presumably will be more responsive to abrupt changes in the market. These include the use of price filters or sensitivity bands surrounding the moving averages, and the addition of a third moving average, both of which were part of ProfitTaker's original system architecture. Other widely used methods include Bollinger Bands and the popular 4-9-18 day moving-average combination.

Forecasting Moving Averages Of all the variations on simple moving-average approaches that have been developed to date, displacing moving averages probably comes the closest to what is desired, since it is an attempt to convert them into an "anticipatory" technical indicator. Yet, even a displaced moving average has one glaring weakness: the simplistic assumption that the moving-average value at a future time will be the same as its calculated value today. Basically, this is nothing other than a primitive prediction of the moving-average value for a time period in the future, in which that value is assumed to be identical to today's calculated moving-average value - a highly unrealistic assumption under real-world trading conditions. Why not take this concept one step further and actually predict moving averages? By doing so, their smoothing benefits are retained, while their lag is totally eliminated once and for all.

Related Inter-market Additionally, by incorporating intermarket inputs into the neural network design, the moving-average predictions are not limited to single-market inputs. In the case of the VantagePoint T-Bond program, for instance, the inputs into the forecast of the moving averages take into consideration the open, high, low, close, volume, and open interest for the past 10 years from actual T-Bond contracts during their most actively traded periods, plus nine related markets (T-Bond cash, NY Light Crude Oil, CRB Index, deutschemark, US dollar Index, Eurodollar, Comex gold, Japanese yen, and S&P 500 Index) that have been determined to have a significant effect on Treasury bonds.

Since correctly identifying trend direction is crucial to successful trading, trend forecasting, compared with trend following, offers a promising new way for traders to identify trends and changes in their direction as they are happening, not after the fact. Coupling neural networks with intermarket analysis, VantagePoint forecasts trends by predicting moving averages for up to 4 days in the future. To do this, VantagePoint is composed of five separate neural networks that are each responsible for predicting a particular output variable. One neural network predicts tomorrow's high, a second predicts tomorrow's low, and a third network predicts a "neural index" which indicates when the market is about to make a top or a bottom. The fourth network predicts what a 5-day moving average of closes will be 2 days in the future, while the fifth network predicts what a 10-day moving average of closes will be 4 days in the future.

While the moving-average predictions are used to determine trend direction, the predicted highs and lows are used to set entry and exit points and stops. This predicted high-low range is analogous to support and resistance lines in traditional single-market technical analysis, except that VantagePoint's predicted daily range is based on the neural network pattern recognition capabilities of the neural networks coupled with the intermarket analysis of 10 related markets.

Position traders utilize the predicted highs and lows to help set entry points, then use the predicted high-low range on subsequent days to tighten stops. For example, if you are long Treasury bonds and the market is expected to continue to move up tomorrow, you might set a trailing stop for tomorrow a few ticks below tomorrow's predicted low which acts as a support level. This would decrease the likelihood of being stopped out prematurely as the result of intraday volatility in the market, and yet protect profits in the event of an abrupt market downturn in which the predicted low is breached.

The predictions of the next day's high and low are useful for determining entry and exit points for day trading as well. If the forecasted indicators suggest that tomorrow will be an up day, day traders can wait for the market to trade down toward the predicted low and then enter a long position a few ticks above the predicted low, to be closed out intraday just below the predicted high. The reverse would involve entering a short position just below the predicted high on a day expected to be down and exiting the position a few ticks above the predicted low. This may be done several times daily.

Editor's Comments Obviously, there are numerous procedures that you can do with neural networks. Mendelsohn is absolutely correct in saying that you must thoroughly understand what you are doing - both with respect to neural networks and with respect to financial modeling. When you read and thoroughly understand the subsequent chapters, you'll probably realize that there are vast untapped areas to apply neural networks to - such as to exits and to position sizing.

Want to see how you can use VantagePoint
to give you an edge in your trading? Click Here >>>