A new dimension
to financial data

At RAM we have been working for over 12 years on building several factors for stock selection, which seek to capture various inefficiencies such as momentum, growth, quality, value, low-risk, etc. Although we have designed every single factor with the view to extract information about a given inefficiency as efficiently as possible, all these factors need to be preprocessed so that our machine learning models can infer accurate predictions from them. Preprocessing is crucial, especially for deep learning models which are using neural networks, for which convergence is clearly helped when data is cleaner. Cleaning the data means working on the distribution of inputs, the potential presence of non-available data, and identifying causes for outliers, in order to potentially retreat them.

Among other techniques, we are using the Scikit-Learn package for these processing steps, in particular their various scalers:

Quantile transformer: Rank transformation in which distances between marginal outliers and inliers are shrunk. Gaussian transformer: Another non-linear transformation in which distances between marginal outliers and inliers are shrunk. Standard Scaler : Removes the mean and scales the data to unit variance. Robust scaler : Removes the median and scales the data according to the quantile range (by default the interquartile range).

Feature selection

As discussed previously, over the course of the last 12 years, we at RAM have built a repository of numerous fea- tures capturing a variety of market inefficiencies on the back of traditional and alternative data sources. We use deep learning to combine all these stock features when predicting a stock’s return, risk and liquidity dynamics. Deep learning approaches provide us with an optimal non-linear combination of these features, revealing their non-linear interactions.

We face numerous challenges when building a deep learning model combining a multitude of features:

• As the number of features and hence the number of model parameters grows, it becomes increasingly difficult to prevent overfitting when training a deep learning algorithm. Very large artificial neural networks lead much more easily to an overfitting of a training dataset, making generalization capability of the model poor.
• The more input features in a deep learning algorithm the higher the likelihood of having highly correlated features in the input set, which is making the optimi- zation convergence of the model much less stable, as it can lead to large variations in the local gradient of these variables.
• Larger input sets naturally require much more com- puting power to be trained, which makes the process of building the model extremely time-consuming.
• Interpretation of the model is increasingly complex with the growth of inputs.

To overcome all these issues, there is a lot to gain by doing a feature selection before training a model when one has a large number (hundreds, thousands or more) inputs.

The feature selection process will first look to identify the most informative and relevant features to predict the output variable. They are the set of variables that should together help predict in the most robust manner the out- put variable.

Univariate feature selection

The simplest way to select input features is to retain those that have the strongest relationship with the output variable. The most basic approach would be to select the features that are most highly correlated with the output variable. To better assess the linear relationship and consider the significance of this correlation, one can use a F-test which estimates the degree of linear dependency between each input and the output variable, selecting the top ranked ones.

But the limit of correlation or F-test scoring is that they only reflect linear relationships between the input and output va- riables, when there are often non-linear interactions at play in our data. In this case it would be more useful to conduct a more general univariate test looking at mutual information between each input and the output variable. Mutual infor- mation being entropy-based, can uncover any statistical dependency to help select the most relevant features.

Linear v Non-Linear Relationships

There is, however, a very strong limit to univariate tests to achieve a robust feature selection, as they retain the best features to explain the output on a stand-alone basis but totally ignore the complementarity or the redundancy of the features retained. In practice, the input features often exhibit a significant degree of correlation, making univariate tests sub-optimal.

Linear v Non-Linear Relationships

Hence the need for feature selection with a multivariate angle capturing the complementarity of the inputs to identify an overall optimal group of variables.

Regression-based ranking

We can fit a multivariate linear regression model to explainzbr the output variable with all features and rank eachzbr feature along its statistical significance to retain the mostzbr explanatory ones.
The downside in this feature selection approach lies inzbr the fact that a number of highly correlated input factorszbr can still be retained, as often they would tend to havezbr large betas with opposite signs and high statistical significance in a multivariate regression. This subsequently increases the risk of having a sub-optimal set of features in the deep learning model.

Lasso

A feature selection based on a Lasso model can help to overcome the limits of a linear regression by reducing the likelihood of multiple highly correlated features being retained. A Lasso adds a penalty that sums up the absolute scale of the regression betas, a “L1” penalty, which forces many parameters to be zero as the strength of the regularization increases. Weak and redundant features rapidly end up with zero coefficients, making the Lasso well-suited for a robust feature selection.

Recursive feature elimination (RFE)

Recursive feature elimination (RFE) is a process that aims to ensure one misses the least interactions between features, by gradually taking into account how these interactions change as we reduce the size of the feature set.
It starts by fitting a model with all input variables, then iteratively eliminates groups of k variables at each step with the least importance in the model. One typically evaluates the importance of a variable by measuring the reduction of accuracy of the model when it is eliminated.
The model is then fit again with the reduced feature set, to then exclude k variables again, and so on, until we reach the desired number of input features.
We can also select the right input features into our model using a more general non-linear model capturing the non-linear interactions between the input features, sometimes even using our deep learning model itself (e.g. an Artificial Neural Network). The main downsides here are heavy computing times and the need for very large datasets to complete this RFE feature selection.
Indeed, to ensure no overfitting with the large number of parameters involved, one should use a cross-validation process that ensures generalization, with large data samples required. It is also highly convoluted to have to set the hyperparameters for a more general feature selection model.
RFE is a computing-intensive feature selection technique as multiple fits are required to reach the targeted number of features, but it should best capture the non-linear interaction dynamics of the input factors selected and their complementarity.

Deep Learning Model : what dimension?

Before diving into the engineering of the deep learning model, the tunings of the model parameters, the work of a data scientist setting the building blocks of a deep learning model is not dissimilar to those an architect. One of the important elements of a deep learning model architecture is its dimension. This dimension will depend a lot on the size of the input, i.e. the number of features retained after our feature selection process. But an equally important element will be the size of the data sample we rely on to train the algorithm.

Artificial Neural Network Dimensions

The way we approach dimensioning at RAM is “constructive” in the sense that we tend to start with simple and small artificial neural networks, and then increase width and depth incrementally until we find what seems to be an optimal network size. Given the size of our datasets, the largest which is typically dozens of GB with millions of single stock observation periods, we never reach the thousands of layers found in some of the most recent image recognition deep neural networks.

“ The more data we have in our training and validation samples, the more parameters the algorithm can handle without overfitting and loss of generalization. ”

In the case of an artificial neural network, dimension both represents the width and the depth (number of layers) of a neural network. When one scales up a network from low dimensions, it is either possible to increase the width or the depth of the network. There is solid theoretical foundations for the strong performance of deep networks.
Montufar, Pascanu, Cho and Bengio (2014) demonstrate the strength of a combination of layers to capture complexity, identifying a number of input regions exponential to the depth of the network. ¹

In practice however it does seem that deep learning practitioners tend to exaggerate the number of layers in their networks, sometimes massively over-dimensioning them, while much smaller wider alternatives would both improve accuracy and save them computing costs, see for instance Wide Residual Networks by Zagoruyko & Komodakis, 2016. ²

Schmidhuber, Greff and Srivastava also published a very interesting article aiming to overcome the difficulty and limits of very-deep neural network training (up to close to arbitrary size) in their Highway Networks (2015). ³

¹ Reference: https://arxiv.org/pdf/1402.1869.pdf
² Reference: https://arxiv.org/pdf/1605.07146.pdf
³ Reference: https://arxiv.org/pdf/1505.00387.pdf

As illustrated below, high bias is a characteristic of an underfitted model and high variance is a characteristic of an overfitted model.

Image source : i.stack.imgur.com/t0zit.png

How to control the generalization

To control the generalization of the model, we analyze the convergence of the loss function over training and validation samples. A loss function is a measure of how well the algorithm models the given data.

Good Fit

Ideally, the situation when the loss function reaches a global minimum, the model is said to have a good fit on the data. This case is achievable at the point between overfitting and underfitting. With the passage of time, our model will keep on learning and will be able to generalize. As such, the error for the model on the training and the validation data will keep on decreasing.

Under Fit

Models with a high bias will not learn enough from the training data and will oversimplify the model. Consequently, it cannot capture the underlying trend of the data. As observed on the chart below, this leads to a high error count on both the training and validation data.

Over Fit

Models with high variance overlearns on the training data, capturing all its specificities (including true properties but also noise), and does not generalize on new data. As a result, such models get opposite convergence patterns on training and validation loss function: the more the model learns (decrease in training error), the less it is able to generalize (increase in validation error).

How to improve the generalization

All the steps described in previous chapters are important to help the generalization of the model, including data related matters (size of the dataset, data preprocessing, features selection) and algorithm specific issues (type and dimension of the algorithm). Additionally, several techniques can be used to control the machine during the learning process, to achieve an optimal fit:

• Fold cross validation

  In K Fold cross validation, the data is divided into k subsets. A training/validation process is repeated k times, such that each time, one of the k subsets is used as the validation set and the other k-1 subsets are put together to form a training set. The error is defined as the averaged error over all k trials. This helps to reduce bias/ underfitting risk as most of the data is used for training and reduces variance/overfitting risk as most of the data is also being used for validation.

• Regularization

  One of the most common mechanisms for avoiding overfitting is called regularization. Many regularization approaches have been developed and below we are presenting two internally used techniques.

  Ridge Regression:
  To reduce the impact of irrelevant features on the trained model, another element is added to the loss function. This element leads to the loss function being negatively impacted by high features coefficients. As the function is minimized, the coefficients tend to be reduced, resulting in a simplified model, thus less prone to overfitting.

  Loss = Σ( Ŷi– Yi)2 + λΣ β2

  with

  Σ( Ŷi– Yi)2 = Original Loss

  λΣ β2 = Regularization term

  Dropout:
  Deep neural networks have many parameters and as such, are prone to overfitting. The Dropout is a technique for addressing this problem. The concept is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much*.

  *Dropout: A Simple Way to Prevent Neural Networks from Overfitting.
  Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov; 15(Jun):1929−1958, 2014.

• Early Stopping

  During the training of the machine, there will be a point when the model will stop generalizing and start learning the statistical noise in the training dataset. A technique called Early Stopping is used to avoid overtraining by stopping the training when a trigger is reached. For instance, this trigger can be the point where the validation error starts increasing.

• Learning rate

  

  Learning rate is a hyper-parameter that controls how much we are adjusting the weights of our network following a training step. The lower the value, the lower the impact of the correction. Our approach is to avoid the risk of setting a rate that would be too high or too low by using an adaptive learning rate which evolves during this training process. While using a low learning rate might seem a good idea in terms of ensuring that we do not miss the global minimum, it could also mean that the convergence will take a long time, especially if it gets stuck on a region with the same error (plateau). An increasing learning rate helps the convergence to progress faster and allows for traversal of plateau and saddle points (critical point in which some dimensions observe a local minimum, while other dimensions observe a local maximum). However, when the learning rate is too high, huge steps will be taken and the minimum will never be reached. As observed on the chart below, the loss is reduced very quickly, but then the parameters become unstable and the loss diverge from the minimum on both training and validation set.

  

At RAM, we believe focusing on model interpretability is key when developing stock selection strategies which combine a high number of factors. Interpretability means understanding why a model makes a certain prediction.
It’s even more important since:

• It can highlight the reasons why a given stock has been selected (or not) at a given date and to better understand the drivers of our selections (and their dynamics over time).
• It can help detect the presence of potential outliers in input data, which would require further investigation.
• It helps provide comfort with the predictions of the models and may also teach humans some basic interactions between certain input features and expected forward stock behaviors.
• It can identify potential biases when constructing the input dataset used for backtesting purposes.

However, we are using deep learning models which are, by their very nature, complex. Complex models generally allow to obtain good predictive accuracies, which often comes at the expense of interpretability.
Different approaches can be used. The most direct ones are perturbation-based methods; they directly compute the attribution of an input feature (or set of features) by removing or altering them (adding noise for example) and running a prediction (with a simple forward pass) on the new input, measuring the difference with the original prediction. Other popular approaches include the “gradient- based” methods; they compute the attribution by calculating the partial derivatives of the prediction with respect to each input feature and multiplying them with the input itself. Other popular methods include “relevance backpropagation” and “attention-based methods”, however we will not discuss these here.

Interpretability has been an increasing area of research recently, and many packages have been proposed to enhance model explainability, such as SHAP – SHapley Additive exPlanations, which is an attempt to unify some of these attribution methods.

The SHAP unified approach to interpretability

SHAP is presented as a unified approach to interpreting model predictions, we use it to measure variable importance, and to visualize interactions between some input variables and the response variable. This approach comes from game theory and is based on Shapley values (named after game theorist and Nobel prize winner Lloyd Shapley). They are used to fairly distribute credit or value to each individual player/participant in cooperative situations, or in short, to design a fair payout scheme. In summary, Shapley values calculate the importance of a feature by comparing what a model predicts with and without the feature. However, the order in which a model sees features may have an impact on its predictions, thus this comparison is done in every possible order.
In their paper presented at NIPS 2017, Lundberg and Lee present an approach to compute “Shapley Additive exPlanations”, an additive feature importance measure, and theoretical results (derived from a proof from game theory on the fair allocation of profits). This showed that, in the class of additive feature attribution methods, their approach is the only one which verifies a number of desirable properties, in particular local accuracy (the sum of all the feature importance should sum up to the total importance of the model) and consistency (if we change a model such that it relies more on a feature, then the attributed importance for that feature should not decrease).

The SHAP package: an efficient model interpretation

The SHAP library is designed to calculate Shapley values significantly faster than if a model prediction had to be calculated for every possible combination of features, by implementing model specific algorithms, which take advantage of different model’s structures. For example, SHAP’s integration with gradient boosted decision trees takes advantage of the hierarchy in a decision tree’s features to calculate the SHAP values, in polynomial time instead of the classical exponential runtime. For Python, the SHAP package provides Shapley values calculations and (partly interactive) plots to explore Shapley values across different features and/or observations.

Feature Importance

The figure below shows the distribution of Shapley values, for each feature, across the observations in a test sample used for a stock selection strategy that we’ve implemented at RAM. The most discriminative feature (factor) appears at the top of the chart. The sign shows whether the factor value moved the prediction – a measure of expected forward stock behavior – toward positive (positive SHAP) or negative (negative SHAP).

Feature Importance

The color reflects the order of magnitude of the factor values (low, medium, high). This way the plot immediately reveals that low (blue) values on the top seven factors deteriorate prospects for a given stocks.

It is also important to monitor input factor biases across a stock selection.

At RAM, we also gain comfort in the predictions of our stock selection models by looking at the average factor biases for a selection at a given date. They must make sense on average and be relatively stable historically. The chart below shows an example of average factor biases for a long-only stock selection model based on 51 factors within emerging markets (the higher the better), at a given date.

Factor Biases

Additional interactions charts are available, such as the one below representing SHAP value for the repayment / build-up of debt in our stock selection model.

We were mentioning previously that debt levels and debt dynamics have an unclear impact on expected forward stock behavior, as their interactions with future returns are not linear. The chart below shows that although the repayment / build-up of debt has a slight positive impact on expectations on average, the model distinguishes between stocks ranking positively on a growth factor (in red) and the others (in blue) : the latter ones are clearly expected to be rewarded by the market when they are able to repay their debt, or conversely, to be punished if they accumulate excessive debt.

Below we present another interaction chart showing that trading volume growth has an impact on future returns after conditioning on whether a stock ranks positively on a growth factor: