## Mixture Models for Forecasting Asset Returns

Asset return prediction is difficult. Most traditional time series techniques don’t work well for asset returns. One significant reason is that time series analysis (TSA) models require your data to be stationary. If it isn’t stationary, then you must transform your data until it is stationary.

That presents a problem.

In practice, we observe multiple phenomena that violate the rules of stationarity including non-linear processes, volatility clustering, seasonality, and autocorrelation. This renders traditional models mostly ineffective for our purposes.

### What are our options?

There are many algorithms to choose from, but few are flexible enough to address the challenges of predicting asset returns:

• mean and volatility changes through time
• sometimes future returns are correlated with past returns, sometimes not
• sometimes future volatility is correlated with past volatility, sometimes not
• non-linear behavior

To recap, we need a model framework that is flexible enough to (1) adapt to non-stationary processes and (2) provide a reasonable approximation of the non-linear process that is generating the data.

### Can Mixture Models offer a solution?

They have potential. First, they are based on several well-established concepts.

Markov models – These are used to model sequences where the future state depends only on the current state and not any past states. (memoryless processes)

Hidden Markov models – Used to model processes where the true state is unobserved (hidden) but there are observable factors that give us useful information to guess the true state.

Expectation-Maximization (E-M) – This is an algorithm that iterates between computing class parameters and maximizing the likelihood of the data given those parameters.

An easy way to think about applying mixture models to asset return prediction is to consider asset returns as a sequence of states or regimes. Each regime is characterized by its own descriptive statistics including mean and volatility. Example regimes could include low-volatility and high-volatility. We can also assume that asset returns will transition between these regimes based on probability. By framing the problem this way we can use mixture models, which are designed to try to estimate the sequence of regimes, each regime’s mean and variance, and the transition probabilities between regimes.

The most common is the Gaussian mixture model (GMM).

The underlying model assumption is that each regime is generated by a Gaussian process with parameters we can estimate. Under the hood, GMM employs an expectation-maximization algorithm to estimate regime parameters and the most likely sequence of regimes.

GMMs are flexible, generative models that have had success approximating non-linear data. Generative models are special in that they try to mimic the underlying data process such that we can create new data that should look like original data.

In this upcoming webinar on Mixture Models, we design and evaluate an example strategy to find out if Gaussian mixture models are useful for return prediction, and specifically for identifying market bottoms.

### Next Step

Join us for the webinar, “Can we use Mixture Models to Predict Market Bottoms?” on Tuesday 25th Apr, 8:30 AM MST | 8:00 PM IST. The webinar will be conducted by Brian Christopher, Quantitative researcher, Python developer, CFA charter holder, and founder of Blackarbs LLC, a quantitative research firm. Register now!

## Forecasting Markets using eXtreme Gradient Boosting (XGBoost)

In recent years, machine learning has been generating a lot of curiosity for its profitable application to trading. Numerous machine learning models like Linear/Logistic regression, Support Vector Machines, Neural Networks, Tree-based models etc. are being tried and applied in an attempt to analyze and forecast the markets. Researchers have found that some models have more success rate compared to other machine learning models. eXtreme Gradient Boosting also called XGBoost is one such machine learning model that has received rave from the machine learning practitioners.

In this post, we will cover the basics of XGBoost, a winning model for many kaggle competitions. We then attempt to develop an XGBoost stock forecasting model using the “xgboost” package in R programming.

### Basics of XGBoost and related concepts

Developed by Tianqi Chen, the eXtreme Gradient Boosting (XGBoost) model is an implementation of the gradient boosting framework. Gradient Boosting algorithm is a machine learning technique used for building predictive tree-based models. (Machine Learning: An Introduction to Decision Trees).

Boosting is an ensemble technique in which new models are added to correct the errors made by existing models. Models are added sequentially until no further improvements can be made.

The ensemble technique uses the tree ensemble model which is a set of classification and regression trees (CART). The ensemble approach is used because a single CART, usually, does not have a strong predictive power. By using a set of CART (i.e. a tree ensemble model) a sum of the predictions of multiple trees is considered.

Gradient boosting is an approach where new models are created that predict the residuals or errors of prior models and then added together to make the final prediction.

The objective of the XGBoost model is given as:

Obj = L +

Where,
L is the loss function which controls the predictive power, and
Ω is regularization component which controls simplicity and overfitting

The loss function (L) which needs to be optimized can be Root Mean Squared Error for regression, Logloss for binary classification, or mlogloss for multi-class classification.

The regularization component (Ω) is dependent on the number of leaves and the prediction score assigned to the leaves in the tree ensemble model.

It is called gradient boosting because it uses a gradient descent algorithm to minimize the loss when adding new models. The Gradient boosting algorithm supports both regression and classification predictive modeling problems.

### Sample XGBoost model:

We will use the “xgboost” R package to create a sample XGBoost model. You can refer to the documentation of the “xgboost” package here.

Install and load the xgboost library –

We install the xgboost library using the install.packages function. To load this package we use the library function. We also load other relevant packages required to run the code.

install.packages("xgboost")

library(quantmod); library(TTR); library(xgboost);

Create the input features and target variable – We take the 5-year OHLC and volume data of a stock and compute the technical indicators (input features) using this dataset. The indicators computed include Relative Strength Index (RSI), Average Directional Index (ADX), and Parabolic SAR (SAR). We create a lag in the computed indicators to avoid the look-ahead bias. This gives us our input features for building the XGBoost model. Since this is a sample model, we have included only a few indicators to build our set of input features.

# Read the stock data
symbol = "ACC"
fileName = paste(getwd(),"/",symbol,".csv",sep="") ;
colnames(df) = c("Date","Time","Close","High", "Low", "Open","Volume")

# Define the technical indicators to build the model
rsi = RSI(df$Close, n=14, maType="WMA") adx = data.frame(ADX(df[,c("High","Low","Close")])) sar = SAR(df[,c("High","Low")], accel = c(0.02, 0.2)) trend = df$Close - sar

# create a lag in the technical indicators to avoid look-ahead bias
adx$ADX = c(NA,head(adx$ADX,-1))
trend = c(NA,head(trend,-1))

Our objective is to predict the direction of the daily stock price change (Up/Down) using these input features. This makes it a binary classification problem. We compute the daily price change and assigned a positive 1 if the daily price change is positive. If the price change is negative, we assign a zero value.

# Create the target variable
price = df$Close-df$Open
class = ifelse(price > 0,1,0)

Combine the input features into a matrix – The input features and the target variable created in the above step are combined to form a single matrix. We use the matrix structure in the XGBoost model since the xgboost library allows data in the matrix format.

# Create a Matrix
model_df = data.frame(class,rsi,adx$ADX,trend) model = matrix(c(class,rsi,adx$ADX,trend), nrow=length(class))
model = na.omit(model)
colnames(model) = c("class","rsi","adx","trend")

Split the dataset into training data and test data – In the next step, we split the dataset into training and test data. Using this training and test dataset we create the respective input features set and the target variable.

# Split data into train and test sets
train_size = 2/3
breakpoint = nrow(model) * train_size

training_data = model[1:breakpoint,]
test_data = model[(breakpoint+1):nrow(model),]

# Split data training and test data into X and Y
X_train = training_data[,2:4] ; Y_train = training_data[,1]
class(X_train)[1]; class(Y_train)

X_test = test_data[,2:4] ; Y_test = test_data[,1]
class(X_test)[1]; class(Y_test)

Train the XGBoost model on the training dataset –

We use the xgboost function to train the model. The arguments of the xgboost function are shown in the picture below.

The data argument in the xgboost function is for the input features dataset. It accepts a matrix, dgCMatrix, or local data file. The nrounds argument refers to the max number of iterations (i.e. the number of trees added to the model). The obj argument refers to the customized objective function. It returns gradient and second order gradient with given prediction and dtrain.

# Train the xgboost model using the "xgboost" function
dtrain = xgb.DMatrix(data = X_train, label = Y_train)
xgModel = xgboost(data = dtrain, nround = 5, objective = "binary:logistic")

Output – The output is the classification error on the training data set.

Cross-validation

We can also use the cross-validation function of xgboost i.e. xgb.cv. In this case, the original sample is randomly partitioned into nfold equal size subsamples. Of the nfold subsamples, a single subsample is retained as the validation data for testing the model, and the remaining (nfold – 1) subsamples are used as training data. The cross-validation process is then repeated nrounds times, with each of the nfold subsamples used exactly once as the validation data.

# Using cross validation
dtrain = xgb.DMatrix(data = X_train, label = Y_train)
cv = xgb.cv(data = dtrain, nround = 10, nfold = 5, objective = "binary:logistic")

Output – The xgb.cv returns a data.table object containing the cross validation results.

Make predictions on the test data

To make predictions on the unseen data set (i.e. the test data), we apply the trained XGBoost model on it which gives a series of numbers.

# Make the predictions on the test data
preds = predict(xgModel, X_test)

# Determine the size of the prediction vector
print(length(preds))

# Limit display of predictions to the first 6
print(head(preds))

Output –

These numbers do not look like binary classification {0, 1}. We have to, therefore, perform a simple transformation before we are able to use these results. In the example code shown below, we are comparing the predicted number to the threshold of 0.5. The threshold value can be changed depending upon the objective of the modeler, the metrics (e.g. F1 score, Precision, Recall) that the modeler wants to track and optimize.

prediction = as.numeric(preds > 0.5)
print(head(prediction))

Output –

Measuring model performance

Different evaluation metrics can be used to measure the model performance. In our example, we will compute a simple metric, the average error. It compares the predicted score with the threshold of 0.50.

For example: If the predicted score is less than 0.50, then the (preds > 0.5) expression gives a value of 0. If this value is not equal to the actual result from the test data set, then it is taken as a wrong result.

We compare all the preds with the respective data points in the Y_test and compute the average error. The code for measuring the performance is given below. Alternatively, we can use hit rate or create a confusion matrix to measure the model performance.

# Measuring model performance
error_value = mean(as.numeric(preds > 0.5) != Y_test)
print(paste("test-error=", error_value))

Output –

Plot the feature importance set – We can find the top important features in the model by using the xgb.importance function.

# View feature importance from the learnt model
importance_matrix = xgb.importance(model = xgModel)
print(importance_matrix)

Plot the XGBoost Trees

Finally, we can plot the XGBoost trees using the xgb.plot.tree function. To limit the plot to a specific number of trees, we can use the n_first_tree argument. If NULL, all trees of the model are plotted.

# View the trees from a model
xgb.plot.tree(model = xgModel)

# View only the first tree in the XGBoost model
xgb.plot.tree(model = xgModel, n_first_tree = 1)

### Conclusion

This post covered the popular XGBoost model along with a sample code in R programming to forecast the daily direction of the stock price change. Readers can catch some of our previous machine learning blogs (links given below). We will be covering more machine learning concepts and techniques in our coming posts.

### Next Step

If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to be a successful trader. Enroll now!

• ACC.csv

## Mean Reversion in Time Series

Time series data is simply a collection of observations generated over time. For example, the speed of a race car at each second, daily temperature, weekly sales figures, stock returns per minute, etc. In the financial markets, a time series tracks the movement of specific data points, such as a security’s price over a specified period of time, with data points recorded at regular intervals. A time series can be generated for any variable that is changing over time. Time series analysis comprises of techniques for analyzing time series data in an attempt to extract useful statistics and identify characteristics of the data. Time series forecasting is the use of a mathematical model to predict future values based on previously observed values in the time series data.

The graph shown below represents the daily closing price of Aluminium futures over a period of 93 trading days, which is a time series.

### Mean Reversion

Mean reversion is the theory which suggests that prices, returns, or various economic indicators tend to move to the historical average or mean over time. This theory has led to many trading strategies which involve the purchase or sale of a financial instrument whose recent performance has greatly differed from their historical average without any apparent reason. For example, let the price of gold increase on average by INR 10 every day and one day the price of gold increases by INR 40 without any significant news or factor behind this rise, then by the mean reversion principle we can expect the price of gold to fall in the coming days such that the average change in price of gold remains the same. In such a case, the mean reversionist would sell gold, speculating the price to fall in the coming days. Thus, making profits by buying the same amount of gold he had sold earlier, now at a lower price.

A mean-reverting time series has been plotted below, the horizontal black line represents the mean and the blue curve is the time series which tends to revert back to the mean.

A collection of random variables is defined to be a stochastic or random process. A stochastic process is said to be stationary if its mean and variance are time invariant (constant over time). A stationary time series will be mean reverting in nature, i.e. it will tend to return to its mean and fluctuations around the mean will have roughly equal amplitudes. A stationary time series will not drift too far away from its mean because of its finite constant variance. A non-stationary time series, on the contrary, will have a time varying variance or a time varying mean or both, and will not tend to revert back to its mean. In the financial industry, traders take advantage of stationary time series by placing orders when the price of a security deviates considerably from its historical mean, speculating the price to revert back to its mean. They start by testing for stationarity in a time series. Financial data points, such as prices, are often non-stationary, i.e. they have means and variances that change over time. Non-stationary data tends to be unpredictable and cannot be modeled or forecasted. A non-stationary time series can be converted into a stationary time series by either differencing or detrending the data. A random walk (the movements of an object or changes in a variable that follow no discernible pattern or trend) can be transformed into a stationary series by differencing (computing the difference between Yt and Yt -1). The disadvantage of this process is that it results in losing one observation each time the difference is computed. A non-stationary time series with a deterministic trend can be converted into a stationary time series by detrending (removing the trend). Detrending does not result in loss of observations. A linear combination of two non-stationary time series can also result in a stationary, mean-reverting time series. The time series (integrated of at least order 1), which can be linearly combined to result in a stationary time series are said to be cointegrated.

Shown below is a plot of a non-stationary time series with a deterministic trend (Yt = α + βt + εt) represented by the blue curve and its detrended stationary time series (Yt – βt = α + εt) represented by the red curve.

### Trading Strategies based on Mean Reversion

One of the simplest mean reversion related trading strategies is to find the average price over a specified period, followed by determining a high-low range around the average value from where the price tends to revert back to the mean. The trading signals will be generated when these ranges are crossed – placing a sell order when the range is crossed on the upper side and a buy order when the range is crossed on the lower side. The trader takes contrarian positions, i.e. goes against the movement of prices (or trend), expecting the price to revert back to the mean. This strategy looks too good to be true and it is, it faces severe obstacles. The lookback period of the moving average might contain a few abnormal prices which are not characteristic to the dataset, this will cause the moving average to misrepresent the security’s trend or the reversal of a trend. Secondly, it might be evident that the security is overpriced as per the trader’s statistical analysis, yet he cannot be sure that other traders have made the exact same analysis. Because other traders don’t see the security to be overpriced, they would continue buying the security which would push the prices even higher. This strategy would result in losses if such a situation arises.

Pairs Trading is another strategy that relies on the principle of mean reversion. Two co-integrated securities are identified, the spread between the price of these securities would be stationary and hence mean reverting in nature. An extended version of Pairs Trading is called Statistical Arbitrage, where many co-integrated pairs are identified and split into buy and sell baskets based on the spreads of each pair. The first step in a Pairs Trading or Stat Arb model is to identify a pair of co-integrated securities. One of the commonly used tests for checking co-integration between a pair of securities is the Augmented Dickey-Fuller Test (ADF Test). It tests the null hypothesis of a unit root being present in a time series sample. A time series which has a unit root, i.e. 1 is a root of the series’ characteristic equation, is non-stationary. The augmented Dickey-Fuller statistic, also known as t-statistic, is a negative number. The more negative it is, the stronger the rejection of the null hypothesis that there is a unit root at some level of confidence, which would imply that the time series is stationary. The t-statistic is compared with a critical value parameter, if the t-statistic is less than the critical value parameter then the test is positive and the null hypothesis is rejected.

### Co-integration check – ADF Test

Consider the Python code shown below for checking co-integration:

We start by importing relevant libraries, followed by fetching financial data for two securities using the quandl.get() function. Quandl provides financial and economic data directly in Python by importing the Quandl library. In this example, we have fetched data for Aluminium and Lead futures from MCX. We then print the first five rows of the fetched data using the head() function, in order to view the data being pulled by the code. Using the statsmodels.api library, we compute the Ordinary Least Squares regression on the closing price of the commodity pair and store the result of the regression in the variable named ‘result’. Next, using the statsmodels.tsa.stattools library, we run the adfuller test by passing the residual of the regression as the input and store the result of this computation the array “c_t”. This array contains values like the t-statistic, p-value, and critical value parameters. Here, we consider a significance level of 0.1 (90% confidence level). “c_t[0]” carries the t-statistic, “c_t[1]” contains the p-value and “c_t[4]” stores a dictionary containing critical value parameters for different confidence levels. For co-integration we consider two conditions, firstly we check whether the t-stat is lesser than the critical value parameter (c_t[0] <= c_t[4][‘10%’]) and secondly whether the p-value is lesser than the significance level (c_t[1] <= 0.1). If both these conditions are true, we print that the “Pair of securities is co-integrated”, else print that the “Pair of securities is not cointegrated”.

### Output

To know more about Pairs Trading, Statistical Arbitrage and the ADF test you can check out the self-paced online certification course on “Statistical Arbitrage Trading“ offered jointly by QuantInsti and MCX to learn how to trade Statistical Arbitrage strategies using Python and Excel.

### Next Step

If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to build a promising career in algorithmic trading. Enroll now!

## Forecasting Stock Returns using ARIMA model

“Prediction is very difficult, especially about the future”. Many of you must have come across this famous quote by Neils Bohr, a Danish physicist. Prediction is the theme of this blog post. In this post, we will cover the popular ARIMA forecasting model to predict returns on a stock and demonstrate a step-by-step process of ARIMA modeling using R programming.

### What is a forecasting model in Time Series?

Forecasting involves predicting values for a variable using its historical data points or it can also involve predicting the change in one variable given the change in the value of another variable. Forecasting approaches are primarily categorized into qualitative forecasting and quantitative forecasting. Time series forecasting falls under the category of quantitative forecasting wherein statistical principals and concepts are applied to a given historical data of a variable to forecast the future values of the same variable. Some time series forecasting techniques used include:

• Autoregressive Models (AR)
• Moving Average Models (MA)
• Seasonal Regression Models
• Distributed Lags Models

### What is Autoregressive Integrated Moving Average (ARIMA)?

ARIMA stands for Autoregressive Integrated Moving Average. ARIMA is also known as Box-Jenkins approach. Box and Jenkins claimed that non-stationary data can be made stationary by differencing the series, Yt. The general model for Yt is written as,

Yt1Yt1 2Yt2…ϕpYtpt + θ1ϵt1+ θ2ϵt2 +…θqϵtq

Where, Yt is the differenced time series value, ϕ and θ are unknown parameters and ϵ are independent identically distributed error terms with zero mean. Here, Yt is expressed in terms of its past values and the current and past values of error terms.

The ARIMA model combines three basic methods:

• AutoRegression (AR) – In auto-regression the values of a given time series data are regressed on their own lagged values, which is indicated by the “p” value in the model.
• Differencing (I-for Integrated) – This involves differencing the time series data to remove the trend and convert a non-stationary time series to a stationary one. This is indicated by the “d” value in the model. If d = 1, it looks at the difference between two time series entries, if d = 2 it looks at the differences of the differences obtained at d =1, and so forth.
• Moving Average (MA) – The moving average nature of the model is represented by the “q” value which is the number of lagged values of the error term.

This model is called Autoregressive Integrated Moving Average or ARIMA(p,d,q) of Yt.  We will follow the steps enumerated below to build our model.

Step 1: Testing and Ensuring Stationarity

To model a time series with the Box-Jenkins approach, the series has to be stationary. A stationary time series means a time series without trend, one having a constant mean and variance over time, which makes it easy for predicting values.

Testing for stationarity – We test for stationarity using the Augmented Dickey-Fuller unit root test. The p-value resulting from the ADF test has to be less than 0.05 or 5% for a time series to be stationary. If the p-value is greater than 0.05 or 5%, you conclude that the time series has a unit root which means that it is a non-stationary process.

Differencing – To convert a non-stationary process to a stationary process, we apply the differencing method. Differencing a time series means finding the differences between consecutive values of a time series data. The differenced values form a new time series dataset which can be tested to uncover new correlations or other interesting statistical properties.

We can apply the differencing method consecutively more than once, giving rise to the “first differences”, “second order differences”, etc.

We apply the appropriate differencing order (d) to make a time series stationary before we can proceed to the next step.

Step 2: Identification of p and q

In this step, we identify the appropriate order of Autoregressive (AR) and Moving average (MA) processes by using the Autocorrelation function (ACF) and Partial Autocorrelation function (PACF).  Please refer to our blog, “Starting out with Time Series” for an explanation of ACF and PACF functions.

Identifying the p order of AR model

For AR models, the ACF will dampen exponentially and the PACF will be used to identify the order (p) of the AR model. If we have one significant spike at lag 1 on the PACF, then we have an AR model of the order 1, i.e. AR(1). If we have significant spikes at lag 1, 2, and 3 on the PACF, then we have an AR model of the order 3, i.e. AR(3).

Identifying the q order of MA model

For MA models, the PACF will dampen exponentially and the ACF plot will be used to identify the order of the MA process. If we have one significant spike at lag 1 on the ACF, then we have an MA model of the order 1, i.e. MA(1). If we have significant spikes at lag 1, 2, and 3 on the ACF, then we have an MA model of the order 3, i.e. MA(3).

Step 3: Estimation and Forecasting

Once we have determined the parameters (p,d,q) we estimate the accuracy of the ARIMA model on a training data set and then use the fitted model to forecast the values of the test data set using a forecasting function. In the end, we cross check whether our forecasted values are in line with the actual values.

### Building ARIMA model using R programming

Now, let us follow the steps explained to build an ARIMA model in R. There are a number of packages available for time series analysis and forecasting. We load the relevant R package for time series analysis and pull the stock data from yahoo finance.

library(quantmod);library(tseries);
library(timeSeries);library(forecast);library(xts);

# Pull data from Yahoo finance
getSymbols('TECHM.NS', from='2012-01-01', to='2015-01-01')

# Select the relevant close price series
stock_prices = TECHM.NS[,4]

In the next step, we compute the logarithmic returns of the stock as we want the ARIMA model to forecast the log returns and not the stock price. We also plot the log return series using the plot function.

# Compute the log returns for the stock
stock = diff(log(stock_prices),lag=1)
stock = stock[!is.na(stock)]

# Plot log returns
plot(stock,type='l', main='log returns plot')

Next, we call the ADF test on the returns series data to check for stationarity. The p-value of 0.01 from the ADF test tells us that the series is stationary. If the series were to be non-stationary, we would have first differenced the returns series to make it stationary.

# Conduct ADF test on log returns series


In the next step, we fixed a breakpoint which will be used to split the returns dataset in two parts further down the code.

# Split the dataset in two parts - training and testing
breakpoint = floor(nrow(stock)*(2.9/3))

We truncate the original returns series till the breakpoint, and call the ACF and PACF functions on this truncated series.

# Apply the ACF and PACF functions
par(mfrow = c(1,1))
acf.stock = acf(stock[c(1:breakpoint),], main='ACF Plot', lag.max=100)
pacf.stock = pacf(stock[c(1:breakpoint),], main='PACF Plot', lag.max=100)

We can observe these plots and arrive at the Autoregressive (AR) order and Moving Average (MA) order.

We know that for AR models, the ACF will dampen exponentially and the PACF plot will be used to identify the order (p) of the AR model. For MA models, the PACF will dampen exponentially and the ACF plot will be used to identify the order (q) of the MA model. From these plots let us select AR order = 2 and MA order = 2. Thus, our ARIMA parameters will be (2,0,2).

Our objective is to forecast the entire returns series from breakpoint onwards. We will make use of the For Loop statement in R and within this loop we will forecast returns for each data point from the test dataset.

In the code given below, we first initialize a series which will store the actual returns and another series to store the forecasted returns.  In the For Loop, we first form the training dataset and the test dataset based on the dynamic breakpoint.

We call the arima function on the training dataset for which the order specified is (2, 0, 2). We use this fitted model to forecast the next data point by using the forecast.Arima function. The function is set at 99% confidence level. One can use the confidence level argument to enhance the model. We will be using the forecasted point estimate from the model. The “h” argument in the forecast function indicates the number of values that we want to forecast, in this case, the next day returns.

We can use the summary function to confirm the results of the ARIMA model are within acceptable limits. In the last part, we append every forecasted return and the actual return to the forecasted returns series and the actual returns series respectively.

# Initialzing an xts object for Actual log returns
Actual_series = xts(0,as.Date("2014-11-25","%Y-%m-%d"))

# Initialzing a dataframe for the forecasted return series
forecasted_series = data.frame(Forecasted = numeric())

for (b in breakpoint:(nrow(stock)-1)) {

stock_train = stock[1:b, ]
stock_test = stock[(b+1):nrow(stock), ]

# Summary of the ARIMA model using the determined (p,d,q) parameters
fit = arima(stock_train, order = c(2, 0, 2),include.mean=FALSE)
summary(fit)

# plotting a acf plot of the residuals
acf(fit$residuals,main="Residuals plot") # Forecasting the log returns arima.forecast = forecast.Arima(fit, h = 1,level=99) summary(arima.forecast) # plotting the forecast par(mfrow=c(1,1)) plot(arima.forecast, main = "ARIMA Forecast") # Creating a series of forecasted returns for the forecasted period forecasted_series = rbind(forecasted_series,arima.forecast$mean[1])
colnames(forecasted_series) = c("Forecasted")

# Creating a series of actual returns for the forecasted period
Actual_return = stock[(b+1),]
Actual_series = c(Actual_series,xts(Actual_return))
rm(Actual_return)

print(stock_prices[(b+1),])
print(stock_prices[(b+2),])

}

Before we move to the last part of the code, let us check the results of the ARIMA model for a sample data point from the test dataset.

From the coefficients obtained, the return equation can be written as:

Yt = 0.6072*Y(t-1)  – 0.8818*Y(t-2) – 0.5447ε(t-1) + 0.8972ε(t-2)

The standard error is given for the coefficients, and this needs to be within the acceptable limits. The Akaike information criterion (AIC) score is a good indicator of the ARIMA model accuracy. Lower the AIC score better the model. We can also view the ACF plot of the residuals; a good ARIMA model will have its autocorrelations below the threshold limit. The forecasted point return is -0.001326978, which is given in the last row of the output.

Let us check the accuracy of the ARIMA model by comparing the forecasted returns versus the actual returns. The last part of the code computes this accuracy information.

# Adjust the length of the Actual return series
Actual_series = Actual_series[-1]

# Create a time series object of the forecasted series
forecasted_series = xts(forecasted_series,index(Actual_series))

# Create a plot of the two return series - Actual versus Forecasted
plot(Actual_series,type='l',main='Actual Returns Vs Forecasted Returns')
lines(forecasted_series,lwd=1.5,col='red')
legend('bottomright',c("Actual","Forecasted"),lty=c(1,1),lwd=c(1.5,1.5),col=c('black','red'))

# Create a table for the accuracy of the forecast
comparsion = merge(Actual_series,forecasted_series)
comparsion$Accuracy = sign(comparsion$Actual_series)==sign(comparsion$Forecasted) print(comparsion) # Compute the accuracy percentage metric Accuracy_percentage = sum(comparsion$Accuracy == 1)*100/length(comparsion$Accuracy) print(Accuracy_percentage) If the sign of the forecasted return equals the sign of the actual returns we have assigned it a positive accuracy score. The accuracy percentage of the model comes to around 55% which looks like a decent number. One can try running the model for other possible combinations of (p,d,q) or instead use the auto.arima function which selects the best optimal parameters to run the model. ### Conclusion To conclude, in this post we covered the ARIMA model and applied it for forecasting stock price returns using R programming language. We also crossed checked our forecasted results with the actual returns. In our upcoming posts, we will cover other time series forecasting techniques and try them in Python/R programming languages. ### Next Step If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to build a promising career in algorithmic trading. Enroll now! ## Performance Metrics, Risk Metrics, and Strategy Optimization – An Overview The performance of a trading strategy is measured with a set of parameters. For example, if you are trading in equity then your returns are compared against the benchmark index. The consistency of returns of the strategy also proves to be a significant factor. (more…) ## Pairs Trading on ETF – EPAT Project Work This article is the final project submitted by the author as part of his coursework in Executive Programme in Algorithmic Trading (EPAT™) at QuantInsti. You can check out our Projects page and have a look at what our students are building after reading this article. ### About the Author Edmund Ho did his Bachelors in commerce from University of British Columbia, He completed his Masters in Investment Management from Hong Kong University of Science and Technology. Edmund was enrolled in the 27th Batch of EPAT™, and this report is part of his final project work. ### Project Summary ETFs are very popular for pairs trading simply because it eliminates the firm-specific factors. On top of that, most of the ETFs are short-able so we don’t have to worry about the short constraint. In this project, we are trying to build a portfolio using 3 ETF pairs in oil (USO vs XLE), technology (XLK vs IYW), and financial sectors (XLF vs PSCF). Over the long run, the overall performance of the miners is highly correlated with the commodities. In short term, they may have divergences due to individual company’s performance or overall equity market performance, and hence the short term arbitrage opportunities may exist. In technology sector, we attempt to seek mispricing on both large cap technology ETFs. Last, we attempt to see if arbitrage opportunity exists between the large and mid-cap financial ETFs. ### Pair 1 – Oil Sector USO vs XLE #### Cointegration Test The above charts were generated in R Studio. The in sample data generated between Jan 1st, 2011 and Dec 31, 2014. First, we plot the price for the pairs and it gives us an impression that both price series are quite similar. Then we perform the regression analysis for USO vs XLE (return USO = Beta * Return XLE + Residual) and find the beta or hedge ratio to be 0.7493. Next, we apply the hedge ratio to generate the spread returns. We can see the spread returns deviate closely around 0, which shows the characteristic of cointegrating pattern. At last, we apply the Augmented Dickey-Fuller test with a confidence level at 0.2 and check if the pairs pass the ADF test. The results are as follow: ###### Augmented Dickey-Fuller Test data:(spread) Dickey-Fuller = -3.0375, Lag order = 0, p-value = 0.1391 alternative hypothesis: stationary [1] “The spread is likely Cointegrated with a pvalue of 0.139136738842547” With the p-value of 0.1391, the pairs satisfied the cointegration test, and we will go ahead and back-test the pairs in next section. #### Strategy Back-Testing The above back-testing result were generated in R Studio. The back-testing period was using the in-sample data similar to the cointegration test. Our trading strategy is relatively simple as follow: • If the spread is greater than +/- 1.5 standard deviations of its rolling lookback period of 120 days’ standard deviation, then we go short / long accordingly. • At all time, only 1 open position • Close the long/short position when the spread reverts to its mean/moving average. The above back-testing result were generated in R-Studio using the PerformanceAnalystics package. During the in sample back-testing period, the strategy achieved a cumulated return of 121.03% where the SPY (S&P500) had a cumulative return of 61.78%. This translated into an annualized return of 22% and 12.82%, respectively. In terms of risk analysis, the strategy had much lower annualized standard deviation of 11.63% vs. 15.35% in SPY. The worst drawdown percentage for the strategy was 6.39% vs. 19.42% in SPY. The annualized Sharpe ratio was superior in our strategy at 1.89 vs. 0.835 in SPY. Please note that all of the above calculations did NOT factor into transaction cost. #### Out of sample test For the out of sample period between Jan 1st 2015 and Dec 31, 2015, the pairs did not pass the ADF test suggested by a high p-Value at 0.3845. The phenomenon could be explained by the sharp decline in cruel oil price where the equity market was persisting in an uptrend. If we look at the spread returns, they first seem to be cointegrating around 0 but with a much larger deviation suggested by the chart below. The actual spread obviously did not suggest a cointegrating pattern as indicated by the high p-Value. Next, we will go ahead and back-test the same strategy using the out of sample data despite the pair fails the cointegration test. The hedge ratio was found to be 1.1841. The key back-testing results were generated in R-Studio as follow: USO and XLE Stat Arb SPYAnnualized Return 0.09862893 -0.007623957 USO and XLE Stat Arb SPYCumulative Return 0.09821892 -0.007593818 USO and XLE Stat Arb SPYAnnualized Sharpe Ratio (Rf=0%) 0.5632756 -0.04884633 USO and XLE Stat Arb SPYAnnualized Standard Deviation 0.1750989 0.1560804 USO and XLE Stat Arb SPYWorst Drawdown 0.1706643 0.1228571 At a first glance, the strategy seems outperform the SPY in all aspects, but due to the lookback period which was set same as in-sample back-testing data (120 days for consistency), this strategy only had 1 trade during the out of sample period, which may not reflect the situation going forward. However, this shows that it is not necessary to have a perfect cointegrating pairs in order to extract profit opportunities. In reality, only a few perfect pairs would pass the test. ### Pair 2 – Large Cap Technology XLK vs. IYW #### Cointegration Test It should not be a surprise that XLK and IYW has a strong linear relationship as demonstrated in the regression analysis with a hedge ratio at 0.903. The two large cap technology ETFs are very similar in nature except for its size, volume, expense ratio, etc. However, if we take a closer look at the actual return spreads, it doesn’t seem to satisfy the cointegration test. If we run the ADF test, the result shows they are not likely to be cointegrated with a p-Value at 0.5043. Augmented Dickey-Fuller Test data: (spread) Dickey-Fuller = -2.1748, Lag order = 0, p-value = 0.5043 alternative hypothesis: stationary [1] “The spread is likely NOT Cointegrated with a pvalue of 0.504319921216107” The purpose to run the strategy in this pair is to see if there is any mispricing (short term deviation) in the pair in order to profit from it. In the USO and XLE example, we observed that profit opportunity may still exist despite the pair failing the cointegration test. Here, we will go ahead to test the pair and see if any profit opportunity exists. #### Back-testing Result  XLK and IYW Stat Arb SPYAnnualized Return -0.003581305 0.1282006 XLK and IYW Stat Arb SPYCumulative Return -0.01420635 0.6177882 XLK and IYW Stat Arb SPYAnnualized Sharpe Ratio (Rf=0%) -0.2839318 0.8347157 XLK and IYW Stat Arb SPYAnnualized Standard Deviation 0.01261326 0.153586 XLK and IYW Stat Arb SPYWorst Drawdown 0.02235892 0.1942388 The back-testing results illustrated that the strategy performed very poorly during the back-testing period between Jan 1st 2011 and Dec 31, 2014. This demonstrates that the two ETFs are highly correlated to each other and it’s very hard to extract profit opportunity from them. In order for a statistical arbitrage strategy to work, we need a pair with some volatility in their spreads but they should eventually show a mean-reverting pattern. In next section, we will perform the same analysis on the Financial ETFs. ### Pair 3 – Financial Sectors XLF vs. PSCF Cointegration Test In this pair, we attempt to seek a trading opportunity between the large financial cap ETF XLF and the small financial cap ETF PSCF. From the price series they both show very similar pattern. In terms of regression analysis, they obviously show a strong correlation with a hedge ratio of 0.9682. The spread return also illustrates some cointegrating pattern with the spread deviating around 0. The ADF test with test value set at 80% confidence shows the pair is likely to be cointegrated with a p-value at 0.1026. Augmented Dickey-Fuller Test data:(spread) Dickey-Fuller = -3.1238, Lag order = 0, p-value = 0.1026 alternative hypothesis: stationary [1] “The spread is likely Cointegrated with a pvalue of 0.102608136882834” #### Back-testing Result XLF and PSCF Stat Arb SPYAnnualized Return 0.01212355 0.1282006 XLF and PSCF Stat Arb SPYCumulative Return 0.04923268 0.6177882 XLF and PSCF Stat Arb SPYAnnualized Sharpe Ratio (Rf=0%) 0.1942203 0.8347157 XLF and PSCF Stat Arb SPYAnnualized Standard Deviation 0.06242163 0.153586 XLF and PSCF Stat Arb SPYWorst Drawdown 0.07651392 0.1942388 Although the pair satisfied the cointegration test with a low p-value, the back-testing results demonstrated a below average performance when we compare to the index return. ### Conclusion In this project, we chose 3 different pairs of ETFs to back test our simple mean-reverting strategy. The back test results show superior performance on USO/XLE, but not on the other two pairs. We can conclude that in order for the pair trading strategy to work, we do not need a pair that shows strong linear relationship, but a long term mean reverting pattern is essential to obtain a decent result. In the pair XLK/IYW, we attempt to seek for mispricing between the two issuers, however, in an efficient ETF market in US, mispricing on such big ETFs is very rare, hence this strategy performs very poorly on this pair. On the other hand, the correlation and cointegration test on the pair XLF/PSCF illustrate the pair is an ideal candidate to trade the statistical arbitrage strategy, however, the back-testing results show the other way around. Any statistical arbitrage strategy we are essentially playing with the volatility, but if there is not enough volatility around the spreads to begin with, like the pair in XLK/IYW, the profit opportunity is trivial. In the pair USO/XLE, the volatility around the spreads is ideal and the cointegration test shows the pair has a mean-reverting pattern, therefore it is not a surprise this pair prevails in the back-testing results. ### Next Step Read about other strategies in this article on Algorithmic Trading Strategy Paradigms. If you also want to learn more about Algorithmic Trading, then click here. • Project_Download.rar • Project_Cointegration_Test.R • Project_Backtest_Test.R ## Sentiment Analysis on News Articles using Python Know how to perform sentiment analysis on news articles using Python Programming Language In our previous post on sentiment analysis we briefly explained sentiment analysis within the context of trading, and also provided a model code in R. The R model was applied on an earnings call conference transcript of an NSE listed company, and the output of the model was compared with the quarterly earnings numbers, and by charting the one-month stock price movement post the earnings call date. QuantInsti also conducted a webinar on “Quantitative Trading Using Sentiment Analysis” where Rajib Ranjan Borah, Director & Co-founder, iRageCapital and QuantInsti, covered important aspects of the topic in detail, and is a must watch for all enthusiast wanting to learn & apply quantitative trading strategies using sentiment analysis. Taking these initiatives on sentiment analysis forward, in this blog post we attempt to build a Python model to perform sentiment analysis on news articles that are published on a financial markets portal. We will build a basic model to extract the polarity (positive or negative) of the news articles. In Rajib’s Webinar, one of the slides details the sensitivity of different sectors to company and sectorial news. In the slide, the Pharma sector ranks at the top as the most sensitive sector, and in this blog we will apply our sentiment analysis model on specific news articles pertaining to select Indian Pharma companies. We will determine the polarity, and then check how the market reacted to these news. For our sample model, we have taken ten Indian Pharma companies that make the NIFTY Pharma index. ### Building the Model Now, let us dive straight in and build our model. We use the following Python libraries to build the model: • Requests • Beautiful Soup • Pattern #### Step 1: Create a list of the news section URL of the component companies We identify the component companies of the NIFTY Pharma index, and create a dictionary in python which contains the company names as the keys, while the dictionary values comprise the respective company abbreviation used by the financial portal site to form the news section URL. Using this dictionary we create a python list of the news section URLs for the all components companies. import csv import time import requests from bs4 import BeautifulSoup from pattern.en import ngrams Base_url = "http://www.moneycontrol.com" # Build a dictionary of companies and their abbreviated names companies = {'cadilahealthcare':'CHC','piramalenterprises':'PH05', 'glenmarkpharma':'GP08','glaxosmithklinepharmaceuticals':'GSK', 'sunpharmaceuticalindustries':'SPI','lupinlaboratories':'LL', 'cipla':'C','aurobindopharma':'AP', 'drreddyslaboratories':'DRL','divislaboratories':'DL03'} # Create a list of the news section urls of the respective companies url_list = ['http://www.moneycontrol.com/company-article/{}/news/{}#{}'.format(k,v,v) for k,v in companies.iteritems()] print url_list  #### Step 2: Extract the relevant news articles web-links from the company’s news section page Using the Python list of the news section URLs, we run a Python For loop which pings the portal with every URL in our Python list. We use the requests.get function from the Python requests library (which is a simple HTTP library). The requests module allows you to send HTTP/1.1 requests. One can add headers, form data, multipart files, and parameters with simple Python dictionaries, and also access the response data in the same way. The text of the response object is then applied to create a Beautiful Soup object. Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with a given parser to provide for ways of navigating, searching, and modifying the parse tree. HTML parsing basically means taking in the HTML code and extracting relevant information like the title of the page, paragraphs in the page, headings, links, bold text etc. The news section webpage on the financial portal site contains 20 news articles per page. We target only the first page of the news section, and our objective is to extract the links for all the news articles that appear on the first page using the parsed HTML. We inspect the HTML, and use the find_all method in the code to search for a tag that has the CSS class name as “arial11_summ”. This enables us to extract all the 20 web-links. Fortunes of the R&D intensive Indian Pharma sector are driven by sales in the US market and by approvals/rejections of new drugs by US Food and Drug Administration (USFDA). Hence, we will select only those news articles pertaining to the US Food and Drug Administration (USFDA) and the US market. Using keywords like “US”, “USA”, and “USFDA” in a If statement which is nested within the Python For Loop, we get us our final list of all the relevant news articles. # Create an empty list which will contain the selected news articles List_of_links = [] # Extract the relevant news articles weblinks from the news section of selected companies for urls in url_list: html = requests.get(urls) soup = BeautifulSoup(html.text,'html.parser') # Create a BeautifulSoup object # Retrieve a list of all the links and the titles for the respective links word1,word2,word3 = "US","USA","USFDA" sub_links = soup.find_all('a', class_='arial11_summ') for links in sub_links: sp = BeautifulSoup(str(links),'html.parser') # first convert into a string tag = sp.a if word1 in tag['title'] or word2 in tag['title'] or word3 in tag['title']: category_links = Base_url + tag["href"] List_of_links.append(category_links) time.sleep(3) # Print the select list of news articles weblinks #for p in List_of_links: print p #### Step 3: Remove the duplicate news articles based on news title It may happen that the financial portal publishes important news articles pertaining to the overall pharma sector on every pharma company’s news section webpage. Hence, it becomes necessary to weed out the duplicate news articles that appear in our Python list before we run our sentiment analysis model. We call the set function on our Python list which we generated in Step 2 to give us a list with no duplicate news articles. # Remove the duplicate news articles based on News Title unique_links = list(set(List_of_links)) for q in unique_links: print q # Create a dictionary of positive/negative words related to the Pharma Sector reader = csv.reader(open('dict.csv', 'r')) pharma_dict = dict((rows[0],rows[1]) for rows in reader) # Creating an empty list which will be filled later with news article links, and Polarity values (pos/neg) df =[] print df  #### Step 4: Extract the main text from the selected news articles In this step we run a Python For Loop and for every news article URL, we call the requests.get() on the URL, and then convert the text of response object into a Beautiful Soup object. Finally, we extract the main text using the find and get_text methods from the Beautiful Soup module. # Open the choosen news articles and extract the main text for selected_links in unique_links: results_url = selected_links #print results_url results = requests.get(results_url) results_text = BeautifulSoup(results.text) extract_text = results_text.find(class_='arti_cont') final_text = extract_text.get_text() #### Step 5: Pre-processing the extracted text We will use the n-grams function from the Pattern module to pre-process our extracted text. The ngrams() function returns a list of n-grams (i.e., tuples of n successive words) from the given string. Since we are building a simple model, we use a value of one for the n argument in the n-grams function. The Pattern module contains other useful functions for pre-processing like parse, tokenize, tag etc. which can be explored to conduct an in-depth analysis.  # Pre-processing the extracted text using ngrams function from the pattern package final_text1 = ngrams(final_text, n=1, punctuation=".,;:!?()[]{}''\"@#$^&*+-|=~_", continuous=False)
#print final_text1

#### Step 6: Compute the Sentiment analysis score using a simple dictionary approach

To compute the overall polarity of a news article we use the dictionary method. In this approach a list of positive/negative words help determine the polarity of a given text. This dictionary is created using the words that are specific to the Pharma sector. The code checks for positive/negative matching words from the dictionary with the processed text from the news article.

  # Checking if any of the words in the news article text matches with the words in the Pharma dictionary(pos/neg)
new_dict = {}
positive_score,negative_score = 0,0

for x in final_text1:
if pharma_dict.has_key(x[0]):
new_dict[x[0]] = pharma_dict[x[0]]

positive_list = [] ; negative_list = [];
for key, value in new_dict.items():
if value == 'positive': positive_list.append(key)
if value == 'negative': negative_list.append(key)

# Compute the positive score, the negative score for each news articles
positive_score = len(positive_list) ; negative_score = len(negative_list);
#print positive_list ; print negative_list ;
#print positive_score ; print negative_score;

#### Step 7: Create a Python list of model output

The final output from the model is populated in a Python list. The list contains the URL, positive score and the negative score for each of the selected news articles on which we conducted sentiment analysis.

  # Appending the empty list to create the final Sentiment analysis output
var_nos = [results_url,positive_score, negative_score]
df.append(var_nos)

# Print the final list of the Sentiment Analysis
for item in df:
print(item)`

Final Output

#### Step 8: Plot NIFTY vs NIFTY Pharma returns

Shown below is a plot of NIFTY vs NIFTY Pharma for the months of October-November 2016. In our NIFTY Pharma plot we have drawn arrows highlighting some of the press releases on which we ran our sentiment analysis model. The impact of the uncertainty regarding the US Presidential election results, and the negative news for the Indian Pharma sector emanating from the US is clearly visible on NIFTY Pharma as it fell substantially from the highs made in late October’2016. Thus, our attempt to gauge the direction of the Pharma Index using the Sentiment analysis model in Python programming language is giving us accurate results (more or less).

Next Step:

One can build more robust sentiment models using other approaches and trade profitably. As a next step we would recommend watching QuantInsti’s webinar on “Quantitative Trading Using Sentiment Analysis” by Rajib Ranjan Borah. Watch it by clicking on the video below:

Also, catch our other exciting Python trading blogs and if you are interested in knowing more about our EPAT course feel free to contact our QuantInsti team by clicking here.

• Sentiment Analysis of News Article – Python Code
• dict(1).csv
• Nifty and Nifty Pharma(1).csv
• Pharma vs Nifty.py

## Sentiment Trading Indicators and Strategies – Part 1

Sentiment Trading Indicators and Strategies – Part 1

“Market decides the price”

True to the saying, it does not matter whether your valuation of a security is true to its fundamentals and that the your valuation model is correct to all its sensitive inputs, what matters is whether the market is in consensus of your valuation and this is what will decide the price of the security. Hence what matters is the price that is decided by the market. These prices are decided by the intrinsic pricing along with the sentiment in the market as a price of a security is a function of its intrinsic value and behavioral factor of the investor.

It’s a common norm that the price of the security will deviate from its intrinsic value and analysts evaluate the security as being under-priced, at par or overpriced but then simply look back at the saying quoted in the beginning of the article. Having said that, we arrive at our topic of discussion: Market Sentiment.

### Market sentiment

A market sentiment is a general attitude and feeling of the investors with regards to the present price and the forecasted price of a security, index or other market instruments. This attitude can be a positive, neutral or a negative one. The basis of this attitude formed is in relation to factors like the macroeconomic reports, world events, fundamental analysis, technical analysis, historical price patterns, sector wise reports and non-economic factors (news, weather etc.).

#### Bullish Sentiment

The sentiment is said to be bullish when there is a positive attitude. In a bull market, the prices are expected to move in an upward direction. There is greed in the market when it is being pulled by the bull.

#### Bearish Sentiment

The sentiment is said to be bearish when there is a negative attitude. In a bear market, the prices are expected to move in a downward direction. There is fear in the market when is being dragged/suppressed by the bear.

Sentiment trading is taking up positions in the market, when it is driven by such bulls or bears. These positions can be momentum based i.e. going with the consensus opinion or market sentiment and if it’s a bull we invest high and sell higher or vice versa.

These positions can even be contrarian i.e. opposite to the market sentiment. A contrarian profits from the theory that when there is certain crowd behavior regarding a security, it gives rise to certain exploitable mispricing (overpricing an already prevailing rise in security) and that a great bull is followed by fall in the prices of the security due to corrections or vice versa. We have talked about momentum based trading strategies in our article about Algorithmic Trading Strategy & Paradigms.

“A great bull is followed by fall in the prices of the security due to corrections or vice versa”

### Sentiment indicators

Sentiment trading also needs indicators. These sentiment indicators attempt to estimate or gauge the investor activity for any signs of bullishness or bearishness before bull or bear takes hold of the market. These sentiment trading indicators give you the optimistic or pessimistic indication of a particular security before it is reflected in its prices. These indicators help us to front-run in the market before the sentiment is reflected in the market prices.

Sentiment trading indicators can be qualitative like opinion polls. There are many companies that provide these services of surveying market professionals and investors to conduct periodic polls and publish them. Some of the opinion polls are Investors Intelligence Advisors Sentiment reports, Market Vane Bullish Consensus, Consensus Bullish Sentiment Index. All these poll investment professionals and reports come from American Association of Individual Investors (AAII) that poll individual investors. These polls are regular and data is compiled over to represent in graphs and visualisation to understand a particular sentiment. Most widely used polls are US based polls.

Sentiment trading indicators can even be quantitative which are calculated from market data like security price, traded volume, open interest etc. A few of these indicators are as follows:

1. Put/Call Ratio
2. Arms Index or Short term trading Index (TRIN)
3. VIX
4. Margin Debt
5. Mutual funds cash position
6. Short interest ratio

In this article, we will be giving you an overview of Put Call Ratio and TRIN.

### Put/Call Ratio

Put/Call ratio is a very significant sentiment trading indicator. Investors use this while doing sentiment trading and one of the more popular markets among them is the options market since options provides leverage for trading opportunities, flexibility and the asymmetric payoffs which are considered as lottery by some (buy/long players usually) and premiums/steady income instruments by some (write/short players usually). However, when someone buys a call option or a put option there is an expectation from the buyer’s side that the market will move in an upward direction or downward direction respectively. Owing to the active trades (volume).

Put/call ratio is one of the most dependable market sentiment indicators.

The put/call ratio = put volume/call volume

Put volume is the amount of the put options bought in the market. Whenever an investor buys a put, the investors expect a fall in the security’s price. While Call volume is the amount of call options bought in the market. Whenever an investor buys a call, the investors expect a rise in the security’s price.

Hence a ratio greater than 1 indicates that the put options are being bought more than the call options, which in turn implies that the sentiment is bearish and the investors are expecting the market to go down in near future. A ratio less than 1 indicates that since call options are bought more in the market, the overall sentiment is bullish and the market will go upwards in near future. The trader carries out his trades according to the sentiment trading indicator.

##### Sentiment Trading Strategy based on Put/Call Ratio

This ratio is generally viewed as a contrarian indicator and the strategy is as follows:

• When this ratio is considerably greater than 1, there is an extreme bearish sentiment and a possible oversold market and a correction is likely. Hence BUY. These corrections will take market upwards and a long position at such a bottom will yield a good payoff to our contrarian strategy.
• When this ratio is considerably less than 1, there is an extremely bullish sentiment and a possible overbought market and a correction is likely. Hence SELL.

While there are no standard numbers that at which point in the ratio one should buy or sell, a trader should look at the general trend where the ratio moves in the higher or lower value. Once there are spikes or trenches and the ratio deviates from these trading ranges, these are the times one should act and take a position.

All the data providers provide the option volume data of the leading exchanges in the world and this indicator can be easily created on a spread sheet.

While dealing with the contrarian strategies it is important to check even the fundamentals of the underlying. If there are extremely weak fundamentals and the price drop is justified by it, then a contrarian strategy may not work.

### Arms Index or short term trading index (TRIN)

This index was developed by Richard Arms in 1967, also known as TRading INdex (TRIN) , is a common ‘flow of funds’ indicator. By flow of funds, we mean that this indicator signals the amount of money invested in securities of an index/exchange and help us in interpreting a sentiment. To understand TRIN we first need to understand some common terms:

• Advancing Stocks: The number of stocks in an index or exchange that have closed in green or after upside movement from previous day close.
• Declining Stocks: The number of stocks in an index or exchange that have closed in red or after downside movement from the previous days close.
• Declining volume: Total volume of these declining stocks.

TRIN = [(No. of advancing stocks/ No. of declining stocks) / (Advancing volume/Declining volume)]

When an ARMS index is above 1, it indicates a bearish sentiment. This happens because when the number of stocks closing in red is more than those in green it produces an AD stocks ratio of less than 1 and rationally this indicates a far greater volume traded in red stocks than green stocks. As a result, the AD volume ratio will even be lesser that the AD stock ratio and the entire ratio becomes greater than 1.

Similarly, when an ARMS index is below 1, it indicates a bullish sentiment. A ratio close to 1 indicates a balanced market. So when there are spikes (upward movements) in TRIN, they coincide with large daily losses in markets and trenches (downward movements) in TRIN coincide with large daily gains in the market.

##### Sentiment Trading Strategy based on Put/Call Ratio

While scaling this indicator on a chart, we will observe a range in which this indicator For trading purposes the time when we will take contrarian positions are :

• When this ratio is considerably greater than 1 (somewhere near 3), there is an extreme bearish sentiment and a possible oversold market and a correction is likely. Hence BUY. These corrections will take market upwards and a long position at such a bottom will yield a good payoff to our contrarian strategy.
• When this ratio is considerably less than 1(somewhere near 0.5), there is an extremely bullish sentiment and a possible overbought market and a correction is likely. Hence SELL.

While these are not standard numbers, one can always find a trading range and deviations from this moving average on either sides can help us to take contrarian positions while trading.

Each sentiment trading indicator and strategy associated with it has its own short comings; so as to understand the real sentiment in the market what we can do is take a look at more than 1 sentiment indicators to gauge the market sentiment. If there are conclusive evidences that it is a bull or bear and more importantly (since we are dealing in contrarian strategies), it’s an overbought bull or an oversold bear, we can take appropriate positions in the market and maximize our returns.

### Conclusion

In the above write-up, we have discussed only 2 such sentiment trading indicators that are used for analysis in the market. However there are many such indicators gauging sentiment from breadth, volatility, volume, flow of funds, shorts or margins taken up in the market and every indicator can be interpreted in one’s own style of investing. We have focused on a contrarian style of investing where we are capitalizing on the general optimism or pessimism prevailing in the market, observing a particular trend, taking up a position at the point where trend reverses and exiting the trade once the reversal reaps profits.

Always remember, when you trade, do not use these indicators in isolation. Use indications from more than 1 sentiment indicators and try to understand the fundamentals and rationality behind such patterns, but be brave enough to take up the contrarian position and capitalize on the fear or greed of other investors.

### Next Step

While you wait for the next article in this series on Sentiment Trading indicators, we would like you read these awesome article on sentiment analysis:

Sentiment Analysis in Trading Using R [WORKING MODEL]

Are you looking for resources on Algorithmic Trading? Do you want to learn Algorithmic Trading? Then check out Quantinsti’s course on Algorithmic Trading.

## Using Pivot Point to predict movement in trade markets

Pivot Point to predict market movement

We have already discussed about Candlestick Trading Strategy in a previous article. If you missed it, just click here to read it. Many techniques are used to understand the behavior of financial markets. Many who use candlestick strategy often agree unanimously that using this particular strategy in isolation might not be beneficial. Candlestick techniques help in evaluating opportunities more efficiently as compared to the one where historical data on bar graph is used for evaluation.

### Why?

Figure 1 – Bar vs Candlestick

Clearly, candlestick approach gives a lucid graphic presentation of data as compared to a simple bar chart. This helps traders to respond quickly to changes in the market.

### What is Pivot Point?

Floor traders used Pivot Points to set key levels while dealing in fast moving environments. They calculated Pivot points for the day by looking at the previous day’s high, low and closing prices.

“A pivot point is a technical analysis indicator used to determine the overall trend of the market over different time frames. The pivot point itself is simply the average of the high, low and closing prices from the previous trading day. On the subsequent day, trading above the pivot point is thought to indicate ongoing bullish sentiment, while trading below the pivot point indicates bearish sentiment” [1]

Pivot points are used strategically to catch support and resistance points, thus, helping in identifying areas where reversal of price movement is possible. Pivot points are used to determine where reversals might take place and the trader can ideate their investment strategy that has higher probability of hitting the bull’s eye. Using pivot point calculation, we can consider trade entry or exit. As a rule of thumb; trading above P is a bearish, and below P is bullish (where P is the pivot point). Regardless of which level you choose, both are indicators of major price movement. Let us now take a look at how we arrive at this P.

### Calculating Pivot Points

Calculation of pivot point can be performed by a few methods however; we are going to stick to the 5-Point system which uses the following parameters:

1. Previous Day’s High (H)
2. Previous day’s Low (L)
3. Previous day’s Close (C)
4. 2 Support Levels (S1 and S2)
5. 2 Resistance Levels (R1 and R2)

Since this equation will have the above mentioned 5 price points, therefore it is called 5-Point system.

To calculate P, we equate:

R2 = P + (H – L) = P + (R1 – S1)

R1 = (P x 2) – L

P = (H + L + C) / 3

S1 = (P x 2) – H

S2 = P – (H – L) = P – (R1 – S1)

Say for example, I am looking at the Sensex charts for the month of September 2016.

Then,

H = 29,077.28

L =  27,716.78

C = 27,865.96

P = (H + L + C) / 3 = 28220.0067

Similarly, we put the values of P, H, L & C in our other equations and find values of :

R1 = 28723.2333

R2 = 29580.5067

S1 = 27362.7333

S2 = 26859.5067

### Time Frames in Pivot Points

#### Prior Day

We will look at the prior day’s numbers when we have to calculate pivot points for a time frame of 1, 5, 10 & 15 minutes. So, we will be basing our intraday charts on the previous day’s H, L and C. The value of P, once set, remains constant for that day.

#### Prior Week

We will look at the prior week’s numbers when we have to calculate pivot points for a time frame of 30 & 60 minutes. So, we will be basing our intraweek charts on the previous week’s H, L and C. The value of P, once set, remains constant for till the end of the week.

#### Prior Month

When we have a time frame of an entire day, then we look at the prior month’s data. So, if we want the value of P for this month (i.e. October 2016), then we will base our calculations on the H, L & C of September 2016. Like in the previous cases, the value of P will remain constant throughout the month. From the example in the previous section, we arrived at P= 28220.0067 for the month of September 2016. When the month is completed, then new value of P will be calculated on the 1st day of November.

Kindly note that the 1st day is the first trading day of the month, not necessarily the 1st of the month.

### How traders use Pivot Points

The pivot point sets the general tone for price motion. A flow above P is fine and indicates power. Remember the fact that P is based on the prior period’s stats. A move above the pivot point indicates strength with a goal to reach the primary resistance, R1. A smash above first resistance suggests even more strength with a goal to the second resistance level, R2.

The vice-versa is equally applicable. A move below the pivot point shows weakness with a target to S1. A break below S1 shows even more weakness.

Traders use these values to decide their course of action. If the prices come down to S1 and stay put, then may start expecting an upturn from the support level. In the same way, if the prices go up and reach R1 or any other Resistance level and break, then they may expect a decline in the making. In order to strengthen this expectation, one must also look at the indicator signal.

### To conclude

The pivot point (P) acts as the primary support or resistance indicator and therefore, we predict that the price movement at this point will be highest. Trade pundits use pivot points with the following objectives:

1. ##### Overall trend determination

Like we mentioned before, a downward movement will indicate bearish and upward movement is sign for bullish.

1. ##### Deciding Entry & Exit

Traders decide based on the pivot point, when and where to enter or exit the market. For instance, if I see that the price is breaking at resistance level (R) then I will put a limit order to buy 500 shares. If the same trend applies to support level (S) then I will set a stop-loss.

So, you see that pivot point technique is very efficient and is useful for traders to predict market movement. But how far you succeed in capitalizing this depends on how well you can use this with other technical indicators like candlesticks.

### Next step

Candlestick and Pivot Points are movement based techniques. You may also like to read about other popular algorithmic trading strategies like statistical arbitrage and news based strategies. We suggest you to go through our series on how statistics is used in trading strategies.

Since you will need to be analytical & quantitative while getting into or upgrading to algorithmic trading it is imperative to learn programming (some if not all) and build foolproof systems and execute right strategies. To enable a fine learning experience QuantInsti offers a comprehensive course called Executive Programme in Algorithmic Trading (EPAT) with lecture recordings and life time access and support.

Written by Harikumar Krishnamoorthy

‘Looks can be deceiving,’ a wise person once said. The phrase holds true for Algorithmic Trading Strategies. The term Algorithmic trading strategies might sound very fancy or too complicated. However, the concept is very simple to understand, once the basics are clear. In this article, I will be telling you about algorithmic trading strategies with some interesting examples.

If you look at it from the outside, an algorithm is just a set of instructions or rules. These set of rules are then used on a stock exchange to automate the execution of orders without human intervention. This concept is called Algorithmic Trading.

Let me start with a very simple trading strategy. Those who are already into trading would know about S.M.A and for those who don’t; S.M.A is Simple Moving Average. S.M.A can be calculated using any predefined and fixed number of days. An algorithmic trading strategy based on S.M.A can be simplified in these four simple steps:

1. Calculate 5 day SMA
2. Calculate 20 day SMA
3. Take a long position when the 5 day SMA is larger than or equal to 20 day SMA
4. Take a short position when the 5 day SMA is smaller than 20 day SMA

We refer to this algorithmic trading strategy as Moving Average Crossover Strategy. This was just a simple example. Now don’t get down to thinking that it is all going to be a bed of roses. Even if it were, then be prepared for the thorns.In everyday trading, far more complex trading algorithms are used to generate algorithmic trading strategies.

All the algorithmic trading strategies that are being used today can be classified broadly into the following categories:

• Momentum/Trend Following
• Arbitrage
• Statistical Arbitrage
• Market Making

Let me go into some detail.

#### Momentum based Strategies

Assuming that there is a particular trend in the market. As an algo trader, you are following that trend. Further to our assumption, the markets fall within the week. Now, you can use stats to determine if this trend is going to continue. Or if it will change in the coming weeks. Accordingly, you will make your next move. You have based your algorithmic trading strategy on the market trends which you determined by using statistics.

This method of following trends is called Momentum Based Strategy.

There are numerous ways to implement this algorithmic trading strategy and I have discussed this in detail in one of our previous articles called “Methodology of Quantifying News for Automated Trading

#### Arbitrage

If we assume that a pharma-corp is to be bought by another company, then the stock price of our corp could go up. This is triggered by the acquisition which is a corporate event. If you are planning to invest based on the pricing inefficiencies that may happen during a corporate event (before or after), then you are using an event-driven strategy. Bankruptcy, acquisition, merger, spin-offs etc could be the event that drives such kind of an investment strategy.

These strategies can be market neutral and used by hedge fund and proprietary traders widely.

#### Statistical Arbitrage

When an arbitrage opportunity arises because of misquoting in prices, it can be very advantageous to algo trading strategy. Although such opportunities exist for a very short duration as the prices in the market get adjusted quickly. And that’s why this is the best use of algorithmic trading strategies, as an automated machine can track such changes instantly.

For instance, if Apple’s price falls under $1 then Microsoft will fall by$0.5 but Microsoft has not fallen, so you will go and sell Microsoft to make a profit. You can read about the common misconceptions people have about Statistical Arbitrage here.

#### Market Making

To understand Market making, let me first talk about Market Makers.

According to Wikipedia:

A market maker or liquidity provider is a company, or an individual, that quotes both a buy and a sell price in a financial instrument or commodity held in inventory, hoping to make a profit on the bid-offer spread, or turn.

Market making provides liquidity to securities which are not frequently traded on the stock exchange. The market maker can enhance the demand-supply equation of securities. Let me give you an example:

Let’s assume you have Martin, a market maker, who buys for Rs. 500 from the market and sell it at 505. He will give you a bid-ask quote of Rs. 505-500. The profit of Rs. 5 cannot be sold or exchanged for cash without substantial loss in value. When Martin takes a higher risk then the profit is also higher.

I found Michael Lewis’ book ‘Flash Boys’ in Indian Bull Market pretty interesting and it talks about liquidity, market making and HFT in great detail. Check it out after you finish reading this article.

Since you will need to be analytical & quantitative while getting into or upgrading to algorithmic trading it is imperative to learn programming (some if not all) and build foolproof systems and execute right algorithmic trading strategy. Reading this article on Automated Trading with Interactive Brokers using Python will be very beneficial for you. You can read the article here.

Now that I have introduced you to algorithmic trading strategies, I will be throwing some light on the strategy paradigms and modeling ideas pertaining to each strategy.

• Market Making
• Statistical Arbitrage
• Momentum
• Machine Learning Based

#### Market Making

As I had mentioned earlier, the primary objective of Market making is to infuse liquidity in securities that are not traded on stock exchanges. In order to measure the liquidity, we take the bid-ask spread and trading volumes into consideration.

The trading algorithms tend to profit from the bid-ask spread. I will be referring to our buddy, Martin, again in this section. Martin being a market maker is a liquidity provider who can quote on both buy and sell side in a financial instrument hoping to profit from the bid-offer spread. Martin will accept the risk of holding the securities for which he has quoted the price for and once the order is received, he will often immediately sell from his own inventory. He might seek an offsetting offer in seconds and vice versa.

When it comes to illiquid securities, the spreads are usually higher and so are the profits. Martin will take a higher risk in this case. Several segments in the market lack investor interest due to lack of liquidity as they are unable to gain exit from several small- and mid-cap stocks at any given point in time.

Market Makers like Martin are helpful as they are always ready to buy and sell at the price quoted by them. In fact, much of high frequency trading (HFT) is passive market making. The strategies are present on both sides of the market (often simultaneously) competing with each other to provide liquidity to those who need.

So, when is this strategy most profitable?

This strategy is profitable as long as the model accurately predicts the future price variations.

Modeling ideas based on this Paradigm

The bid-ask spread and trade volume can be modeled together to get the liquidity cost curve which is the fee paid by the liquidity taker. If the liquidity taker only executes orders at the best bid and ask, the fee will be equal to the bid ask spread times the volume. When the traders go beyond best bid and ask taking more volume, the fee becomes a function of the volume as well.

Trade volume is difficult to model as it depends on the liquidity takers execution strategy. The objective should be to find a model for trade volumes that is consistent with price dynamics. Market making models are usually based on one of the two:

1. The first focuses on inventory risk. The model is based on preferred inventory position and prices based on the risk appetite.
2. The second is based on adverse selection which distinguishes between informed and noise trades. Noise trades do not possess any view on the market whereas informed trades do. When the view of the liquidity taker is short term, its aim is to make short term profit utilizing the statistical edge. In the case of long term view, the objective is to minimize the transaction cost. The long-term strategies and liquidity constraints can be modeled as noise around the short-term execution strategies.

To know more about Market Makers, you can check out this interesting article on QuantInsti’s blog.

#### Statistical Arbitrage

If Market making is the strategy that makes use of the bid-ask spread, Statistical Arbitrage seeks to profit from statistical mispricing of one or more assets based on the expected value of these assets.

A more academic way to explain statistical arbitrage is to spread the risk among thousand to million trades in a very short holding time to, expecting to gain profit from the law of large numbers. Statistical Arbitrage Algorithms are based on mean reversion hypothesis, mostly as a pair.

Modelling ideas

Pairs trading is one of the several strategies collectively referred to as Statistical Arbitrage Strategies. In pairs trade strategy, stocks that exhibit historical co-movement in prices are paired using fundamental or market-based similarities. The strategy builds upon the notion that the relative prices in a market are in equilibrium, and that deviations from this equilibrium eventually will be corrected.

When one stock outperforms the other, the outperformer is sold short and the other stock is bought long with the expectation that the short term diversion will end in convergence. This often hedges market risk from adverse market movements i.e. makes the strategy beta neutral. However, the total market risk of a position depends on the amount of capital invested in each stock and the sensitivity of stocks to such risk.

#### Momentum

Momentum Strategies seek to profit from the continuance of existing trend by taking advantage of market swings.

“In simple words, buy high and sell higher and vice versa.”

And how do we achieve this?

In this particular algo-trading strategy we will take short-term positions in stocks that are going up or down until they show signs of reversal. It is counter-intuitive to almost all other well-known strategies. Value investing is generally based on long-term reversion to mean whereas momentum investing is based on the gap in time before mean reversion occurs.

Momentum is chasing performance, but in a systematic way taking advantage of other performance chasers who are making emotional decisions. There are usually two explanations given for any strategy that has been proven to work historically, either the strategy is compensated for the extra risk that it takes or there are behavioral factors due to which premium exists.

There is a long list of behavioral biases and emotional mistakes that investors exhibit due to which momentum works. However, this is easier said than done as trends don’t last forever and can exhibit swift reversals when they peak and come to an end. Momentum trading carries a higher degree of volatility than most other strategies and tries to capitalize on the market volatility. It is important to time the buys and sells correctly to avoid losses by using proper risk management techniques and stop losses. Momentum investing requires proper monitoring and appropriate diversification to safeguard against such severe crashes.

Modelling ideas

Firstly, you should know how to detect Price momentum or the trends. As you are already into trading, you know that trends can be detected by following stocks and ETFs that have been continuously going up for days, weeks or even several months in a row. For instance, identify the stocks trading within 10% of their 52 weeks high or look at the percentage price change over the last 12 or 24 weeks. Similarly to spot a shorter trend, include a shorter term price change.

If you remember, back in 2008, the oil and energy sector was continuously ranked as one of the top sectors even while it was collapsing. We can also look at earnings to understand the movements in stock prices. Strategies based on either past returns (“price momentum strategies”) or on earnings surprise (known as “earnings momentum strategies”) exploit market under-reaction to different pieces of information. An earnings momentum strategy may profit from the under-reaction to information related to short-term earnings. Similarly, a price momentum strategy may profit from market’s slow response to a broader set of information including longer-term profitability.

#### Machine Learning based

In Machine Learning based trading, algorithms are used to predict the range for very short term price movements at a certain confidence interval. The advantage of using Artificial Intelligence (AI) is that humans develop the initial software and the AI itself develops the model and improves it over time. A large number of funds rely on computer models built by data scientists and quants but they’re usually static, i.e. they don’t change with the market. ML based models on the other hand can analyze large amounts of data at high speed and improve themselves through such analysis.

Modeling idea

A form of machine leaning called “Bayesian networks” can be used to predict market trends while utilizing a couple of machines. An AI which includes techniques such as evolutionary computation (which is inspired by genetics) and deep learning might run across hundreds or even thousands of machines. It can create a large and random collection of digital stock traders and test their performance on historical data. It then picks the best performers and uses their style/patterns to create a new of evolved traders. This process repeats multiple times and a digital trader that can fully operate on it’s own is created.

This process repeats multiple times and a digital trader that can fully operate on it’s own is created.

This process repeats multiple times and a digital trader that can fully operate on it’s own is created.

These were some important strategy paradigms and modelling ideas.  Next, we will go through the step by step procedure to build a trading strategy.

You can learn these Paradigms in great detail in QuantInsti’s Executive Programme in Algorithmic Trading (EPAT), one of the most extensive algorithmic trading courses available online with lecture recordings and lifetime access and support.

### Building an algorithmic trading strategy

From algo trading strategies to paradigms and modeling ideas, I come to that section of the article where I will tell you how to build a basic algorithmic trading strategy.

That is the first question that must have come to your mind, I presume. The point is that you have already started by knowing the basics and paradigms of algorithmic trading strategies while reading this article. Now, that our bandwagon has it’s engine turned on, it is time to press on the accelerator.

#### And how exactly is this done?

I will explain how an algorithmic trading strategy is built, step by step. The concise description will give you an idea about the entire process.

1. Decide upon the genre/strategy paradigm

The first step is to decide the strategy paradigm. It can be Market Making, Arbitrage based, Alpha generating, Hedging or Execution based strategy. For this particular instance, I will choose pair trading which is a statistical arbitrage strategy that is market neutral (Beta neutral) and generates alpha, i.e.makes money irrespective of market movement.

1. Establish Statistical significance

You can decide on the actual securities you want to trade based on market view or through visual correlation (in the case of pair trading strategy). Establish if the strategy is statistically significant for the selected securities. For instance, in the case of pair trading, check for co-integration of the selected pairs.

Now, code the logic based on which you want to generate buy/sell signals in your strategy. For pair trading check for “mean reversion”; calculate the z-score for the spread of the pair and generate buy/sell signals when you expect it to revert to mean. Decide on the “Stop Loss” and “Profit Taking” conditions.

• Stop Loss– A stop-loss order limits an investor’s loss on a position in a security. It fires an order to square off the existing long or short position to avoid further losses and helps to take emotion out of trading decisions.
• Take Profit– take-profit orders are used to automatically close out existing positions in order to lock in profits when there is a move in a favorable direction.
1. Quoting or Hitting strategy

It is very important to decide if the strategy will be “quoting” or “hitting”. Execution strategy to a great extent decides how aggressive or passive your strategy is going to be.

• Quoting– In pair trading you quote for one security and depending on if that position gets filled or not you send out the order for the other. In this case, the probability of getting a fill is lesser but you save bid-ask on one side.
• Hitting- In this case, you send out simultaneous market orders for both securities. The probability of getting a fill is higher but at the same time slippage is more and you pay bid-ask on both sides.

The choice between the probability of fill and Optimized execution in terms of slippage and timed executive is what this is if I have to put it that way. If you choose to quote, then you need to decide what are quoting for, this is how pair trading works. If you decide to quote for the less liquid security, slippage will be less but the trading volumes will come down liquid securities on the other hand increase the risk of slippage but trading volumes will be high.

Using stats to check causality is another way of arriving at a decision, i.e. change in which security causes change in the other and which one leads. The causality test will determine the “lead-lag” pair; quote for the leading and cover the lagging security.

1. Backtesting & Optimization

How do you decide if the strategy you chose was good or bad?

How do you judge your hypothesis?

This is where back-testing the strategy comes as an essential tool for estimation of the performance of the designed hypothesis based on historical data. A strategy can be considered to be good if the backtest results and performance statistics back the hypothesis.

Hence, it is important to choose historical data with a sufficient number of data points. This is to create a sufficient number of sample trades (at least 100+ trades) covering various market scenarios (bullish, bearish etc.). Ensure that you make provision for brokerage and slippage costs as well. This will get you more realistic results but you might still have to make some approximations while backtesting. For instance, while backtesting quoting strategies it is difficult to figure out when you get a fill. So, the common practice is to assume that the positions get filled with the last traded price.

###### What kind of tools should you go for, while backtesting?

Since backtesting for algorithmic trading strategies involves a huge amount of data, especially if you are going to use tick by tick data. So, you should go for tools which can handle such mammoth load of data.

###### R or MATLAB?

R is excellent for dealing with huge amounts of data and has a high computation power as well. Thus, making it one of the better tools for backtesting. Also, R is open source and free of cost. We can use MATLAB as well but it comes with a licensing cost.

1. Risk and Performance Evaluation

“With great strategy, comes greater scrutiny,” A Great Thinker once quoted.

Fine, I just ripped off Ben Parker’s famous quotation from the Spiderman movie (not the Amazing one). But trust me, it is 100% true. No matter how confident you seem with your strategy or how successful it might turn out previously, you must go down and evaluate each and everything in detail.  There are several parameters that you would need to monitor when analyzing a strategy’s performance and risk. Some important metrics/ratios are mentioned below:

• Total Returns (CAGR)- Compound Annual Growth Rate (CAGR). It is the mean annual growth rate of an investment over a specified period of time longer than one year.
• Hit Ratio- Order to trade ratio.
• Average Profit per Trade- Total profit divided by the total number of trades
• Average Loss per trade- Total loss divided by the total number of trades
• Maximum Drawdown– Maximum loss in any trade
• Volatility of Returns- Standard deviation of the “returns”
• Sharpe Ratio- Risk adjusted returns, i.e. excess returns (over risk free rate) per unit volatility or total risk.