Decoding the Black Box running Trading Systems

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+


What is a Trading System?

A “trading system”, more commonly referred as a “trading strategy” is nothing but a set of rules, which when applied to the given input data generate entry and exit signals (buy/sell).

Although formulating a trading strategy seems like an easy task, in reality, it is not! Creating a profitable trading strategy requires exhaustive quantitative research, and the brains behind a quantitative trading strategy are known as “Quants” in the algorithmic trading world. We can define a quant as a professional employed by a quantitative trading firm who applies advanced mathematical and statistical models with the sole objective to create an alpha-seeking strategy.

By an alpha-seeking strategy, we mean a profitable trading strategy that can consistently generate returns that are independent of the direction of the overall market.

For those outside the algorithmic trading world, the work of quants and the quantitative trading strategies appear opaque and complex, hence the term “Black Box” Trading. In this post, we will attempt to unravel the black box, and try to decipher the mechanics of black box trading.

How do Trading Systems operate?

Any trading system, conceptually, is nothing more than a computational block that interacts with the exchange on two different streams.

  1. Receives market data
  2. Sends order requests and receives replies from the exchange.

The market data that is received typically informs the system of the latest order book. It might contain some additional information like the volume traded so far, the last traded price and quantity for a scrip. However, to make a decision on the data, the trader might need to look at old values or derive certain parameters from history. To cater to that, a conventional system would have a historical database to store the market data and tools to use that database. The analysis would also involve a study of the past trades by the trader. Hence another database for storing the trading decisions as well. Last, but not the least, a GUI interface for the trader to view all this information on the screen.

The entire trading system can now be broken down into

  • The exchange(s) – the external world
  • The server
    • Market Data receiver
    • Store market data
    • Store orders generated by the user
  • Application
    • Take inputs from the user including the trading decisions
    • Interface for viewing the information including the data and orders
    • An order manager sending orders to the exchange

What you call a Trading System is actually a CEP System

A CEP System stands for Complex Event Processing System. This lengthy term may sound very convoluted, but once you learn complex events and the components that make a CEP system, you will appreciate this clear-box system.

A complex event is nothing but a set of incoming events. These include stock trends, market movements, news etc. Complex event processing is performing computational operations on complex events in short time. The operations can include detecting complex patterns, building correlations and relationships such as causality and timing between many incoming events.

CEP systems process events in real time and this is a key feature of a CEP system. The faster the processing of events, the better a CEP system is. For example, if a trading system is designed to detect a profit-making opportunity for the next 1 second, but the time taken by the CEP system exceeds this threshold, then the trading system won’t be able to make any profits.

The CEP system comprises of four parts: a CEP engine, CEP rules, CEP WS and CEP result interface. The two primary components of any CEP system are the CEP engine and the set of CEP rules. The CEP engine processes incoming events based on CEP rules. These rules and the events that go as an input to the CEP engine are determined by the trading system (trading strategy) applied.

For a quant, the majority of his work is concentrated in this CEP system block.  A quant will spend most of his time in formulating trading strategies; performing rigorous backtesting, optimization, and position-sizing among other things. This is done to ensure the viability of the trading strategy in real markets. No single strategy can guarantee everlasting profits. Hence, quants are required to come up with new strategies on a regular basis to maintain an edge in the markets.

There are a number of popular trading systems that are widely used in current markets. These range from Momentum strategies, Statistical arbitrage, Market making etc. See our very insightful blog on Algorithmic Trading Strategies, Paradigms and Modelling Ideas to know more about these trading systems.

Order Management in Automated Trading Systems

The signals generated by an algorithmic system can be either executed manually or in an automated way. When the signals are executed in an automated manner, we can call this entire system as an “Automated trading system”.  Automation of the orders is done by the “Order Manager” module. The order manager module comprises of different execution strategies which execute the buy/sell orders based on a pre-defined logic. Some of the popular execution strategies include VWAP, TWAP etc. There are different processes like order routing, order encoding, transmission etc. that form part of this module. See our blog on Order Management System (OMS) to know more about these processes.

Risk management in Automated Trading Systems

Since automated trading systems work without any human intervention, it becomes pertinent to have thorough risk checks to ensure that the trading systems perform as designed. The absence of risk checks or a faulty risk management can lead to enormous irrecoverable losses for a quantitative firm as seen in the past. Thus, a risk management system (RMS) forms a very critical component of any automated trading system. There are 2 places where Risk Management is handled in algo trading systems:

Within the application – We need to ensure those wrong parameters are not set by the trader. It should not allow a trader to set grossly incorrect values nor any fat-finger errors.

Before generating an order in OMS – Before the order flows out of the system we need to make sure it goes through some risk management system. This is where the most critical risk management check happens. See our blog on “Changing trends in trading risk management” to know more about risk management aspects and risk handling in an automated trading system.

High-Frequency Trading Systems

Building an automated trading system involves high costs and resources. Building such a system in-house may not be feasible for some quant firms. Such firms can opt for institutional automated trading platforms which allow for high-frequency trading, execution and order management across equities, foreign exchange, options, and futures. These platforms allow their clients to completely control and customize their proprietary algorithms while maintaining the confidentiality of their trading strategies.

Popular Automated Trading Systems

Building an entire automated trading system can be beyond the scope of an individual retail trader. For traders who want to explore the algorithmic way of trading can opt for automated trading systems that are available in the markets on a subscription basis. A trader can subscribe to these automated systems and use the algorithmic trading strategies that are made available to the users on these systems. We have highlighted some of the popular automated trading systems in our blog, “Top Algo Trading Platforms in India”. Traders who know programming can formulate and backtest their strategies in programming platforms like Python and R.

Build Your Own Algorithmic Trading Systems

By now, you must have realized that “Black Box” trading is not as complex as it sounds.  Wannabe traders can learn to build their own algorithmic trading strategies and trade profitably in the markets. The following steps can serve as a rough guideline for building an algorithmic trading strategy:

  1. Ideation or strategy hypothesis – come up with a trading idea which you believe would be profitable in live markets. The idea can be based on your market observations or can be borrowed from trading books, research papers, trading blogs, trading forums or any other source.
  2. Get the required data – To test your idea you would require historical data. You can get this data from sites like Google finance, Yahoo finance or from a paid data vendor
  3. Strategy writing – Once you have the data, you can start coding your strategy for which you can use tools like Excel, Python or R programming.
  4. Backtesting your strategy – Once coded, you need to test whether your trading idea gives good returns on the historical data. Backtesting would involve optimization of inputs, setting profit targets and stop loss, position-sizing etc.
  5. Paper trading your strategy – After the backtesting step, you need to paper trade your strategy first. This would mean testing your strategy on a simulator which simulates market conditions. There are brokers which provide platforms for paper trading your strategy.
  6. Taking your strategy live – if the strategy is profitable after paper trading you can take it live. You can open an account with a suitable broker that provides the algorithmic trading facility.

The number of exchanges that allow algorithmic trading for professional, as well as retail traders, has been growing with each passing year, and more and more traders are turning to algorithmic trading. We hope that this article was insightful for our readers and would encourage them to upgrade their way of trading. So what are you waiting for? Go Algo!!

Next Step

If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to build a promising career in algorithmic trading. If you are interested in exploring self-paced trading courses you can also visit Quantra where we have listed short courses like “Getting started with Algorithmic Trading” and “Python for Trading”.


Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Making a Career in Algorithmic Trading

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

By Milind Paradkar

The advent of algorithmic trading in the late last century caused a massive tectonic shift in the way trading took place in exchanges worldwide. Be it trading in stocks, derivatives, Forex or commodities, trading firms worldwide adopted algorithmic trading in a big way.

The last couple of decades have seen an exponential growth in the algorithmic trading market and it continues to grow at a significant pace. According to the “Global Algorithmic Trading Market 2016-2020” report published by Research and Markets last year, the global algorithmic trading market is expected to grow at a CAGR of 10.3% during the period 2016-2020.

In order to remain competitive and earn big profits year after year, big banks, hedge funds, and other trading firms have been hiring top talent from various universities and colleges worldwide. This is turn has led to a surge in algorithmic trading/HFT jobs. Scores of students, engineering graduates, and developers want to explore and build a promising career in algorithmic trading today. This said, many of the wannabe quants & developers are unaware of the nature of the work in algorithmic trading firms and the skill sets needed to make a foray into this coveted algorithmic trading world.

QuantInsti recently conducted a webinar on “Career development – Jobs in algorithmic/HFT trading”. The webinar offered a unique chance for attendees to interact with a team of quants & HFT developers on a one-to-one level and ask career-related queries to the esteemed panelists. The panel of speakers included Mr. Sunith Reddy (Head of Technology, iRageCapital Broking), Mr. Puneet Singhania (Director, Master Trust) and Mr. Gopinath Ramkumar (Market Risk Quant)

In this post, we highlight some of the most important takeaways from the webinar for job seekers in high-frequency trading jobs, quant jobs, and algorithmic trading jobs.

Key Takeaways:


Automated trading is not free from human intervention

A very important point that wannabe quant/developers ought to know is that automated trading does not mean it is free from human intervention. Automated trading has caused the focus of human intervention to shift from the process of trading to a more behind-the-scenes role, which involves devising newer alpha-seeking strategies on a regular basis.


[tweetshareinline tweet=”Analytical approach to problem solving matters!” username=””]


Top skill sets for algorithmic trading

Analytical skills – Having an analytical bent of mind is a very important quality for any quant trader/developer, and is given a high weightage in an interview. For example, an interviewing candidate may be given a huge data set and asked to find patterns from the data. Candidates get evaluated on how they approach any given problem and their ability to justify their solutions objectively.

Math and programming skills – As the core of algorithmic trading revolves around algorithms, data, and programming, having reasonable programming skills and a basic understanding of statistics and calculus is important for any job seeker in algo/HFT trading. For example, if a candidate is applying in a firm that deploys low latency strategies, then an expert level of programming would be expected from such a candidate.

To know more on skills sets required, check out this infographic – Top skills for nailing Quant or Trader interview


[tweetshareinline tweet=”No single strategy can guarantee everlasting profits” username=””]


Understand the strategy development process

While devising any strategy, it is important to understand the risk and rewards associated with that strategy in order to determine whether it has an edge in the markets. This is done during the backtesting of a strategy. The frequency of trading, instruments traded, leverage also needs to be taken into consideration before going live with the strategy live in the markets. Quants should also remember that no single strategy can guarantee profits year-after-year. Quants are required to formulate new strategies on a regular basis using advanced mathematical models & statistics and overhaul the old strategies to remain profitable in the markets. To learn about various algorithmic trading strategies, you can check out this blog post – Algorithmic Trading Strategies, Paradigms and Modelling Ideas

Techies need to understand financial markets

Quantitative trading involves dealing with large datasets, trading in different instruments like stocks, derivatives, Forex etc. Hence, even if you are coming from a non-finance technology background, as a developer in a quant firm, you need to have a fair understanding of the financial markets. Trading firms usually make their new recruits spend time on different desks (e.g. quant desk, trading, risk management desk) to gain an understanding of the markets.

Should I learn C++ or Python?

Python is good for conceptualizing and backtesting of strategies. Python offers many libraries for validation and visualization of results. It can also be used by firms for strategies that are not dependent on low latency. See here to know what makes Python the most preferred language for Algorithmic Trading. On the other hand, C++ is usually used by firms that trade very low latency strategies. Thus, if the objective of an aspiring developer is to get into an HFT firm, then irrespective of the language that he starts with, he will have to finally end up learning C++.


[tweetshareinline tweet=”Know your resume well!” username=”quantinsti”]

Points that recruiters look in a resume

Recruiters appreciate if candidates are honest on their resume, candidates should be able to back any points that they have mentioned. Candidates are expected to demonstrate a strong understanding in the core areas that are highlighted in their resume. Recruiters also tend to give positive weightage if the candidate has undertaken a project work or published any research papers in his/her areas of interests.

Salaries for traders and programmers

It is a known fact that salaries & bonuses are lucrative in algorithmic trading firms. However, there is no common compensation policy followed across algorithmic trading firms. For example, salaries paid to tech guys in similar roles can vary from one firm to another. In some firms, bonuses get equally split between traders and programmers based on the profitability of a strategy. Compensation can also vary depending also on the type of the trading firms (e.g. Family office/bank/HFT firm etc.) and the strategies (low-frequency/high-frequency) that are deployed by the firms.


These were some of the important points that aspiring quants/developers should keep in mind as they prepare themselves for a successful career in algorithmic trading. You can click here to catch the entire webinar – “Career development – Jobs in algorithmic/HFT trading”. In case you need any further guidance, feel free to contact us.

Next Step:

If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to be a successful trader. Enroll now!



Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Learn Algorithmic Trading: A Step by Step Guide

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

With the boom in technological advancements in trading and financial market applications, algorithmic trading and high-frequency trading is being welcomed and accepted by exchanges all over the world. Within a decade, it is the most common way of trading in the developed markets and rapidly spreading in the developing economies.

Algorithmic Trading & the Industry Requirements

For beginners who want to venture into algorithmic trading, this article will serve as a guide to all the things that are essential to get you trading the algorithmic way. Let us start by defining algorithmic trading first. There is a lot of confusion between algorithmic trading, automated trading, and HFT (high-frequency) trading.

Difference between Algorithmic trading, Quantitative trading, Automated trading and High-Frequency trading

Algorithmic trading – Algorithmic trading means turning a trading idea into a trading strategy via an algorithm. The trading strategy thus created can be backtested with historical data to check whether it will give good returns in real markets. The algorithmic trading strategy can be executed either manually or in an automated way.

Quantitative trading – Quantitative trading involves using advanced mathematical and statistical models for creating and executing trading strategies.

Automated trading – Automated trading means completely automating the order generation, submission, and the order execution process.

HFT (high-frequency) trading – Trading strategies can be categorised as low-frequency, medium-frequency and high-frequency strategies as per the holding time of the trades. High-frequency strategies are algorithmic strategies which get executed in an automated way in quick time, usually on a sub-second time scale. Such strategies hold their trade positions for a very short time and try to make wafer-thin profits per trade, executing millions of trades every day.

An important point to note here is that automated trading does not mean it is free from human intervention. Automated trading has caused the focus of human intervention to shift from the process of trading to a more behind-the-scenes role, which involves devising newer alpha-seeking strategies on a regular basis.

Entry requirements into algorithmic firm

In the past, entry into algorithmic trading firms used to be restricted to PhDs in Physics, Mathematics or Engineering Sciences, who could build sophisticated quant models for trading.  However, in recent years there has been an explosive growth of online education industry, offering comprehensive algorithmic trading programs to wannabe algorithmic traders. This has made it possible to get into this domain without having to go through the long (8-10 years) academic route.

In the sections below, we outline the core areas that any wannabe algorithmic trader ought to focus on. We also present our readers with a comprehensive picture of the different ways and means through which these essential skill sets can be acquired.

Step 1: Core areas

Algorithmic trading is a multi-disciplinary field which requires knowledge in three domains, namely,

  • Quantitative Analysis/Modeling
  • Programming Skills
  • Trading/Financial Markets Knowledge

Quantitative Analysis

If you are a trader who is used to trade using fundamental and technical analysis, you would need to shift gears to start thinking quantitatively. Working on statistics, time-series analysis, statistical packages such as Matlab, R should be your favourite activities. Exploring historical data from exchanges and designing new trading strategies should excite you.  Problem-solving skills are highly valued by recruiters across trading firms.

Trading Knowledge

A professional coder/developer in a trading firm is expected to have a good fundamental knowledge of financial markets such as types of trading instruments (stocks, options, currencies etc.), types of strategies (trend following, mean reversal etc.), arbitrage opportunities, options pricing models, risk management. This knowledge will be crucial when you interact with the quants and will help in creating robust programs.

View some popular algo strategies here -> Algorithmic Trading Strategies, Paradigms and Modelling Ideas

Programming Skills

The strategies created by the quants are implemented in the live markets by the programmers. If you want to excel in the technology driven domain of automated trading, you should be willing to learn new skills and you shouldn’t be disinclined to any field. So if you have never printed “hello world” by compiling your own coding program, it’s time to download the compiler of your interest – C++/Java/Python/Ruby and start doing it! The best way to learn to program is to practice, practice and practice. Sound knowledge of programming languages like Python/C++/Java/R is a pre-requisite for a quant developer job in trading firms. You can read a couple of our popular blog posts on Programming below:

Why Python Algorithmic Trading is Preferred Choice among Traders?

Popular Python trading platform for Algorithmic Trading

Step 2: Ways to become an Algo trading professional

Getting started with books

Books are a great resource to get started in algorithmic trading. You will find many good books written on different algorithmic trading topics by some well-known authors. As an example, to hone your knowledge in derivatives, the “Options, Futures, and Derivatives” book authored by John C. Hull is considered a very good read for beginners. For algorithmic trading, one can read the “Algorithmic Trading: Winning Strategies and Their Rationale” book by Dr. Ernest Chan.

Find a list of good reads here -> Essential Books on Algorithmic Trading

Free resources  

In addition to the books, beginners can follow various blogs on algorithmic trading; watch YouTube videos, catch trading podcasts (e.g. Chat with Traders), attend online webinars (list of webinars hosted by QuantInsti), or get registered on platforms like quantopian to learn to code. One can also register for the free courses that are available on various online learning portals like Coursera, Udemy, Udacity, edX, & Open Intro.

Although these free resources are a good starting point, one should note that some of these have their own shortcomings. For example, books do not give you a hands-on experience in trading. Free courses on online portals can be subject specific and may offer very limited knowledge to serious learners. Another important point to note is the lack of interaction with experienced market practitioners when you opt for some of these free courses.

Learn from Professionals/Experts/Market Practitioners

The building blocks in learning Algorithmic trading are Statistics, Derivatives, Matlab/R, and programming languages like Python. It becomes necessary to learn from the experiences of market practitioners, which you can do only by implementing strategies practically alongside them. You can join any organization as a trainee or intern to get familiarized with their work ethics and market best practices. If it’s not possible for you to join any such organization then you can opt for classroom courses/workshops or paid online courses. Most of the classroom courses/workshops are delivered in the form of 2 days to 2 weeks long workshops or as a part of Financial Engineering degree programs. On the online front, there are online learning portals such as QuantInsti, Coursera, Udemy, Udacity, edX, & Open Intro, they have expert faculty from mathematics and computer science backgrounds who share their experiences and strategy ideas/tactics with you during the course.

Keeping in mind the need for an online program for working professionals, we at QuantInsti™, offer a comprehensive hands-on course called Executive Programme in Algorithmic Trading (EPAT™). The salient features of the course are listed in the table below. The objective of the course is to make students market ready upon successful completion of the course work. For those who want to learn high-frequency trading, there are limited dedicated resources to do the same.

It is often seen that students who would like to get placed in high-frequency trading firms or in quantitative roles, go for MFE programs. Most of the MFE programs give a very good overview of mathematical concepts including Calculus, PDE, pricing models. For learning quantitative trading, what is also required is the implementation of these skills/theories on actual market data under simulated environment. It is always better to get trained by practitioners and traders themselves if the aim is to go out there and make some money! However, if you would like to pursue research in these fields, then taking a more academic path is recommended.

Step 3: Get placed, learn more and implement on the job

Once you get placed in an algorithmic trading firm, you are expected to apply and implement your algorithmic trading knowledge in real markets for your firm. As a new recruit, you are also expected to have knowledge of other processes as well, which are part of your workflow chain.

As an example, firms which trade low latency strategies will usually have their platform built on C++, whereas in trading firms where latency is not a critical parameter, trading platforms can be based on a programming language like Python. Thus, it becomes essential for wannabe and new quant developers to have an understanding of both the worlds.

New recruits working on specific projects may be given a brief training to get a good grasp on the subject. Trading firms usually make their new recruits spend time on different desks (e.g. quant desk, programming, risk management desk) which gives them a fair understanding of the work process followed in the organization. To put it in subtle words, learning in the algorithmic world never stops!!


This article gave an overview of algorithmic trading, the core areas to focus on, and the resources that serious wannabe traders can explore to learn algorithmic trading. So, if you wish to master this new domain and build an exciting career in algorithmic trading start learning today!

Next Step:

If you want to learn more about Executive Programme in Algorithmic Trading (EPAT™) you can contact us here. We will be glad to address any queries that you have regarding the course or on algorithmic trading in general. Contact us.



Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Machine Learning: An Introduction to Decision Trees

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

Decision Trees for Machine LearningBy Milind Paradkar

Decision tree defined

A decision tree is one of the widely used algorithms for building classification or regression models in data mining and machine learning. A decision tree is so named because the output resulting from it is the form of a tree structure.

Visualizing a sample dataset and decision tree structure

Consider a sample stock dataset as shown in the table below. The dataset comprises of Open, High, Low, Close Prices and Volume indicators (OHLCV) for the stock.

Let us add some technical indicators (RSI, SMA, LMA, ADX) to this dataset. We also add the “Change” column which signifies the daily returns based on the close price of the stock. In the “Change” column, “Up” denotes positive return, whereas “Down” signifies a negative return. The dataset after adding the various technical indicators is shown below.

Now, suppose we want to predict the daily change (Up/Down) using these technical indicators. We can do this by applying a decision tree algorithm on this dataset. First, we will have to split our dataset into two parts; training dataset and test dataset. We train the decision tree algorithm on this training dataset and thereafter, apply it on a test dataset to make the predictions. The resulting output from the training dataset can be viewed in the form of a tree structure as shown below.

Sample illustration of a Decision Tree

Before we understand how a decision tree algorithm works, let us first understand the tree structure.

Components of a decision tree

A decision tree structure consists of a root node, test nodes, and decision nodes (leaves). The root node is the main node in a decision tree. In the above decision tree, we start with RSI>50, thus the RSI indicator is used as the root node.

After the root node, each test node splits the data into further parts according to some set criteria. In the decision tree above, we have SMA> 80, LMA>55, and ADX>25. The data is split further based on these indicators which act as test nodes.

The final node is the leaf node. In the decision tree diagram, the nodes containing the Up/Down class along with the probability value indicating the probability of the target attribute are the leaf nodes.

Thus, the entire dataset gets classified by navigating from the root node of the tree decision tree down to the leaf, according to the set criteria’s.

Components of a Decision Tree

How does a decision tree algorithm work?

Now that we have understood the decision tree structure, let us understand how a decision tree is constructed.

Decision Tree Inducer

A decision tree is constructed by a decision tree inducer. There are various decision tree inducers such as ID3, C4.5, CART etc. A decision tree inducer is basically an algorithm that automatically constructs a decision tree from a given dataset. Typically, the goal of a decision tree inducer is to construct an optimal decision tree based on a specified target function. An example of a target function can be a minimization of the number of nodes of the decision tree. Another example can be minimizing the generalization error.

Although, we said that the inducer constructs a decision tree automatically, in reality, the decision tree inducer follows a certain method. There are two prevalent methods used by the decision tree inducers in general. These include the widely used top-down method and the less popular, bottom-up method.

Top-down Method Explained

Under the top-down method, an inducer creates the decision tree in a top-down, recursive manner. Let us use our previously created technical indicators to understand this top-down, recursive method.

The technical indicators that we created were RSI, SMA, LMA, and the ADX indicator. Our objective is to predict the daily change (Up/Down) using these technical indicators. These indicators are considered as the attributes by our decision tree algorithm.

From among these indicators (attributes), which of the indicator (attribute) will be used by the inducer as the root node to start with? How will the remaining attributes be split?

All this is decided by the inducer using a discrete splitting function. The discrete splitting function uses certain criteria (e.g. information gain, gini index, distance measure) to determine the best attribute to start with and split the training dataset.

In each iteration, the inducer algorithm partitions the training dataset using the outcome of a discrete function of the input attribute. After the selection of an appropriate split, each node further subdivides the training set into smaller subsets, until no further splitting is possible or when a stopping criterion is fulfilled. After the complete creation of the tree, it is pruned using certain pruning rules to reduce classification errors.

This is how a decision tree gets constructed which can be used for making predictions in machine learning. In our future posts, we will demonstrate how to construct a decision tree in python and will also explore some machine learning models based on decision trees.

Next Step

If you want to learn various aspects of Algorithmic trading then check out the Executive Programme in Algorithmic Trading (EPAT™). The course covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. EPAT™ equips you with the required skill sets to be a successful trader. Enroll now!

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Top Skills for nailing Quant or Trader Interview

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

Want to secure a Quant or Trader’s job? Following are the areas that you should focus on to get your dream job.


1) Equity Derivatives/Options

Derivatives are highly traded instruments. Knowledge of option pricing models, greeks, volatility, hedging, and various option strategies are a must.

2) Programming

Sound programming skills are required for backtesting, writing low latency and super-efficient codes.

3) Statistics & Probability

Probability and statistics form a key part of trading. Basic statistics, time series, multivariate analysis etc. is used for formulating strategies, and risk-management.

4) Markets and the Economy

Good knowledge of how markets and economy work.

5) Numerical & Brain Teasers

Numerical and thinking questions test the ability to work out the answer with sound reasoning.

6) Question about You

These are asked to determine if you are a good fit for the job.

7) Job Awareness Questions

Job Awareness questions evaluate your understanding of the job profile.

Get Free Career Advice from a leading HFT firm’s Head of Technology and a panel of quants/traders on 25th Jan at 6 PM IST by registering here:

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Implementing Pairs Trading Using Kalman Filter

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

Implementing Pairs Trading Using Kalman Filter

By Dyutiman Das

This article is the final project submitted by the author as a part of his coursework in Executive Programme in Algorithmic Trading (EPAT™) at QuantInsti. Do check our Projects page and have a look at what our students are building.


Some stocks move in tandem because the same market events affect their prices. However, idiosyncratic noise might make them temporarily deviate from the usual pattern and a trader could take advantage of this apparent deviation with the expectation that the stocks will eventually return to their long term relationship. Two stocks with such a relationship form a “pair”. We have talked about the statistics behind pairs trading in a previous article.

This article describes a trading strategy based on such stock pairs. The rest of the article is organized as follows. We will be talking about the basics of trading an individual pair, the overall strategy that chooses which pairs to trade and present some preliminary results. In the end, we will describe possible strategies for improving the results.

Pair trading

Let us consider two stocks, x and y, such that

y = \alpha + \beta x + e

\alpha and \beta are constants and e is white noise. The parameters {\alpha, \beta} could be obtained from a linear regression of prices of the two stocks with the resulting spread 

e_{t} = y_{t} – (\alpha + \beta x_{t})

Let the standard deviation of this spread be \sigma_{t}. The z-score of this spread is

z_{t} = e_{t}/\sigma_{t}

Trading Strategy

The trading strategy is that when the z-score is above a threshold, say 2, the spread can be shorted, i.e. sell 1 unit of y and buy \beta units of x. we expect that the relationship between x and y will hold in the future and eventually the z-score will come down to zero and even go negative and then the position could be closed. By selling the spread when it is high and closing out the position when it is low, the strategy hopes to be statistically profitable. Conversely, if the z-score is below a lower threshold say -2, the strategy will go long the spread, i.e. buy 1 unit of y and sell \beta units of x and when the z score rises to zero or above the position can be closed realizing a profit.

There are a couple of issues which make this simple strategy difficult to implement in practice:

  1. The constants \alpha and \beta are not constants in practice and vary over time. They are not market observables and hence have to be estimated with some estimates being more profitable than others.
  2. The long term relationship can break down, the spread can move from one equilibrium to another such that the changing {\alpha,\beta} gives an “open short” signal and the spread keeps rising to a new equilibrium such that when the “close long” signal come the spread is above the entry value resulting in a loss.

Both of these facts are unavoidable and the strategy has to account for them.

Determining Parameters

The parameters {\alpha, \beta} can be estimated from the intercept and slope of a linear regression of the prices of y against the prices of x. Note that linear regression is not reversible, i.e. the parameters are not the inverse of regressing x against y. So the pairs (x,y) is not the same as (y,x). While most authors use ordinary least squares regression, some use total least squares since they assume that the prices have some intraday noise as well. However, the main issue with this approach is that we have to pick an arbitrary lookback window.

In this paper, we have used Kalman filter which is related to an exponential moving average. This is an adaptive filter which updates itself iteratively and produces \alpha, \beta, e and \sigma simultaneously. We use the python package pykalman which has the EM method that calibrates the covariance matrices over the training period.

Another question that comes up is whether to regress prices or returns. The latter strategy requires holding equal dollar amount in both long and short positions, i.e. the portfolio would have to be rebalanced every day increasing transaction cost, slippage, and bid/ask spread. Hence we have chosen to use prices which is justified in the next subsection.

Stability of the Long Term Relationship

The stability of the long term relationship is determined by determining if the pairs are co-integrated. Note that even if the pairs are not co-integrated outright, they might be for the proper choice of the leverage ratio. Once the parameters have been estimated as above, the spread time series e_{t} is tested for stationarity by the augmented Dickey Fuller (ADF) test. In python, we obtain this from the adfuller function in the statsmodels module. The result gives the t-statistics for different confidence levels. We found that not many pairs were being chosen at the 1% confidence level, so we chose 10% as our threshold.

One drawback is that to perform the ADF test we have to choose a lookback period which reintroduces the parameter we avoided using the Kalman filter.

Choosing Sectors and Stocks

The trading strategy deploys an initial amount of capital. To diversify the investment five sectors will be chosen: financials, biotechnology, automotive etc. A training period will be chosen and the capital allocated to each sector is decided based on a minimum variance portfolio approach. Apart from the initial investment, each sector is traded independently and hence the discussion below is limited to a single sector, namely financials.

Within the financial sector, we choose about n = 47 names based on large market capitalization. We are looking for stocks with high liquidity, small bid/ask spread, ability to short the stocks etc.  Once the stock universe is defined we can form n (n-1) pairs, since as mentioned above (x,y) is not the same as (y,x). In our financial portfolio, we would like to maintain up to five pairs at any given time. On any day that we want to enter into a position (for example the starting date) we run a screen on all the n (n-1) pairs and select the top pair(s) according to some criteria some of which are discussed next.

Choosing Pairs

For each pair, the signal is obtained from the Kalman filter and we check if |e| > nz \sigma, where nz is the z-score threshold to be optimized. This ensures that this pair has an entry point. We perform this test first since this is inexpensive. If the pair has an entry point, then we choose a lookback period and perform the ADF test.

The main goal of this procedure is not only to determine the list of pairs which meets the standards but rank them according to some metrics which relates to the expected profitability of the pairs.

Once the ranking is done we enter into the positions corresponding to the top pairs until we have a total of five pairs in our portfolio.


In the following, we calibrated the Kalman filter over Cal11 and then used the calibrated parameters to trade in Cal12. In the following, we kept only one stock-pair in the portfolio.

In the tests shown we kept the maximum allowed drawdown per trade to 9%, but allowed a maximum loss of 6% in one strategy and only 1% in the other. As we see from above the performance improves with the tightening of the maximum allowed loss per trade. The Sharpe ratio (assuming zero index) was 0.64 and 0.81 respectively while the total P&L was 9.14% and 14%.

The thresholds were chosen based on the simulation in the training period.

Future Work

  1. Develop better screening criterion to identify the pairs with the best potentials. I already have several ideas and this will be ongoing research.
  2. Optimize the lookback window and the buy/sell Z-score thresholds.
  3. Gather more detailed statistics in the training period. At present, I am gathering statistics of only the top 5 (based on my selection criteria). However, in future, I should record statistics of all pairs that pass. This will indicate which trades are most profitable.
  4. In the training period, I am measuring profitability by the total P&L of the trade, from entry till the exit signal is reached. However, I should also record max profit so that I could determine an earlier exit threshold.
  5. Run the simulation for several years, i.e. calibrate one year and then test the next year. This will generate several year’s worths of out-of-sample tests. Another window to optimize is the length of the training period and how frequently the Kalman filter has to be recalibrated.
  6. Expand the methodology to other sectors beyond financials.
  7. Explore other filters instead of just Kalman filter.

Next Steps

If you are a coder or a tech professional looking to start your own automated trading desk. Learn automated trading from live Interactive lectures by daily-practitioners. Executive Programme in Algorithmic Trading (EPAT™) covers training modules like Statistics & Econometrics, Financial Computing & Technology, and Algorithmic & Quantitative Trading. Enroll now!



The work presented in the article has been developed by the author, Mr. Dyutiman Das. The underlying codes which form the basis for this article are not being shared with the readers. For readers who are interested in further readings on implementing pairs trading using Kalman Filter, please find the article below.

Link: Statistical Arbitrage Using the Kalman Filter by Jonathan Kinlay


Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Market Impact Cost

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+

By Milind Paradkar

Market impact cost, a very important component of trading costs get closely tracked by portfolio managers as it can make or break a fund’s performance. In this post, we will throw some light on market impact cost, and identify its sources and the different means adopted by portfolio managers to mitigate it.  (more…)

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more

Recommended Quant Readings for you – Best of 2016!

Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Recommenced Quant Articles of 2016

Recommenced Quant Articles of 2016

As 2016 nears its finish line, here we are with the list of recommended reading on our blog with the top-rated blog posts, as voted by you! Enjoy the last few days doing what you love most! Read on.

System Architecture of Algorithmic Trading

This one is straight out of a lecture in the curriculum of QuantInsti’s Executive Programme in Algorithmic Trading (EPAT™). It compares the traditional trading structure with algorithmic trading architecture and highlights the complexities in the latter. The post explains the three core components of the trading server: Complex Event Processing Engine (the brain), Order Management System (the limbs) and the Data Storage component. Life Cycle of the entire system is also explained so that the readers under what happens when a data package is received from the exchange, where trading decisions happen, how risk is monitored and how are orders managed.

 Backtesting platforms for quants

There are many platforms out there and for beginners, it is often confusing to pick the most relevant for them. The posts list out the USPs of available platforms so that you can make an informed choice before you start using a platform for backtesting. It is important to make this decision carefully as you would require to spend enough time on one platform to get comfortable with it!

A Pairs Trading quant project, a working strategy with open sourced code in R

In this highly insightful article, QuantInsti’s EPAT™ graduate, Jacques Joubert shares his project work on Statistical Arbitrage in R programming language. For readers who are more comfortable in Excel, they can download a pair trading model in Excel here to get started. He talks briefly about the history of Statistical Arbitrage before moving on to the strategy and its markdown in R programming language.

Algorithmic Trading Strategies

What are the different Algo Trading Strategies? What are the strategy paradigms and modelling ideas associated with each strategy? How do we build an Algo trading strategy? These are some of the key questions answered in this in-depth article. QuantInsti’s article on Algorithic Trading Strategies covers the following:

  • Momentum based strategies
  • Arbitrage
  • Statistical Arbitrage
  • Market Making
  • Machine Learning Based

Python as programming language for DIY traders

Python has sufficed as one of the most popular programming languages for algorithmic traders. In this set of articles, we have talked about Zipline, building technical indicators and the benefits of learning Python for trading. The articles came into light during the webinar on Automated trading using Python conducted by Dr. Yves Hilpisch. This year, we also had Dr. Hui Liu conducting a webinar on implementing Python in Interactive Broker’s C++ based API. Both Dr. Yves and Dr. Hui, who are two of the renowned names in the field of automated trading, have joined QuantInsti’s impressive line-up of outstanding faculty for EPAT™.

Learn Machine Learning for Trading

Machine Learning and Artificial Intelligence are the most sought-after streams of technology in this era. As trading has become automated, Machine Learning’s importance has only become critical for maintaining competency in the market. From fetching historical information to placing orders to buy or sell in the market, machine learning is an integral part of Automated trading and we have covered it in detail on our blog.

5 things to know about Algorithmic Trading to get started (in India)

As Algorithmic trading picks up pace in India, more and more conventional traders and beginners are wanting to know about this lucrative field. However, owing to shortage of resources in the market, QuantInsti decided to churn out a very primitive article for amateurs who want to step out in the world of algorithmic trading. Explained in basic language, this article covers all the things one needs to know before starting algorithmic trading.

Next Step

We would love to hear from you – why you liked any or all. If you would like to read something specific in 2017, all suggestions are welcome!


Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+
Read more