By QuantInsti
TL;DR
Bala Murugan Ganeshan, an EPAT alumnus, developed a dual-model deep learning system to predict Bank Nifty index and stock open prices for trading decisions. Using 1-minute and daily data, he tackled challenges like data imbalance, overfitting, and limited hardware by employing tools like TensorFlow, Keras, XGBoost, and SMOTE. His project aligned model predictions to decide trades and emphasised the importance of data engineering and mentorship. This work showcases how AI and structured learning can empower traders to build effective, real-world trading systems.
Predicting market movements has always fascinated traders, especially in high-volatility indices like Bank Nifty. Bala Murugan Ganeshan, an EPAT alumnus, took on the challenge of predicting both index and stock prices using deep learning.
About Bala Murugan Ganeshan:
Bala Murugan Ganeshan is a passionate data science and analytics professional with a strong interest in algorithmic trading and quantitative finance. With a background in engineering and rich experience across domains, Bala pursued the Executive Programme in Algorithmic Trading (EPAT) to sharpen his skills in quantitative modelling and AI-driven trading systems.
In Bala Murugan’s words:
“Structured mentorship, like what EPAT offers, helps turn complex projects into successful learning journeys.”
When he’s not experimenting with neural networks or backtesting trading strategies, Bala enjoys diving into data structures, building scalable systems, and keeping up with the latest in fintech innovation.
About the EPAT project:
Bala’s EPAT project is an approach that blended minute-level and daily data, tackled real-world trading problems like data imbalance and overfitting, and leveraged both traditional indicators and modern AI techniques.
His project on predicting Bank Nifty and constituent stock prices using deep learning reflects his keen analytical mind and ability to bridge financial theory with machine learning applications.
In this article, we walk through Bala’s innovative project that brings together predictive modelling, data science, and practical trading strategy design.
Project Overview: Dual Deep Learning Models for Trading Decisions
Bala’s project was built around two deep-learning models:
- Model 1: Predicts the open price of the Bank Nifty index using high-frequency (1-minute) data.
- Model 2: Predicts the open prices of Bank Nifty’s top five constituents-HDFC Bank, ICICI Bank, Axis Bank, Kotak Mahindra Bank, and SBI, based on daily data.
“The logic was simple. We’d only trade when both models aligned, either both predicted an upward movement (go long), or both predicted a downward movement (go short).”
Focusing on open price prediction made strategic sense, as most index movement occurs overnight, leading to gap-ups or gap-downs.
Data Sourcing and Management
Handling data efficiently was central to this project:
- Index Model:
1. 21 months of one-minute data for training
2. 6 months of one-minute data for testing
- Stock Model:
1. 20 years of daily data for training
2. 1 year of daily data for testing
Cleaning minute-level data for stocks proved difficult due to corporate actions and inconsistencies. So, Bala wisely stuck to daily data for the stock model and used his own MySQL database for seamless storage and cleaning.
Tech Stack and Libraries
Bala’s toolset brought together the best of deep learning and classical data science:
- TensorFlow (backend) and Keras (frontend) - for building neural networks
- Scikit-learn - for evaluation metrics
- XGBoost - to determine feature importance
- TA-Lib - for computing technical indicators
- Imbalanced-learn (IMB Learn) – to address class imbalance using SMOTE
- StandardScaler - for data normalisation
Key Features Used
For each model, Bala engineered a robust set of features:
- Bank Nifty Index Model (minute-level data):
OHLCV from 1-minute intervals, resulting in 5 variables × 360 minutes per day
- Stock Price Model (daily data):
OHLC + a wide range of technical indicators, including:
1. Momentum: RSI, ADX
2. Volatility: ATR, Bollinger Bands
3. Volume-based: MFI
4. Trend-based: Moving Averages
To avoid dimensionality overload, Bala used XGBoost to rank and filter the most predictive features. Top combinations included:
- Open - Close
- Close - Low
Key Challenges and How He Tackled Them
Bala addressed multiple practical hurdles during his project:
1. Too Many Features
- Solution: Used feature importance from XGBoost to keep only the top contributors.
2. Imbalanced Classes
- Problem: 70-30 split between up and down movements
- Solution: Applied SMOTE to oversample the minority class.
3. Scaling Issues
- Solution: Employed StandardScaler to normalise inputs and avoid model bias.
4. Limited Computing Power
- Bala didn’t have a GPU, which made model training slow.
- Workaround: Instead of using all 360 minutes of data, he used only the first and last hour, which he found most influential on the next day's open.
5. Overfitting
Implemented Keras callbacks like:
- EarlyStopping - stops training when validation accuracy drops but training accuracy continues to rise.
- ModelCheckpoint - saves the best-performing model during training.
Learnings and Takeaways
Reflecting on the project, Bala highlighted some of his key takeaways:
“Data engineering takes up most of the time, cleaning, scaling, balancing, it’s all foundational for model performance.”
- Simplify models when hardware is limited.
- Choose features and timeframes wisely.
- Structured mentorship, like what EPAT offers, helps turn complex projects into successful learning journeys.
Final Thoughts
Bala’s project is a shining example of how deep learning can be integrated with financial intuition and structured trading logic. By using model consensus for directional trading, carefully managing data, and dealing with real-world constraints, he created a strategy with practical application potential.
Projects like this show that with the right blend of technical tools, market insight, and educational support, retail traders and quants alike can build intelligent, data-driven trading systems.
EPAT Experience in Bala’s Words
“Joining EPAT was one of the best decisions I made in my career. The curriculum was rigorous and industry-aligned, and the faculty brought in real-world trading experience. But what stood out most was the mentorship and project guidance, it helped me convert theory into a live, working model. The exposure to practical tools and continuous support gave me the confidence to take on complex challenges like this project.”
Frequently Asked Questions
Q: What should traders understand before applying deep learning models to predict market indices or stock prices?
A: Deep learning models, such as LSTMs, CNNs, and Transformers, are capable of capturing complex, nonlinear patterns in financial data including OHLCV, technical indicators, and sentiment. These models are commonly used for:
- Classification (example, predicting market direction: up/down)
- Regression (example, estimating future prices or returns)
- Anomaly detection (example, identifying rare events or outliers)
- Representation learning (example, compressing market states using autoencoders)
However, traders must approach deep learning with caution:
- Large data requirements: Deep learning models typically need large volumes of training data to generalise well. This poses a challenge when working with daily or monthly frequency data, where the sample size may be insufficient. In such cases, simpler models or hybrid approaches might outperform deep nets.
- Overfitting: Due to the high number of parameters, deep models are prone to overfitting, especially in noisy, low-signal environments like financial markets.
- Training instability: Issues like vanishing or exploding gradients can hinder convergence, particularly in RNN-based models.
- Market non-stationarity: Financial time series often violate the stationarity assumptions many models implicitly rely on.
- High noise levels: The signal-to-noise ratio in financial data is low, making it harder for models to extract meaningful patterns.
- Regime shifts: Models trained on historical data may fail during structural changes in the market.
- Low interpretability: It is often difficult to understand or justify model decisions, making them risky for high-stakes applications.
While deep learning holds promise, it should be supported by rigorous validation, walk-forward testing, and domain expertise. Other architectures like GANs are more suitable for generating synthetic data, simulating rare events, or enhancing datasets, rather than for direct predictive modelling.
Q. What are the challenges of working with minute-level and daily financial data?
A: Challenges include handling large volumes of minute-level data, inconsistencies due to corporate actions in daily data, and ensuring accurate data alignment for training and testing. Efficient storage (example, databases) and preprocessing are critical.
Q. What is the role of SMOTE in financial machine learning models?
A: SMOTE (Synthetic Minority Oversampling Technique) is one of several methods used to address class imbalance in financial datasets, such as when upward or downward price movements are disproportionately represented. Here’s how SMOTE and related techniques help:
- SMOTE (Synthetic Minority Oversampling Technique)
Generates synthetic samples for the minority class through interpolation (example, underrepresented price movements), improving model generalization. - ADASYN (Adaptive Synthetic Sampling)
Focuses on generating synthetic samples for “hard-to-learn” minority instances (example, rare volatility spikes), adapting sampling density based on local data distribution. - Random Undersampling
Balances the dataset by reducing majority-class instances (example, trimming overrepresented stable price periods), though it may discard useful information. - Tomek Links
Identifies and removes borderline majority-class samples (example, reducing overlap in bullish vs. bearish labels) to sharpen class separation. - NearMiss
Selects majority-class examples closest to minority instances (example, identifying subtle regime shifts) to maintain meaningful decision boundaries. - SMOTE-ENN (Edited Nearest Neighbors)
Enhances SMOTE by combining oversampling with the removal of noisy or misclassified samples (example, improving signal detection in noisy markets). - Class Weighting
Modifies the loss function to give higher importance to errors on the minority class (example, penalising false negatives in crash prediction). - GANs (Generative Adversarial Networks)
Uses deep learning to generate realistic synthetic data (example, simulating rare market conditions) for training more robust models. - RUSBoost
Combines random undersampling with boosting techniques (example, handling imbalanced credit default datasets) for better performance. - Cluster Centroids
Replaces majority-class samples with cluster centroids (example, summarising repetitive price behaviour) to reduce data volume while preserving structure.
These techniques are essential for improving prediction quality when dealing with imbalanced financial data, a common challenge in real-world trading, fraud detection, or risk modelling.
Q. How can one manage overfitting in deep learning-based trading models?
A: Managing overfitting is crucial when building deep learning models for trading. Common strategies include:
- L1/L2 Regularization: Adds penalties to large weights during training (example, using kernel_regularizer in Keras) to encourage simpler, more generalisable models.
- Dropout: Randomly deactivates a fraction of neurons during training (typically 20–50% in dense layers), reducing reliance on specific pathways and improving robustness.
- Early Stopping: Stops training when the validation loss stops improving, helping to avoid unnecessary learning on noise.
- TimeSeriesSplit: A time-aware cross-validation method from scikit-learn that preserves temporal order, ensuring training data precedes test data and avoiding data leakage, which is critical in financial time series.
Together, these techniques help enhance generalization and reduce the risk of models overfitting to historical market noise.
Q. Why is feature selection important in market prediction models?
A: Feature selection is critical in market prediction to reduce noise, improve model performance, and enhance interpretability. Using too many features can lead to overfitting, inefficient training, longer training times, and less reliable predictions, especially in financial data, which often includes high-dimensional, noisy inputs.
Techniques like XGBoost feature importance help in ranking and selecting only the most predictive variables, improving both speed and accuracy.
Key approaches include:
- Manual Feature Selection: Involves domain knowledge to select relevant features like technical indicators (example, RSI, MACD), price patterns, or fundamental ratios.
- Model-Based Selection: Algorithms like XGBoost, Random Forest, and Lasso can rank feature importance based on their contribution to prediction performance.
- Filter Methods: Use statistical tests (example, mutual information, chi-square) to assess the relationship between features and the target variable.
- Wrapper Methods: Techniques like Recursive Feature Elimination (RFE) iteratively remove features to find the optimal subset.
Additionally, Auto Feature Engineering automates the process of creating and selecting useful features:
- Libraries like Featuretools or tsfresh generate time series features (lags, rolling stats, etc.) automatically.
- Deep Feature Synthesis (DFS) can build higher-level features by stacking primitives over time-based data.
- AutoML frameworks (example, H2O, AutoGluon) combine feature selection and model optimisation end-to-end.
By combining smart feature selection with automated feature engineering, traders can improve model generalisation, reduce computational cost, and extract deeper insights from market data.
Q. How can traders build models without access to high-end hardware like GPUs?
A: Traders without access to high-end local hardware can still build effective models by leveraging cloud-based platforms like Google Colab, AWS, or Azure. These platforms provide access to GPUs and TPUs on a pay-as-you-go or free-tier basis, enabling training of computationally intensive models without upfront hardware investment.
Also, traders can simplify models by reducing data input size (example, using only the first and last hour of trading which are the key time windows) and model complexity, while still capturing meaningful patterns.
Q. What are common evaluation tools used in trading model development?
A: Evaluating trading models goes beyond standard ML metrics like accuracy or R². In trading, the focus is on how model signals perform as real trades under changing market conditions.
Key tools include:
- Backtesting with realistic costs and execution assumptions.
- Walkforward validation to simulate live retraining and avoid overfitting.
- Automated parameter tuning to reduce bias and adapt to current data.
- Regime detection to align strategies with market phases.
Instead of accuracy, use trading-specific metrics:
- Hit rate, Win/Loss ratio, Sortino ratio, and Ulcer Index
These measure real-world performance and risk-adjusted returns more effectively.
Q. What technical indicators are commonly used in stock price prediction?
A: Indicators across categories are used, including:
Momentum-based: RSI, ADX
Volatility-based: ATR, Bollinger Bands
Volume-based: MFI
Trend-based: Moving Averages
Q. Why is data engineering critical in machine learning for trading?
A: Data engineering is the backbone of any trading ML pipeline. Even the most advanced models will fail if built on inconsistent, incomplete, or poorly structured data. Effective data engineering ensures your inputs are accurate, aligned, and ready for modelling and live execution. Here's why it's critical:
1. Data Acquisition & Ingestion
- Connecting to APIs (example, Interactive Brokers, Binance, Alpaca)
- Downloading historical data (price, volume, fundamentals, news)
- Handling real-time data streams, missing ticks, market holidays, and anomalies
2. Data Storage & Architecture
- Using efficient storage formats (example, PostgreSQL, SQLite, Parquet)
- Organising raw, staging, and production datasets
- Applying indexing, partitioning, and version control for fast access and traceability
3. Data Preprocessing
- Cleaning, deduplication, outlier detection
- Adjusting for corporate actions (splits, dividends)
- Resampling and creating rolling windows for various timeframes
4. Feature Engineering Infrastructure
- Automating indicators like RSI, MACD, VWAP
- Generating lag features, volatility bands, percent changes
- Encoding events like earnings reports or macroeconomic news
5. Backtest-Ready Data Preparation
- Aligning features and target variables
- Preventing lookahead and survivorship bias
- Structuring data across asset classes (equities, crypto, FX, options)
6. Data Validation & Monitoring
- Running schema checks and timestamp integrity tests
- Detecting time drift or missing intervals
- Performing regular sanity checks to ensure data reliability
7. Live Deployment Support
- Building online pipelines for real-time feature generation
- Using fault-tolerant systems (example, Kafka, Redis)
- Creating snapshots for rollback and audit purposes.
In short, data engineering ensures that every stage, from ingestion to live trading, is built on a solid and reliable foundation. It minimises errors, reduces latency, and ensures your trading strategy remains robust in both research and production environments.
Next Steps
To build a strong foundation in trading and quantitative finance, follow a structured learning path. Start with Stock Market Basics to understand market structure, financial instruments, and participant roles.
Then, learn to gather and work with data in Getting Market Data: Stocks, Crypto, News & Fundamentals, focusing on accessing, cleaning, and structuring various data types.
Finally, apply machine learning to build intelligent trading systems with the Machine Learning & Deep Learning in Trading (Beginner Track), covering concepts like classification, regression, and model evaluation using real financial datasets. This sequence provides a complete learning path from market knowledge to data sourcing and ML application, equipping you to build practical, data-driven trading strategies.
If you're aiming for a comprehensive, career-oriented learning path, the Executive Programme in Algorithmic Trading (EPAT) is highly recommended. EPAT covers Python programming, machine learning, backtesting, and model evaluation, with real-world trading applications and industry mentorship—ideal for professionals serious about algorithmic trading.