Data selection to avoid overfitting for foreign exchange intraday trading with machine learning

• Published in: Applied soft computing Vol. 108; p. 107461
• Main Authors: Peng, Yuan-Long, Lee, Wei-Po
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.09.2021
• Subjects: Algorithmic trading; Foreign exchange; Machine learning; Overfitting; Algorithmic trading; Foreign exchange; Machine learning; Overfitting
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2021.107461

Algorithmic trading requires tuning hyperparameters to fit the time series data; however, it often suffers from overfitting of data that can lead to loss of money in action. Further, only a few studies discuss how to select trading exchange pairs and frequencies in response to the fitness of machine learning models. To cope with these problems, we developed a log-distance path loss model (to measure and reduce the overfitting in data modeling and determine exchange pairs and frequencies effectively. We conducted several experiments for different metrics using several influential factors such as machine learning models, learning objectives, trading strategies, and hyperparameter turning cases to validate the proposed approach. The obtained results indicate that the proposed metric is significantly superior to other methods in terms of accuracy, in-sample return (i.e., return of training data), and F1-score. Thus, using our path loss metric to guide data modeling, we provide a method to deal with the overfitting problem and yield positive trading returns. •A metric can measure over-fitting in FX trading.•Selecting trading frequency and currency pair will lead to positive return.•Results were validated with extensive experiments including four popular machine learning models.•We defined an objective that simultaneously considering the long and short position returns and the spread.•Except L1/L2 regularization and dropout, selecting trading data is one of solutions to over-fitting in FX trading.