A Deep Reinforcement Learning Framework for Optimal Trade Execution

• Published in: Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track Vol. 12461; pp. 223 - 240
• Main Authors: Lin, Siyu, Beling, Peter A.
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Deep reinforcement learning; Optimal trade execution; Shaped reward structure; US equities; Zero ending inventory constraint
• ISBN: 9783030676698; 3030676692
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-67670-4_14

In this article, we propose a deep reinforcement learning based framework to learn to minimize trade execution costs by splitting a sell order into child orders and execute them sequentially over a fixed period. The framework is based on a variant of the Deep Q-Network (DQN) algorithm that integrates the Double DQN, Dueling Network, and Noisy Nets. In contrast to previous research work, which uses implementation shortfall as the immediate rewards, we use a shaped reward structure, and we also incorporate the zero-ending inventory constraint into the DQN algorithm by slightly modifying the Q-function updates relative to standard Q-learning at the final step. We demonstrate that the DQN based optimal trade execution framework (1) converges fast during the training phase, (2) outperforms TWAP, VWAP, AC and 2 DQN algorithms during the backtesting on 14 US equities, and also (3) improves the stability by incorporating the zero ending inventory constraint.