Using Modified Rainbow for Enhancing Reinforcement Learning for Stock Trading-NASDAQ’s Stocks as Examples

Abstract

Algorithmic trading in the stock market has being attracting significant commercial interests in financial industry but it’s recognized as a challenging stochastic control problem This paper establishes a trading environment based on the real trading rules of stock market and uses two techniques of reinforcement learning - Markov decision process (MDP) and Deep Q-Learning (DQN) which are enhanced by the six state-of-the-art optimization methods in Rainbow In addition a compatible optimization method called Sibling Search is designed for the defined environment and several experiments show that Sibling Search can improve performance significantly Combined Rainbow with Sibling Search and it is a novel method called Sibling Rainbow Using other optimization methods and Sibling Search in Sibling Rainbow the environment learns and simulates transactions on 10 NASDAQ stocks as experiments The experiments show that the environment even with a high transaction tax of 0 25% in the testing data the returns increased 11% after using Sibling Search Using the original six methods in Rainbow the returns can reach up to 43% in one year and using the Sibling Rainbow the return in one year reaches up to 300% This confirms reinforcement learning can be used for algorithmic trading in stock market and higher performance can be gained by using optimization in Rainbow and Sibling Rainbow This study contributes in the field with followings: 1 Design the Markov Decision Process (MDP) for stock trading tasks which can be easily extended for future researches 2 Successfully apply Q-Learning to learn trading behavior with an average 54% hit rate in automatic trading 3 Propose Sibling Search a method to increase data usage which is proved to enhance the performance of Q-Learning 4 Increase the optimization method in Rainbow with Sibling Search the first attempt to modify Rainbow for stock trading which can be used as the baseline for future studies

Date of Award	2019
Original language	English
Supervisor	Tzone-I Wang (Supervisor)