Applying Deep Reinforcement Learning to Algorithmic Trading
Abstract
At the moment, there is a large volume of literature on exchange trading. Obviously, every year the mathematical base of work is becoming more complicated along with an increase in computing power, machines can process more metrics from year to year and produce more accurate solutions per unit of time. The use of deep learning has already proven itself well, as the application of this approach has given a quantum leap in algorithmic trading.
The article presents an algorithm for trading long contracts with one asset in the financial market in the Python programming language using the LSTM neural network using the Keras library, which is used as a demo example in the Reinforcement Learning discipline. The formalized LSTM model solves the vanishing gradient problem, which can hold the gradient of the objective function relative to the state signal. As applied to our problem, such an improvement in the model allows us to collect data on certain patterns of price changes, that is, when predicting the price of the next step, we rely not only on the data of the previous step, but also on earlier data, when there was a similar state of the environment. Sharpe Ratio is used to determine the optimal strategy and make decisions at each time of application. The optimal minimum time period for the model operation has been determined; the signal transmission delay from the moment the market situation changes until the signal is received by the model, which will be infinitely small, and the computing power will be considered infinitely large. These assumptions give the right to say: when the market situation changes, the model is instantly ready to react and make a decision to sell, buy or hold an asset.
References
[2] Ponomarev E.S., Oseledets E.V., Cichocki A.S. Algorithmic Trading with Reinforcement Learning. Information Processes. 2019; 19(2):122-131. Available at: https://www.elibrary.ru/item.asp?id=38538082 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[3] Kasyanov R.A. MIFID II The Regulation of Algorithmic Trading and High-frequency Algorithmic Trading. Financial Law. 2018; (6):3-7. Available at: https://www.elibrary.ru/item.asp?id=35133781 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[4] Krasnikov V.S. Algorithm for Placing Stop Orders when Trading Long Contracts. Scientific notes of young scientists. 2018; (4):21-30. Available at: https://www.elibrary.ru/item.asp?id=35588843 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[5] Shaipova S.A. High-Frequency Trading and its Development in the Trade Market. Bulletin of the Academy of Law and Management. 2018; (3):146-149. Available at: https://www.elibrary.ru/item.asp?id=36319246 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[6] Xie M., Li H., Zhao Yu. Blockchain financial investment based on deep learning network algorithm. Journal of Computational and Applied Mathematics. 2020; 372:112723. (In Eng.) DOI: https://doi.org/10.1016/j.cam.2020.112723
[7] Borges T.A., Neves R.F. Ensemble of machine learning algorithms for cryptocurrency investment with different data resampling methods. Applied Soft Computing. 2020; 90:106187. (In Eng.) DOI: https://doi.org/10.1016/j.asoc.2020.106187
[8] Palmov S.V., Artyushkina E.S. Deep Learning: Definition and Distinctive Features. Forum molodyh uchenyh. 2020; (3):311-316. Available at: https://www.elibrary.ru/item.asp?id=42857736 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[9] Chistova E.V., Shelmanov A.O., Smirnov I.V. Natural Language Dialogue Modelling with Deep Learning. Proceedings of the Institute for Systems Analysis Russian Academy of Sciences. 2019; 69(1):105-115. (In Russ., abstract in Eng.) DOI: https://doi.org/10.14357/20790279190110
[10] Sineglazov V.M., Koniushenko R.S. Deep learning fuzzy classifier. Electronics and Control Systems. 2019; 2(60):35-42. (In Eng.) DOI: https://doi.org/10.18372/1990-5548.60.13813
[11] Potemkin A.V. Processing Different Information Using the Deep Learning of Neural Networks. Soft Measurements and Computing. 2019; (9):44-48. Available at: https://www.elibrary.ru/item.asp?id=41688926 (accessed 02.09.2020). (In Russ., abstract in Eng.)
[12] Efremtsev V.G., Ejtemtsev N.G., Teterin E.P., Teterin P.E., Gantsovsky V.V. Deep Learning Application for Box-office Evaluation of Images. Computer Optics. 2020; 44(1):127-132. (In Russ., abstract in Eng.) DOI: https://doi.org/10.18287/2412-6179-CO-515
[13] Liu F., et al. State representation modeling for deep reinforcement learning based recommendation. Knowledge-Based Systems. 2020; 205:106170. (In Eng.) DOI: https://doi.org/10.1016/j.knosys.2020.106170
[14] Pandey V., Wang E., Boyles S.D. Deep reinforcement learning algorithm for dynamic pricing of express lanes with multiple access locations. Transportation Research Part C: Emerging Technologies. 2020; 119:102715. (In Eng.) DOI: https://doi.org/10.1016/j.trc.2020.102715
[15] Liu F., Tang R., et al. Top-aware reinforcement learning based recommendation. Neurocomputing. 2020; 417:255-269. (In Eng.) DOI: https://doi.org/10.1016/j.neucom.2020.07.057
[16] Park H., Sim M.K., Choi D.G. An intelligent financial portfolio trading strategy using deep Q-learning. Expert Systems with Applications. 2020; 158:113573. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2020.113573
[17] Lu R., Hong S.H., Zhang X. A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach. Applied Energy. 2018; 220:220-230. (In Eng.) DOI: https://doi.org/10.1016/j.apenergy.2018.03.072
[18] Tymchenko B., Marchenko Ph., Spodarets D. Segmentation of Cloud Organization Patterns from Satellite Images Using Deep Neural Networks. Herald of Advanced Information Technology. 2020; 3(1):352-361. (In Eng.) DOI: https://doi.org/10.15276/hait01.2020.2
[19] Gorshenin A.K., Kuzmin V.Yu. Analysis of Configurations of LSTM Networks for Medium-Term Vector Forecasting. Informatics and Applications. 2020; 14( 1): 10-16. (In Eng.) DOI: https://doi.org/10.14357/19922264200102
[20] Chakole J. B., et al. A Q-learning agent for automated trading in equity stock markets. Expert Systems with Applications. 2021; 163:113761. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2020.113761
[21] Betancourt C., Chen W.-H. Deep reinforcement learning for portfolio management of markets with a dynamic number of assets. Expert Systems with Applications. 2021; 164:114002. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2020.114002
[22] Lei K., Zhang B., Li Y., Yang M., Shen Y. Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading. Expert Systems with Applications. 2020; 140:112872. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2019.112872
[23] Nguyen T.T., et al. A multi-objective deep reinforcement learning framework. Engineering Applications of Artificial Intelligence. 2020; 96:103915. (In Eng.) DOI: https://doi.org/10.1016/j.engappai.2020.103915
[24] Riesener M., Dölle C., Dierkes C., Jank M.-H. Applying Supervised and Reinforcement Learning to Design Product Portfolios in Accordance with Corporate Goals. Procedia CIRP. 2020; 91:127-133. (In Eng.) DOI: https://doi.org/10.1016/j.procir.2020.02.157
[25] Jeong G., Kim H.Y. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems with Applications. 2019; 117:125-138. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2018.09.036
[26] Jeong G., Kim H.Y. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems with Applications. 2019; 117:125-138. (In Eng.) DOI: https://doi.org/10.1016/j.eswa.2018.09.036
[27] Gao W., Su Ch. Analysis on block chain financial transaction under artificial neural network of deep learning. Journal of Computational and Applied Mathematics. 2020; 380:112991. (In Eng.) DOI: https://doi.org/10.1016/j.cam.2020.112991
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication policy of the journal is based on traditional ethical principles of the Russian scientific periodicals and is built in terms of ethical norms of editors and publishers work stated in Code of Conduct and Best Practice Guidelines for Journal Editors and Code of Conduct for Journal Publishers, developed by the Committee on Publication Ethics (COPE). In the course of publishing editorial board of the journal is led by international rules for copyright protection, statutory regulations of the Russian Federation as well as international standards of publishing.
Authors publishing articles in this journal agree to the following: They retain copyright and grant the journal right of first publication of the work, which is automatically licensed under the Creative Commons Attribution License (CC BY license). Users can use, reuse and build upon the material published in this journal provided that such uses are fully attributed.
