Reinforcement learning-based pilot assistance system for management of failures

• Published in: 2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC) pp. 1 - 10
• Main Authors: Amorim, Joao P. F., Ribeiro, Carlos H. C., Cerdeira, Paula B. O.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2023
• Subjects: Aircraft navigation; Decision making; fault handling; Q-learning; quick reference handbook; Reinforcement learning; Routing protocols; Sensitivity; Training; Transfer learning
• ISSN: 2155-7209
• DOI: 10.1109/DASC58513.2023.10311215

The occurrence of abnormal or emergencies in flight is usually managed by aircraft crews which also can have remote assistance from controllers like the ATM (Air Traffic Management) or ANS (Air Navigation System) assisting with flight plans and routes. However, during certain scenarios, the pilot is responsible and must validate information, conditions, and protocols from the Quick Reference Handbook (QRH), which is a manual that contains step-by-step processes applicable to certain critical scenarios. Using the QRH works well where one abnormal situation happens at a time, but, in situations with multiple failures, the decision-making and steps to follow may be unclear and very dependent on pilot experience. We propose herein a learning process for an Intelligent Assistant to assist pilots to mitigate system failures based on the use of Reinforcement Learning. More specifically, the system under study was the Air Management System (AMS), responsible for controlling the air supply to the cabins keeping proper temperature and pressure, an essential function to make the interior livable and comfortable for passengers during flight. We then defined a Deep Q-learning neural network with an experience replay strategy, importance sampling, and an updated reward function that is based on a fault tree for the system, returning negative rewards as weighted sums of consequences such as critical failure in the auxiliary power unit, critical failure in the wing anti-icing, etc. For single-failure situations, the learned model was able to produce an appropriate action policy concerning QRH indications. We produced additional insights on the treatment of multiple failures, an aspect that is not fully tackled in QRHs, with results that suggest a particular sensitivity to failure probabilities and reward function adjustments. The main potential implication of our study is the proposal and assessment of a decision-making support system for pilots in situations that are not fully considered in QRHs.