OH-DRL: An AoI-Guaranteed Energy-Efficient Approach for UAV-Assisted IoT Data Collection

• Published in: IEEE transactions on wireless communications Vol. 24; no. 6; pp. 5009 - 5022
• Main Authors: Yang, Bowen, Yu, Yao, Hao, Xin, Yeoh, Phee Lep, Zhang, Junxiong, Guo, Lei, Li, Yonghui
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Age of information; Algorithms; Array signal processing; Autonomous aerial vehicles; Beamforming; Clusters; Convergence; Data collection; Data models; Energy consumption; Energy efficiency; Flight time; Hovering; Internet of Things; Machine learning; Optimization; Reconfigurable intelligent surfaces; Scheduling; Simulation; simultaneous transmission and reflection; Training; Trajectory; uncrewed aerial vehicles; Unmanned aerial vehicles
• ISSN: 1536-1276; 1558-2248
• DOI: 10.1109/TWC.2025.3545451

In this paper, we propose a hierarchical optimization approach that guarantees the maximum age of information (AoI) for uncrewed aerial vehicle (UAV) assisted Internet-of-Things (IoT) data collection. Our model is based on an energy-efficient simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) beamforming model. We formulate the optimization to minimize the UAV flight energy consumption subject to a maximum average AoI threshold by optimizing the UAV trajectory, IoT device scheduling, and STAR-RIS beamforming. To solve this, we develop an optimization-based hierarchical deep reinforcement learning (OH-DRL) algorithm that decomposes the formulated problem into an inter-cluster UAV visiting policy and STAR-RIS-based intra-cluster IoT scheduling policy. In OH-DRL, we jointly optimize the two policies in a high-level loop and a low-level loop, respectively. In the high-level loop, we design an AoI-guided DRL algorithm to determine the AoI-guaranteed UAV hovering position with minimal flight distance. In the low-level loop, a semidefinite relaxation (SDR)-based optimization algorithm further reduces the UAV's flying time by minimizing the average AoI. Simulation results validate that OH-DRL achieves better convergence performance and energy-saving efficiency across different network scales. Compared to the state-of-the-art DRL algorithm, OH-DRL reduces the UAV flight energy consumption by 14.4% and decreases the number of training episodes required for convergence by 66%