Hybrid Reinforcement Learning for STAR-RISs: A Coupled Phase-Shift Model Based Beamformer

• Published in: IEEE journal on selected areas in communications Vol. 40; no. 9; pp. 2556 - 2569
• Main Authors: Zhong, Ruikang, Liu, Yuanwei, Mu, Xidong, Chen, Yue, Wang, Xianbin, Hanzo, Lajos
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Array signal processing; Beamforming; Channel estimation; Communications systems; Computational modeling; deep reinforcement learning (DRL); Energy consumption; Hybrid control; Machine learning; Markov processes; MISO (control systems); Optimization; Phase shift; Power consumption; reconfigurable intelligent surfaces (RISs); Reinforcement learning; simultaneous transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs); Stars; Surface waves
• DOI: 10.1109/JSAC.2022.3192053
• ISSN: 0733-8716

A simultaneous transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) assisted multi-user downlink multiple-input single-output (MISO) communication system is investigated. In contrast to the existing ideal STAR-RIS model assuming an independent transmission and reflection phase-shift control, a practical coupled phase-shift model is considered. Then, a joint active and passive beamforming optimization problem is formulated for minimizing the long-term transmission power consumption, subject to the coupled phase-shift constraint and the minimum data rate constraint. Despite the coupled nature of the phase-shift model, the formulated problem is solved by invoking a hybrid continuous and discrete phase-shift control policy. Inspired by this observation, a pair of hybrid reinforcement learning (RL) algorithms, namely the hybrid deep deterministic policy gradient (hybrid DDPG) algorithm and the joint DDPG & deep-Q network (DDPG-DQN) based algorithm are proposed. The hybrid DDPG algorithm controls the associated high-dimensional continuous and discrete actions by relying on the hybrid action mapping. By contrast, the joint DDPG-DQN algorithm constructs two Markov decision processes (MDPs) relying on an inner and an outer environment, thereby amalgamating the two agents to accomplish a joint hybrid control. Simulation results demonstrate that the STAR-RIS has superiority over other conventional RISs in terms of its energy consumption. Furthermore, both the proposed algorithms outperform the baseline DDPG algorithm, and the joint DDPG-DQN algorithm achieves a superior performance, albeit at an increased computational complexity.