Deep Reinforcement Learning for Practical Phase Shift Optimization in RIS-assisted Networks over Short Packet Communications

We study the practical phase shift design in a non-ideal reconfigurable intelligent surface (RIS)-aided ultra-reliable and low-latency communication (URLLC) system under finite blocklength (FBL) regime by leveraging a novel deep reinforcement learning (DRL) algorithm named as twin-delayed deep deterministic policy gradient (TD3). First, assuming industrial automation system with multiple actuators, the signal-to-interference-plus-noise ratio (SINR) and achievable rate in FBL regime are identified for each actuator in terms of the phase shift configuration matrix at the RIS. The channel state information (CSI) variations due to feedback delay are also considered that result in channel coefficients' obsolescence. Then, the problem framework is proposed where the objective is to maximize the total achievable FBL rate in all ACs, subject to the practical phase shift constraint at the RIS elements. Since the problem is intractable to solve using conventional optimization methods, we resort to employing an actor-critic policy gradient DRL algorithm based on TD3, which relies on interacting RIS with FA environment by taking actions which are the phase shifts at the RIS elements, to maximize the expected observed reward, which is defined as the total FBL rate. The numerical results show that optimizing the practical phase shifts in the RIS via the proposed TD3 method is highly beneficial to improve the network total FBL rate in comparison with typical DRL methods.