Adaptive Federated Deep Reinforcement Learning for Proactive Content Caching in Edge Computing

• Published in: IEEE transactions on parallel and distributed systems Vol. 33; no. 12; pp. 4767 - 4782
• Main Authors: Qiao, Dewen, Guo, Songtao, Liu, Defang, Long, Saiqin, Zhou, Pengzhan, Li, Zhetao
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; Adaptive algorithms; Caching; Cloud computing; Content caching; Data models; Deep learning; deep reinforcement learning; Delays; Edge computing; Efficiency; Feature extraction; federated learning; Internet of Things; Iterative methods; Machine learning; Optimization; Reinforcement learning; resource constraint; Servers; Training
• DOI: 10.1109/TPDS.2022.3201983
• ISSN: 1045-9219

With the aggravation of data explosion and backhaul loads on 5 G edge network, it is difficult for traditional centralized cloud to meet the low latency requirements for content access. The federated learning ( F L)-based p roactive content c aching (FPC) can alleviate the matter by placing content in local cache to achieve fast and repetitive data access while protecting the users' privacy. However, due to the non-independent and identically distributed (Non-IID) data across the clients and limited edge resources, it is unrealistic for FL to aggregate all participated devices in parallel for model update and adopt the fixed iteration frequency in local training process. To address this issue, we propose a distributed resources-efficient FPC policy to improve the content caching efficiency and reduce the resources consumption. Through theoretical analysis, we first formulate the FPC problem into a stacked autoencoders (SAE) model loss minimization problem while satisfying resources constraint. We then propose an adaptive FPC (AFPC) algorithm combined deep reinforcement learning (DRL) consisting of two mechanisms of client selection and local iterations number decision. Next, we show that when training data are Non-IID, aggregating the model parameters of all participated devices may be not an optimal strategy to improve the FL-based content caching efficiency, and it is more meaningful to adopt adaptive local iteration frequency when resources are limited. Finally, experimental results in three real datasets demonstrate that AFPC can effectively improve cache efficiency up to 38.4<inline-formula><tex-math notation="LaTeX">\%</tex-math> <mml:math><mml:mo>%</mml:mo></mml:math><inline-graphic xlink:href="guo-ieq1-3201983.gif"/> </inline-formula> and 6.84<inline-formula><tex-math notation="LaTeX">\%</tex-math> <mml:math><mml:mo>%</mml:mo></mml:math><inline-graphic xlink:href="guo-ieq2-3201983.gif"/> </inline-formula>, and save resources up to 47.4<inline-formula><tex-math notation="LaTeX">\%</tex-math> <mml:math><mml:mo>%</mml:mo></mml:math><inline-graphic xlink:href="guo-ieq3-3201983.gif"/> </inline-formula> and 35.6<inline-formula><tex-math notation="LaTeX">\%</tex-math> <mml:math><mml:mo>%</mml:mo></mml:math><inline-graphic xlink:href="guo-ieq4-3201983.gif"/> </inline-formula>, respectively, compared with traditional multi-armed bandit (MAB)-based and FL-based algorithms.