Deep Reinforcement Learning for Adaptive Caching in Hierarchical Content Delivery Networks

• Published in: IEEE transactions on cognitive communications and networking Vol. 5; no. 4; pp. 1024 - 1033
• Main Authors: Sadeghi, Alireza, Wang, Gang, Giannakis, Georgios B.
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.12.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Caching; Cellular communication; Cellular networks; Content delivery networks; Content distribution networks; deep Q-network; deep RL; End users; function approximation; Infrastructure; Machine learning; Markov processes; Network topology; Next generation networking; next-generation networks; Nodes; Peak periods; Reinforcement learning; Storage capacity; Vehicle dynamics
• ISSN: 2332-7731; 2332-7731
• DOI: 10.1109/TCCN.2019.2936193

Caching is envisioned to play a critical role in next-generation content delivery infrastructure, cellular networks, and Internet architectures. By smartly storing the most popular contents at the storage-enabled network entities during off-peak demand instances, caching can benefit both network infrastructure as well as end users, during on-peak periods. In this context, distributing the limited storage capacity across network entities calls for decentralized caching schemes. Many practical caching systems involve a parent caching node connected to multiple leaf nodes to serve user file requests. To model the two-way interactive influence between caching decisions at the parent and leaf nodes, a reinforcement learning (RL) framework is put forth. To handle the large continuous state space, a scalable deep RL approach is pursued. The novel approach relies on a hyper-deep Q-network to learn the Q-function, and thus the optimal caching policy, in an online fashion. Reinforcing the parent node with ability to learn-and-adapt to unknown policies of leaf nodes as well as spatio-temporal dynamic evolution of file requests, results in remarkable caching performance, as corroborated through numerical tests.