Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios

WANG Haiyan; CHANG Bo; LUO Jian

doi:10.11959/j.issn.1000-436x.2025108

您当前的位置：

首页 >

文章列表页 >

Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios

Papers | 更新时间：2025-07-04

- Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios
- Journal on Communications Vol. 46, Issue 6, Pages: 153-167(2025)
- 作者机构：
  
  1.南京邮电大学计算机学院，江苏南京 210023
  2.大数据安全与智能处理省高校重点实验室，江苏南京210023
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62272243)
- DOI：10.11959/j.issn.1000-436x.2025108
  CLC： TP393
- Received：12 April 2025，
  
  Revised：2025-06-03，
  
  Published：25 June 2025
- 稿件说明：
移动端阅览
王海艳,常博,骆健.云边端场景下基于多智能体深度强化学习的边缘缓存策略[J].通信学报,2025,46(06):153-167.

WANG Haiyan,CHANG Bo,LUO Jian.Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios[J].Journal on Communications,2025,46(06):153-167.
王海艳,常博,骆健.云边端场景下基于多智能体深度强化学习的边缘缓存策略[J].通信学报,2025,46(06):153-167. DOI： 10.11959/j.issn.1000-436x.2025108.

WANG Haiyan,CHANG Bo,LUO Jian.Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios[J].Journal on Communications,2025,46(06):153-167. DOI： 10.11959/j.issn.1000-436x.2025108.

摘要

云边端场景下，边缘缓存技术旨在通过促进边缘节点间的协同内容分发，减轻回程链路的流量负载并提升服务质量。考虑内容流行度的动态变化，提出了一种基于时间卷积网络的内容请求状态预测（TCNCRSP）模型。在此基础上，以最大化累积奖励为目标，提出了一种基于多智能体深度强化学习算法的边缘缓存策略，通过在云端利用长短期记忆（LSTM）网络对各边缘节点的状态数据进行降维处理，生成低维全局状态，减少状态共享所需的通信成本。实验结果显示，所提方法显著提升了缓存命中率和服务质量，同时降低了系统开销。

Abstract

In cloud-edge-end scenarios

edge caching technology aims to promote collaborative content distribution among edge nodes

thereby alleviating the traffic load on backhaul links and enhancing service quality. Considering the dynamic changes in content popularity

a time convolution network based content request state prediction (TCNCRSP) model for predicting content popularity was proposed. On this basis

aiming to maximize cumulative rewards

a multi-agent deep reinforcement learning algorithm based on edge caching strategy was proposed. This strategy was implemented using long short-term memory (LSTM) network in the cloud to perform dimensionality reduction on the state data of each edge node

thereby generating low-dimensional global states. This approach was used to reduce the communication costs required for state sharing. The experimental results show that the proposed methods significantly improve the cache hit rate and service quality

while also reducing system overhead.

关键词

Keywords

references

ABOLHASSANI B , TADROUS J , ERYILMAZ A . Optimal load-splitting and distributed-caching for dynamic content over the wireless edge [J ] . IEEE/ACM Transactions on Networking , 2023 , 31 ( 5 ): 2178 - 2190 .

CISCO U . Cisco annual Internet report (2018–2023) [R ] . 2018 .

SARAH A , NENCIONI G , KHAN M M I . Resource allocation in multi-access edge computing for 5G-and-beyond networks [J ] . Computer Networks , 2023 , 227 : 109720 .

QIU L , CAO G H . Popularity-aware caching increases the capacity of wireless networks [C ] // Proceedings of the IEEE INFOCOM 2017 - IEEE Conference on Computer Communications . Piscataway : IEEE Press , 2017 : 1 - 9 .

REISS-MIRZAEI M , GHOBAEI-ARANI M , ESMAEILI L . A review on the edge caching mechanisms in the mobile edge computing: a social-aware perspective [J ] . Internet of Things , 2023 , 22 : 100690 .

CHEN X D , TAN T X , CAO G H , et al . Context-aware and energy-aware video streaming on smartphones [J ] . IEEE Transactions on Mobile Computing , 2022 , 21 ( 3 ): 862 - 877 .

TRAN T X , HAJISAMI A , PANDEY P , et al . Collaborative mobile edge computing in 5G networks: new paradigms, scenarios, and challenges [J ] . IEEE Communications Magazine , 2017 , 55 ( 4 ): 54 - 61 .

GREGORI M , GÓMEZ-VILARDEBÓ J , MATAMOROS J , et al . Wireless content caching for small cell and D2D networks [J ] . IEEE Journal on Selected Areas in Communications , 2016 , 34 ( 5 ): 1222 - 1234 .

JIANG W , FENG G , QIN S . Optimal cooperative content caching and delivery policy for heterogeneous cellular networks [J ] . IEEE Transactions on Mobile Computing , 2017 , 16 ( 5 ): 1382 - 1393 .

KWAK J , KIM Y , LE L B , et al . Hybrid content caching in 5G wireless networks: cloud versus edge caching [J ] . IEEE Transactions on Wireless Communications , 2018 , 17 ( 5 ): 3030 - 3045 .

QIAN Y C , WANG R , WU J , et al . Reinforcement learning-based optimal computing and caching in mobile edge network [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 10 ): 2343 - 2355 .

LI Q , SUN Y H , WANG Q W , et al . A green DDPG reinforcement learning-based framework for content caching [C ] // Proceedings of the 2020 12th International Conference on Communication Software and Networks (ICCSN) . Piscataway : IEEE Press , 2020 : 223 - 227 .

WU P Y , LI J , SHI L , et al . Dynamic content update for wireless edge caching via deep reinforcement learning [J ] . IEEE Communications Letters , 2019 , 23 ( 10 ): 1773 - 1777 .

ZHEN Y , CHEN W B , ZHENG L B , et al . Multiagent cooperative caching policy in industrial Internet of Things [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 18 ): 16770 - 16779 .

HASSLINGER G , NTOUGIAS K , HASSLINGER F , et al . Scope and accuracy of analytic and approximate results for FIFO, clock-based and LRU caching performance [J ] . Future Internet , 2023 , 15 ( 3 ): 91 .

ZHONG C , GURSOY M C , VELIPASALAR S . Deep reinforcement learning-based edge caching in wireless networks [J ] . IEEE Transactions on Cognitive Communications and Networking , 2020 , 6 ( 1 ): 48 - 61 .

ZHANG T K , WANG Y , LIU Y W , et al . Cache-enabling UAV communications: network deployment and resource allocation [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 11 ): 7470 - 7483 .

MOAYERI M M , MOMENI H . An intelligent caching approach in mobile edge computing environment [C ] // Proceedings of the 2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP) . Piscataway : IEEE Press , 2024 : 1 - 5 .

YAO C , JIANG C K , LIU Z , et al . Optimal capacity allocation and caching strategy for multi-UAV collaborative edge caching [C ] // Proceedings of the 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM) . Piscataway : IEEE Press , 2021 : 905 - 910 .

LI L X , WANG M , XUE K Y , et al . Delay optimization in multi-UAV edge caching networks: a robust mean field game [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 1 ): 808 - 819 .

ZHANG T K , WANG Y , YI W Q , et al . Joint optimization of caching placement and trajectory for UAV-D2D networks [J ] . IEEE Transactions on Communications , 2022 , 70 ( 8 ): 5514 - 5527 .

TIAN A , FENG B H , ZHOU H C , et al . Efficient federated DRL-based cooperative caching for mobile edge networks [J ] . IEEE Transactions on Network and Service Management , 2023 , 20 ( 1 ): 246 - 260 .

YU D J , WU T , LIU C F , et al . Joint content caching and recommendation in opportunistic mobile networks through deep reinforcement learning and broad learning [J ] . IEEE Transactions on Services Computing , 2023 , 16 ( 4 ): 2727 - 2741 .

LI C L , ZHANG Y , GAO X , et al . Energy-latency tradeoffs for edge caching and dynamic service migration based on DQN in mobile edge computing [J ] . Journal of Parallel and Distributed Computing , 2022 , 166 : 15 - 31 .

CHEN S W , YAO Z , JIANG X F , et al . Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks [J ] . IEEE Transactions on Communications , 2021 , 69 ( 4 ): 2441 - 2456 .

LIU Y N , YANG C , CHEN X , et al . Joint hybrid caching and replacement scheme for UAV-assisted vehicular edge computing networks [J ] . IEEE Transactions on Intelligent Vehicles , 2024 , 9 ( 1 ): 866 - 878 .

SHEN J J , LIN Y , ZHANG Y J , et al . Content caching-assisted vehicular edge computing using multi-agent graph attention reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 2 ): 3509 - 3514 .

ZHOU X B , KE Z H , QIU T . Recommendation-driven multi-cell cooperative caching: a multi-agent reinforcement learning approach [J ] . IEEE Transactions on Mobile Computing , 2024 , 23 ( 5 ): 4764 - 4776 .

ZHENG Y , XIE X , MA W Y . GeoLife: a collaborative social networking service among user, location and trajectory [J ] . IEEE Data Engineering Bulletin , 2010 , 33 ( 2 ): 32 - 39 .

TRAVERSO S , AHMED M , GARETTO M , et al . Temporal locality in today’s content caching [J ] . ACM SIGCOMM Computer Communication Review , 2013 , 43 ( 5 ): 5 - 12 .

PODLIPNIG S , BÖSZÖRMENYI L . A survey of Web cache replacement strategies [J ] . ACM Computing Surveys , 2003 , 35 ( 4 ): 374 - 398 .

YANG Y J , LOU K H , WANG E , et al . Multi-agent reinforcement learning based file caching strategy in mobile edge computing [J ] . IEEE/ACM Transactions on Networking , 2023 , 31 ( 6 ): 3159 - 3174 .

WEI Z C , ZHAO Y , LYU Z W , et al . Cooperative caching algorithm for mobile edge networks based on multi-agent meta reinforcement learning [J ] . Computer Networks , 2024 , 242 : 110247 .

WU T , ZHOU P , WANG B H , et al . Joint traffic control and multi-channel reassignment for core backbone network in SDN-IoT: a multi-agent deep reinforcement learning approach [J ] . IEEE Transactions on Network Science and Engineering , 2021 , 8 ( 1 ): 231 - 245 .

LEKHARU A , SAMANTA A , SUR A , et al . Content-aware caching at the mobile edge network using federated learning [J ] . IEEE Transactions on Emerging Topics in Computational Intelligence , 2025 , 9 ( 2 ): 1166 - 1176 .

Views

1098

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

GAT-based decision mechanism for decentralized joint routing and spectrum access

Research on container network security protection method based on differential game

Multi-cluster computing power resource scheduling algorithm based on DDPG reinforcement learning

Research on container security isolation mechanism based on access permission control game

Task offloading and resource allocation strategy for vehicular edge computing assisted by intelligent reflecting surfaces

Related Author

Zhou Zibo

Ren Baoquan

Zhong Xudong

Liu Qi

Qin Zhen

Wen Jiakun

Xi Long

Cao Yuan

Related Institution

Systems Engineering Institute, Academy of Military Science

Air Force Early Warning Academy

School of Automation and Intelligence, Beijing Jiaotong University

CRSC Research & Design Institute Group Co., Ltd.

China Nuclear Power Engineering Co., Ltd.

AI脑图

AI问答

⁰