云边端场景下基于多智能体深度强化学习的边缘缓存策略

王海艳; 常博; 骆健

doi:10.11959/j.issn.1000-436x.2025108

您当前的位置：

首页 >

文章列表页 >

云边端场景下基于多智能体深度强化学习的边缘缓存策略

学术论文 | 更新时间：2025-07-04

- 云边端场景下基于多智能体深度强化学习的边缘缓存策略
- Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios
- 通信学报 2025年46卷第6期页码：153-167
- 作者机构：
  
  1.南京邮电大学计算机学院，江苏南京 210023
  2.大数据安全与智能处理省高校重点实验室，江苏南京210023
- 作者简介：
  
  [ "王海艳（1974- ），女，江苏东台人，博士，南京邮电大学教授、博士生导师，主要研究方向为服务计算、可信计算、大数据应用与云计算技术、隐私保护技术等。" ]
  [ "常博（1999- ），男，河北张家口人，南京邮电大学硕士生，主要研究方向为边缘计算。" ]
  [ "骆健（1976- ），女，江西赣州人，南京邮电大学副教授，主要研究方向为服务计算、可信计算、服务推荐等。" ]
- 基金信息：
  
  国家自然科学基金资助项目(62272243)
- DOI：10.11959/j.issn.1000-436x.2025108
  中图分类号： TP393
- 收稿：2025-04-12，
  
  修回：2025-06-03，
  
  纸质出版：2025-06-25
- 稿件说明：
移动端阅览
王海艳,常博,骆健.云边端场景下基于多智能体深度强化学习的边缘缓存策略[J].通信学报,2025,46(06):153-167.

WANG Haiyan,CHANG Bo,LUO Jian.Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios[J].Journal on Communications,2025,46(06):153-167.
王海艳,常博,骆健.云边端场景下基于多智能体深度强化学习的边缘缓存策略[J].通信学报,2025,46(06):153-167. DOI： 10.11959/j.issn.1000-436x.2025108.

WANG Haiyan,CHANG Bo,LUO Jian.Edge caching strategy based on multi-agent deep reinforcement learning in cloud-edge-end scenarios[J].Journal on Communications,2025,46(06):153-167. DOI： 10.11959/j.issn.1000-436x.2025108.

摘要

云边端场景下，边缘缓存技术旨在通过促进边缘节点间的协同内容分发，减轻回程链路的流量负载并提升服务质量。考虑内容流行度的动态变化，提出了一种基于时间卷积网络的内容请求状态预测（TCNCRSP）模型。在此基础上，以最大化累积奖励为目标，提出了一种基于多智能体深度强化学习算法的边缘缓存策略，通过在云端利用长短期记忆（LSTM）网络对各边缘节点的状态数据进行降维处理，生成低维全局状态，减少状态共享所需的通信成本。实验结果显示，所提方法显著提升了缓存命中率和服务质量，同时降低了系统开销。

Abstract

In cloud-edge-end scenarios

edge caching technology aims to promote collaborative content distribution among edge nodes

thereby alleviating the traffic load on backhaul links and enhancing service quality. Considering the dynamic changes in content popularity

a time convolution network based content request state prediction (TCNCRSP) model for predicting content popularity was proposed. On this basis

aiming to maximize cumulative rewards

a multi-agent deep reinforcement learning algorithm based on edge caching strategy was proposed. This strategy was implemented using long short-term memory (LSTM) network in the cloud to perform dimensionality reduction on the state data of each edge node

thereby generating low-dimensional global states. This approach was used to reduce the communication costs required for state sharing. The experimental results show that the proposed methods significantly improve the cache hit rate and service quality

while also reducing system overhead.

关键词

Keywords

references

ABOLHASSANI B , TADROUS J , ERYILMAZ A . Optimal load-splitting and distributed-caching for dynamic content over the wireless edge [J ] . IEEE/ACM Transactions on Networking , 2023 , 31 ( 5 ): 2178 - 2190 .

CISCO U . Cisco annual Internet report (2018–2023) [R ] . 2018 .

SARAH A , NENCIONI G , KHAN M M I . Resource allocation in multi-access edge computing for 5G-and-beyond networks [J ] . Computer Networks , 2023 , 227 : 109720 .

QIU L , CAO G H . Popularity-aware caching increases the capacity of wireless networks [C ] // Proceedings of the IEEE INFOCOM 2017 - IEEE Conference on Computer Communications . Piscataway : IEEE Press , 2017 : 1 - 9 .

REISS-MIRZAEI M , GHOBAEI-ARANI M , ESMAEILI L . A review on the edge caching mechanisms in the mobile edge computing: a social-aware perspective [J ] . Internet of Things , 2023 , 22 : 100690 .

CHEN X D , TAN T X , CAO G H , et al . Context-aware and energy-aware video streaming on smartphones [J ] . IEEE Transactions on Mobile Computing , 2022 , 21 ( 3 ): 862 - 877 .

TRAN T X , HAJISAMI A , PANDEY P , et al . Collaborative mobile edge computing in 5G networks: new paradigms, scenarios, and challenges [J ] . IEEE Communications Magazine , 2017 , 55 ( 4 ): 54 - 61 .

GREGORI M , GÓMEZ-VILARDEBÓ J , MATAMOROS J , et al . Wireless content caching for small cell and D2D networks [J ] . IEEE Journal on Selected Areas in Communications , 2016 , 34 ( 5 ): 1222 - 1234 .

JIANG W , FENG G , QIN S . Optimal cooperative content caching and delivery policy for heterogeneous cellular networks [J ] . IEEE Transactions on Mobile Computing , 2017 , 16 ( 5 ): 1382 - 1393 .

KWAK J , KIM Y , LE L B , et al . Hybrid content caching in 5G wireless networks: cloud versus edge caching [J ] . IEEE Transactions on Wireless Communications , 2018 , 17 ( 5 ): 3030 - 3045 .

QIAN Y C , WANG R , WU J , et al . Reinforcement learning-based optimal computing and caching in mobile edge network [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 10 ): 2343 - 2355 .

LI Q , SUN Y H , WANG Q W , et al . A green DDPG reinforcement learning-based framework for content caching [C ] // Proceedings of the 2020 12th International Conference on Communication Software and Networks (ICCSN) . Piscataway : IEEE Press , 2020 : 223 - 227 .

WU P Y , LI J , SHI L , et al . Dynamic content update for wireless edge caching via deep reinforcement learning [J ] . IEEE Communications Letters , 2019 , 23 ( 10 ): 1773 - 1777 .

ZHEN Y , CHEN W B , ZHENG L B , et al . Multiagent cooperative caching policy in industrial Internet of Things [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 18 ): 16770 - 16779 .

HASSLINGER G , NTOUGIAS K , HASSLINGER F , et al . Scope and accuracy of analytic and approximate results for FIFO, clock-based and LRU caching performance [J ] . Future Internet , 2023 , 15 ( 3 ): 91 .

ZHONG C , GURSOY M C , VELIPASALAR S . Deep reinforcement learning-based edge caching in wireless networks [J ] . IEEE Transactions on Cognitive Communications and Networking , 2020 , 6 ( 1 ): 48 - 61 .

ZHANG T K , WANG Y , LIU Y W , et al . Cache-enabling UAV communications: network deployment and resource allocation [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 11 ): 7470 - 7483 .

MOAYERI M M , MOMENI H . An intelligent caching approach in mobile edge computing environment [C ] // Proceedings of the 2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP) . Piscataway : IEEE Press , 2024 : 1 - 5 .

YAO C , JIANG C K , LIU Z , et al . Optimal capacity allocation and caching strategy for multi-UAV collaborative edge caching [C ] // Proceedings of the 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM) . Piscataway : IEEE Press , 2021 : 905 - 910 .

LI L X , WANG M , XUE K Y , et al . Delay optimization in multi-UAV edge caching networks: a robust mean field game [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 1 ): 808 - 819 .

ZHANG T K , WANG Y , YI W Q , et al . Joint optimization of caching placement and trajectory for UAV-D2D networks [J ] . IEEE Transactions on Communications , 2022 , 70 ( 8 ): 5514 - 5527 .

TIAN A , FENG B H , ZHOU H C , et al . Efficient federated DRL-based cooperative caching for mobile edge networks [J ] . IEEE Transactions on Network and Service Management , 2023 , 20 ( 1 ): 246 - 260 .

YU D J , WU T , LIU C F , et al . Joint content caching and recommendation in opportunistic mobile networks through deep reinforcement learning and broad learning [J ] . IEEE Transactions on Services Computing , 2023 , 16 ( 4 ): 2727 - 2741 .

LI C L , ZHANG Y , GAO X , et al . Energy-latency tradeoffs for edge caching and dynamic service migration based on DQN in mobile edge computing [J ] . Journal of Parallel and Distributed Computing , 2022 , 166 : 15 - 31 .

CHEN S W , YAO Z , JIANG X F , et al . Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks [J ] . IEEE Transactions on Communications , 2021 , 69 ( 4 ): 2441 - 2456 .

LIU Y N , YANG C , CHEN X , et al . Joint hybrid caching and replacement scheme for UAV-assisted vehicular edge computing networks [J ] . IEEE Transactions on Intelligent Vehicles , 2024 , 9 ( 1 ): 866 - 878 .

SHEN J J , LIN Y , ZHANG Y J , et al . Content caching-assisted vehicular edge computing using multi-agent graph attention reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 2 ): 3509 - 3514 .

ZHOU X B , KE Z H , QIU T . Recommendation-driven multi-cell cooperative caching: a multi-agent reinforcement learning approach [J ] . IEEE Transactions on Mobile Computing , 2024 , 23 ( 5 ): 4764 - 4776 .

ZHENG Y , XIE X , MA W Y . GeoLife: a collaborative social networking service among user, location and trajectory [J ] . IEEE Data Engineering Bulletin , 2010 , 33 ( 2 ): 32 - 39 .

TRAVERSO S , AHMED M , GARETTO M , et al . Temporal locality in today’s content caching [J ] . ACM SIGCOMM Computer Communication Review , 2013 , 43 ( 5 ): 5 - 12 .

PODLIPNIG S , BÖSZÖRMENYI L . A survey of Web cache replacement strategies [J ] . ACM Computing Surveys , 2003 , 35 ( 4 ): 374 - 398 .

YANG Y J , LOU K H , WANG E , et al . Multi-agent reinforcement learning based file caching strategy in mobile edge computing [J ] . IEEE/ACM Transactions on Networking , 2023 , 31 ( 6 ): 3159 - 3174 .

WEI Z C , ZHAO Y , LYU Z W , et al . Cooperative caching algorithm for mobile edge networks based on multi-agent meta reinforcement learning [J ] . Computer Networks , 2024 , 242 : 110247 .

WU T , ZHOU P , WANG B H , et al . Joint traffic control and multi-channel reassignment for core backbone network in SDN-IoT: a multi-agent deep reinforcement learning approach [J ] . IEEE Transactions on Network Science and Engineering , 2021 , 8 ( 1 ): 231 - 245 .

LEKHARU A , SAMANTA A , SUR A , et al . Content-aware caching at the mobile edge network using federated learning [J ] . IEEE Transactions on Emerging Topics in Computational Intelligence , 2025 , 9 ( 2 ): 1166 - 1176 .

浏览量

503

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于微分博弈的容器网络安全防护方法研究

基于DDPG强化学习的多集群算力资源调度算法

基于访问权限控制博弈的容器安全隔离机制研究

智能反射表面辅助的车载边缘任务卸载和资源分配策略

数字孪生辅助下基于D3QN的车载网络协同卸载算法