空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略

刘亮; 荆腾祥; 段洁; 毛武平; 燕洪成; 马文杰

doi:10.11959/j.issn.1000-436x.2025014

您当前的位置：

首页 >

文章列表页 >

空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略

学术论文 | 更新时间：2025-02-13

- 空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略
- Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network
- 通信学报 2025年46卷第1期页码：93-107
- 作者机构：
  
  1.重庆邮电大学通信与信息工程学院，重庆 400065
  2.中国空间技术研究院，北京 100094
- 作者简介：
  
  [ "刘亮（1979- ），男，重庆人，博士，重庆邮电大学副教授，主要研究方向为空天地一体化网络、边缘智能计算等。" ]
  [ "荆腾祥（2000- ），男，河南焦作人，重庆邮电大学硕士生，主要研究方向为空天地一体化网络、机器学习等。" ]
  [ "段洁（1982- ），女，四川内江人，博士，重庆邮电大学副教授，主要研究方向为空天地一体化网络、网络路由、网络优化等。" ]
  [ "毛武平（1998- ），男，江西吉安人，重庆邮电大学硕士生，主要研究方向为空天地一体化网络、移动边缘计算等。" ]
  [ "燕洪成（1985- ），男，山东淄博人，中国空间技术研究院高级工程师，主要研究方向为航天器电子信息系统。" ]
  [ "马文杰（1973- ），男，山东高密人，中国空间技术研究院高级工程师，主要研究方向为卫星总体及星座总体技术。" ]
- 基金信息：
  
  国家自然科学基金资助项目(62171070;61701058);重庆市教委科学技术研究基金资助项目(KJQN202200615)
- DOI：10.11959/j.issn.1000-436x.2025014
  中图分类号： TN927
- 收稿：2024-11-06，
  
  修回：2025-01-08，
  
  纸质出版：2025-01-25
- 稿件说明：
移动端阅览
刘亮,荆腾祥,段洁等.空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略[J].通信学报,2025,46(01):93-107.

LIU Liang,JING Tengxiang,DUAN Jie,et al.Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network[J].Journal on Communications,2025,46(01):93-107.
刘亮,荆腾祥,段洁等.空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略[J].通信学报,2025,46(01):93-107. DOI： 10.11959/j.issn.1000-436x.2025014.

LIU Liang,JING Tengxiang,DUAN Jie,et al.Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network[J].Journal on Communications,2025,46(01):93-107. DOI： 10.11959/j.issn.1000-436x.2025014.

摘要

针对偏远地区网络覆盖范围有限的问题，将空天地一体化网络与移动边缘计算相结合，可以实现这些地区用户请求的低时延和高可靠传输，并能及时提供缓存服务。考虑到空天地一体化网络拓扑的动态变化和内容流行度不断更新的特点，首先提出了一种空天地一体化边缘协作缓存的网络架构。然后，将边缘服务器的缓存替换问题建模为马尔可夫决策过程。最后，提出了一种联邦离散柔性演员评论家（FDSAC）算法，其核心思想是将加权注意力机制融入联邦学习框架中，并将双向长短期记忆网络集成到DSAC模型。以重构后的奖励函数为优化目标，通过最大化长期负奖励的期望来学习最优的缓存替换策略。仿真结果表明，与其他算法相比，所提算法可以在保护用户隐私的前提下，将用户请求的缓存命中率提高18%，内容的访问时延降低25%。

Abstract

To address the problem of limited network coverage in remote areas

combining space-air-ground integrated network with mobile edge computing could provide low-latency and high-reliability transmissions for user requests in these areas

as well as timely caching services. Considering the dynamic change of the topology of the space-air-ground integrated network and the content popularity being constantly updated

a network architecture of space-air-ground integrated edge collaborative caching was proposed first. Then

the cache replacement problem for edge servers was modeled as a Markov decision process. Finally

a federated discrete soft actor-critic (FDSAC) algorithm was proposed

with the core idea of integrating a weighted attention mechanism into the federated learning framework and incorporating a bidirectional long short-term memory network into the DSAC model. With the reconfigured reward function as the optimization objective

the optimal cache replacement policy was learned by maximizing the expectation of negative long-term rewards. Simulation results show that compared with other algorithm

the proposed algorithm can improve the cache hit rate of user requests by 18% and reduce the access latency of content by 25% while protecting user privacy.

关键词

Keywords

references

NAWAZ F , MOHSIN A , JANJUA N K . Service description languages in cloud computing: state-of-the-art and research issues [J ] . Service Oriented Computing and Applications , 2019 , 13 ( 2 ): 109 - 125 .

黄永明 , 郑冲 , 张征明 , 等 . 大规模无线通信网络移动边缘计算和缓存研究 [J ] . 通信学报 , 2021 , 42 ( 4 ): 44 - 61 .

HUANG Y M , ZHENG C , ZHANG Z M , et al . Research on mobile edge computing and caching in massive wireless communication network [J ] . Journal on Communications , 2021 , 42 ( 4 ): 44 - 61 .

MASOOD A , TUAN D Q , LAKEW D S , et al . A review on AI-enabled content caching in vehicular edge caching and networks [C ] // Proceedings of the 2023 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) . Piscataway : IEEE Press , 2023 : 713 - 717 .

张海君 , 陈安琪 , 李亚博 , 等 . 6G移动网络关键技术 [J ] . 通信学报 , 2022 , 43 ( 7 ): 189 - 202 .

ZHANG H J , CHEN A Q , LI Y B , et al . Key technologies of 6G mobile network [J ] . Journal on Communications , 2022 , 43 ( 7 ): 189 - 202 .

ZHOU J W , CHEN F F , HE Q , et al . Data caching optimization with fairness in mobile edge computing [J ] . IEEE Transactions on Services Computing , 2023 , 16 ( 3 ): 1750 - 1762 .

ZHANG S B , LIU J J . Optimal probabilistic caching in heterogeneous IoT networks [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 4 ): 3404 - 3414 .

YOO S , JEONG S , KIM J , et al . Cache-assisted mobile-edge computing over space-air-ground integrated networks for extended reality applications [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 10 ): 18306 - 18319 .

CHEN S W , YAO Z , JIANG X F , et al . Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks [J ] . IEEE Transactions on Communications , 2021 , 69 ( 4 ): 2441 - 2456 .

WU H D , NASEHZADEH A , WANG P . A deep reinforcement learning-based caching strategy for IoT networks with transient data [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 12 ): 13310 - 13319 .

LIANG Q F , LIU Y , TANG W J . Joint cache placement and content scheduling in integrated LEO satellite-terrestrial networks [C ] // Proceedings of the 2022 IEEE/CIC International Conference on Communications in China (ICCC) . Piscataway : IEEE Press , 2022 : 642 - 648 .

ZHANG H J , XU J L , LIU X N , et al . Joint optimization of caching placement and power allocation in virtualized satellite-terrestrial network [J ] . IEEE Transactions on Wireless Communications , 2023 , 22 ( 11 ): 7932 - 7943 .

JIANG C X , LI Z . Decreasing big data application latency in satellite link by caching and peer selection [J ] . IEEE Transactions on Network Science and Engineering , 2020 , 7 ( 4 ): 2555 - 2565 .

AN K , LI Y S , YAN X J , et al . On the performance of cache-enabled hybrid satellite-terrestrial relay networks [J ] . IEEE Wireless Communications Letters , 2019 , 8 ( 5 ): 1506 - 1509 .

ZHU X M , JIANG C X , KUANG L L , et al . Cooperative multilayer edge caching in integrated satellite-terrestrial networks [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 5 ): 2924 - 2937 .

JI J Q , ZHU K , NIYATO D , et al . Probabilistic cache placement in UAV-assisted networks with D2D connections: performance analysis and trajectory optimization [J ] . IEEE Transactions on Communications , 2020 , 68 ( 10 ): 6331 - 6345 .

ZHANG M Z , EI-HAJJAR M , NG S X . Intelligent caching in UAV-aided networks [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 1 ): 739 - 752 .

BERA A , MISRA S , CHATTERJEE C . QoE analysis in cache-enabled multi-UAV networks [J ] . IEEE Transactions on Vehicular Technology , 2020 , 69 ( 6 ): 6680 - 6687 .

ANOKYE S , AYEPAH-MENSAH D , SEID A M , et al . Deep reinforcement learning-based mobility-aware UAV content caching and placement in mobile edge networks [J ] . IEEE Systems Journal , 2022 , 16 ( 1 ): 275 - 286 .

WU Y , LI J D , LIU J Y , et al . Enabling efficient caching in high mobility UAV communications network under limited backhaul [J ] . China Communications , 2022 , 19 ( 10 ): 207 - 219 .

GU S S , WANG Y , WANG N N , et al . Intelligent optimization of availability and communication cost in satellite-UAV mobile edge caching system with fault-tolerant codes [J ] . IEEE Transactions on Cognitive Communications and Networking , 2020 , 6 ( 4 ): 1230 - 1241 .

BAO J R , PENG X Y , LIU C , et al . Multilayered decentralized coded caching with nonuniform popularity and multilevel cache capacity in space-air-ground integrated networks [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 8 ): 13913 - 13926 .

DU W R , WEI Z X , LI C , et al . On a deep reinforcement learning-based content caching strategy in 6G space-air-ground integrated networks [C ] // Proceedings of the 2023 15th International Conference on Communication Software and Networks (ICCSN) . Piscataway : IEEE Press , 2023 : 204 - 208 .

GU S S , SUN X Y , YANG Z H , et al . Energy-aware coded caching strategy design with resource optimization for satellite-UAV-vehicle-integrated networks [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 8 ): 5799 - 5811 .

YIN F F , LIU Q H , LIU D P , et al . Joint coded caching and resource allocation for multimedia service in space-air-ground integrated networks [J ] . IEEE Transactions on Communications , 2024 , 72 ( 11 ): 6839 - 6853 .

TANG F X , HOFNER H , KATO N , et al . A deep reinforcement learning-based dynamic traffic offloading in space-air-ground integrated networks (SAGIN) [J ] . IEEE Journal on Selected Areas in Communications , 2022 , 40 ( 1 ): 276 - 289 .

DENG R Q , DI B Y , CHEN S Z , et al . Ultra-dense LEO satellite offloading for terrestrial networks: how much to pay the satellite operator? [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 10 ): 6240 - 6254 .

GAN Y H , HE Y J . Trajectory optimization and computing offloading strategy in UAV-assisted MEC system [C ] // Proceedings of the 2021 Computing, Communications and IoT Applications (ComComAp) . Piscataway : IEEE Press , 2021 : 132 - 137 .

WANG X F , WANG C Y , LI X H , et al . Federated deep reinforcement learning for Internet of things with decentralized cooperative edge caching [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 10 ): 9441 - 9455 .

KHAMARI S , RACHEDI A , AHMED T , et al . Adaptive deep reinforcement learning approach for service migration in MEC-enabled vehicular networks [C ] // Proceedings of the 2023 IEEE Symposium on Computers and Communications (ISCC) . Piscataway : IEEE Press , 2023 : 1075 - 1079 .

LYU Z W , WANG Y , LIU M , et al . Service-driven resource management in vehicular networks based on deep reinforcement learning [C ] // Proceedings of the 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications . Piscataway : IEEE Press , 2020 : 1 - 6 .

浏览量

1530

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

ISAC网络中基于特征信息的联邦学习动态压缩方法

DPBR-Adapt：具有层级自适应差分隐私的联邦学习防御方案

面向协作频谱感知的个性化差分隐私联邦学习方法

基于多智能体深度强化学习的智能网联汽车服务迁移优化方法

面向动态算力节点的联邦学习差分隐私重校准方法