Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network

LIU Liang; JING Tengxiang; DUAN Jie; MAO Wuping; YAN Hongcheng; MA Wenjie

doi:10.11959/j.issn.1000-436x.2025014

您当前的位置：

首页 >

文章列表页 >

Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network

Papers | 更新时间：2025-02-13

- Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network
- Journal on Communications Vol. 46, Issue 1, Pages: 93-107(2025)
- 作者机构：
  
  1.重庆邮电大学通信与信息工程学院，重庆 400065
  2.中国空间技术研究院，北京 100094
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62171070;61701058);The Science and Technology Research Program of Chongqing Municipal Education Commission(KJQN202200615)
- DOI：10.11959/j.issn.1000-436x.2025014
  CLC： TN927
- Received：06 November 2024，
  
  Revised：2025-01-08，
  
  Published：25 January 2025
- 稿件说明：
移动端阅览
刘亮,荆腾祥,段洁等.空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略[J].通信学报,2025,46(01):93-107.

LIU Liang,JING Tengxiang,DUAN Jie,et al.Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network[J].Journal on Communications,2025,46(01):93-107.
刘亮,荆腾祥,段洁等.空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略[J].通信学报,2025,46(01):93-107. DOI： 10.11959/j.issn.1000-436x.2025014.

LIU Liang,JING Tengxiang,DUAN Jie,et al.Federated deep reinforcement learning-based edge collaborative caching strategy in space-air-ground integrated network[J].Journal on Communications,2025,46(01):93-107. DOI： 10.11959/j.issn.1000-436x.2025014.

摘要

针对偏远地区网络覆盖范围有限的问题，将空天地一体化网络与移动边缘计算相结合，可以实现这些地区用户请求的低时延和高可靠传输，并能及时提供缓存服务。考虑到空天地一体化网络拓扑的动态变化和内容流行度不断更新的特点，首先提出了一种空天地一体化边缘协作缓存的网络架构。然后，将边缘服务器的缓存替换问题建模为马尔可夫决策过程。最后，提出了一种联邦离散柔性演员评论家（FDSAC）算法，其核心思想是将加权注意力机制融入联邦学习框架中，并将双向长短期记忆网络集成到DSAC模型。以重构后的奖励函数为优化目标，通过最大化长期负奖励的期望来学习最优的缓存替换策略。仿真结果表明，与其他算法相比，所提算法可以在保护用户隐私的前提下，将用户请求的缓存命中率提高18%，内容的访问时延降低25%。

Abstract

To address the problem of limited network coverage in remote areas

combining space-air-ground integrated network with mobile edge computing could provide low-latency and high-reliability transmissions for user requests in these areas

as well as timely caching services. Considering the dynamic change of the topology of the space-air-ground integrated network and the content popularity being constantly updated

a network architecture of space-air-ground integrated edge collaborative caching was proposed first. Then

the cache replacement problem for edge servers was modeled as a Markov decision process. Finally

a federated discrete soft actor-critic (FDSAC) algorithm was proposed

with the core idea of integrating a weighted attention mechanism into the federated learning framework and incorporating a bidirectional long short-term memory network into the DSAC model. With the reconfigured reward function as the optimization objective

the optimal cache replacement policy was learned by maximizing the expectation of negative long-term rewards. Simulation results show that compared with other algorithm

the proposed algorithm can improve the cache hit rate of user requests by 18% and reduce the access latency of content by 25% while protecting user privacy.

关键词

Keywords

references

NAWAZ F , MOHSIN A , JANJUA N K . Service description languages in cloud computing: state-of-the-art and research issues [J ] . Service Oriented Computing and Applications , 2019 , 13 ( 2 ): 109 - 125 .

黄永明 , 郑冲 , 张征明 , 等 . 大规模无线通信网络移动边缘计算和缓存研究 [J ] . 通信学报 , 2021 , 42 ( 4 ): 44 - 61 .

HUANG Y M , ZHENG C , ZHANG Z M , et al . Research on mobile edge computing and caching in massive wireless communication network [J ] . Journal on Communications , 2021 , 42 ( 4 ): 44 - 61 .

MASOOD A , TUAN D Q , LAKEW D S , et al . A review on AI-enabled content caching in vehicular edge caching and networks [C ] // Proceedings of the 2023 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) . Piscataway : IEEE Press , 2023 : 713 - 717 .

张海君 , 陈安琪 , 李亚博 , 等 . 6G移动网络关键技术 [J ] . 通信学报 , 2022 , 43 ( 7 ): 189 - 202 .

ZHANG H J , CHEN A Q , LI Y B , et al . Key technologies of 6G mobile network [J ] . Journal on Communications , 2022 , 43 ( 7 ): 189 - 202 .

ZHOU J W , CHEN F F , HE Q , et al . Data caching optimization with fairness in mobile edge computing [J ] . IEEE Transactions on Services Computing , 2023 , 16 ( 3 ): 1750 - 1762 .

ZHANG S B , LIU J J . Optimal probabilistic caching in heterogeneous IoT networks [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 4 ): 3404 - 3414 .

YOO S , JEONG S , KIM J , et al . Cache-assisted mobile-edge computing over space-air-ground integrated networks for extended reality applications [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 10 ): 18306 - 18319 .

CHEN S W , YAO Z , JIANG X F , et al . Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks [J ] . IEEE Transactions on Communications , 2021 , 69 ( 4 ): 2441 - 2456 .

WU H D , NASEHZADEH A , WANG P . A deep reinforcement learning-based caching strategy for IoT networks with transient data [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 12 ): 13310 - 13319 .

LIANG Q F , LIU Y , TANG W J . Joint cache placement and content scheduling in integrated LEO satellite-terrestrial networks [C ] // Proceedings of the 2022 IEEE/CIC International Conference on Communications in China (ICCC) . Piscataway : IEEE Press , 2022 : 642 - 648 .

ZHANG H J , XU J L , LIU X N , et al . Joint optimization of caching placement and power allocation in virtualized satellite-terrestrial network [J ] . IEEE Transactions on Wireless Communications , 2023 , 22 ( 11 ): 7932 - 7943 .

JIANG C X , LI Z . Decreasing big data application latency in satellite link by caching and peer selection [J ] . IEEE Transactions on Network Science and Engineering , 2020 , 7 ( 4 ): 2555 - 2565 .

AN K , LI Y S , YAN X J , et al . On the performance of cache-enabled hybrid satellite-terrestrial relay networks [J ] . IEEE Wireless Communications Letters , 2019 , 8 ( 5 ): 1506 - 1509 .

ZHU X M , JIANG C X , KUANG L L , et al . Cooperative multilayer edge caching in integrated satellite-terrestrial networks [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 5 ): 2924 - 2937 .

JI J Q , ZHU K , NIYATO D , et al . Probabilistic cache placement in UAV-assisted networks with D2D connections: performance analysis and trajectory optimization [J ] . IEEE Transactions on Communications , 2020 , 68 ( 10 ): 6331 - 6345 .

ZHANG M Z , EI-HAJJAR M , NG S X . Intelligent caching in UAV-aided networks [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 1 ): 739 - 752 .

BERA A , MISRA S , CHATTERJEE C . QoE analysis in cache-enabled multi-UAV networks [J ] . IEEE Transactions on Vehicular Technology , 2020 , 69 ( 6 ): 6680 - 6687 .

ANOKYE S , AYEPAH-MENSAH D , SEID A M , et al . Deep reinforcement learning-based mobility-aware UAV content caching and placement in mobile edge networks [J ] . IEEE Systems Journal , 2022 , 16 ( 1 ): 275 - 286 .

WU Y , LI J D , LIU J Y , et al . Enabling efficient caching in high mobility UAV communications network under limited backhaul [J ] . China Communications , 2022 , 19 ( 10 ): 207 - 219 .

GU S S , WANG Y , WANG N N , et al . Intelligent optimization of availability and communication cost in satellite-UAV mobile edge caching system with fault-tolerant codes [J ] . IEEE Transactions on Cognitive Communications and Networking , 2020 , 6 ( 4 ): 1230 - 1241 .

BAO J R , PENG X Y , LIU C , et al . Multilayered decentralized coded caching with nonuniform popularity and multilevel cache capacity in space-air-ground integrated networks [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 8 ): 13913 - 13926 .

DU W R , WEI Z X , LI C , et al . On a deep reinforcement learning-based content caching strategy in 6G space-air-ground integrated networks [C ] // Proceedings of the 2023 15th International Conference on Communication Software and Networks (ICCSN) . Piscataway : IEEE Press , 2023 : 204 - 208 .

GU S S , SUN X Y , YANG Z H , et al . Energy-aware coded caching strategy design with resource optimization for satellite-UAV-vehicle-integrated networks [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 8 ): 5799 - 5811 .

YIN F F , LIU Q H , LIU D P , et al . Joint coded caching and resource allocation for multimedia service in space-air-ground integrated networks [J ] . IEEE Transactions on Communications , 2024 , 72 ( 11 ): 6839 - 6853 .

TANG F X , HOFNER H , KATO N , et al . A deep reinforcement learning-based dynamic traffic offloading in space-air-ground integrated networks (SAGIN) [J ] . IEEE Journal on Selected Areas in Communications , 2022 , 40 ( 1 ): 276 - 289 .

DENG R Q , DI B Y , CHEN S Z , et al . Ultra-dense LEO satellite offloading for terrestrial networks: how much to pay the satellite operator? [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 10 ): 6240 - 6254 .

GAN Y H , HE Y J . Trajectory optimization and computing offloading strategy in UAV-assisted MEC system [C ] // Proceedings of the 2021 Computing, Communications and IoT Applications (ComComAp) . Piscataway : IEEE Press , 2021 : 132 - 137 .

WANG X F , WANG C Y , LI X H , et al . Federated deep reinforcement learning for Internet of things with decentralized cooperative edge caching [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 10 ): 9441 - 9455 .

KHAMARI S , RACHEDI A , AHMED T , et al . Adaptive deep reinforcement learning approach for service migration in MEC-enabled vehicular networks [C ] // Proceedings of the 2023 IEEE Symposium on Computers and Communications (ISCC) . Piscataway : IEEE Press , 2023 : 1075 - 1079 .

LYU Z W , WANG Y , LIU M , et al . Service-driven resource management in vehicular networks based on deep reinforcement learning [C ] // Proceedings of the 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications . Piscataway : IEEE Press , 2020 : 1 - 6 .

Views

1530

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Dynamic compression method for federated learning based on feature information in ISAC networks

DPBR-Adapt: a hierarchically adaptive differential privacy defence scheme for federated learning

Personalized differential privacy federated learning method for collaborative spectrum sensing

Service migration optimization method for intelligent connected vehicles based on multi-agent deep reinforcement learning

Federated learning with differential privacy recalibration for dynamic computing nodes

Related Author

Deng Bingguang

Peng Jiayin

Hu Ronglei

Bai Chenyang

Wei Zhanzhen

Han Yanyan

Duan Xiaoyi

Zhang Hao

Related Institution

Department of Electronic and Communication Engineering, Beijing Electronic Science and Technology Institute

School of Information Engineering, Minzu University of China

School of Automation, Guangdong University of Technology

School of Cyberspace Security, Beijing Jiaotong University

Nanyang Technological University

AI问答

⁰