Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid

Feng YAN; Xiaowei LIN; Zhenghao LI; Xia XU; Weiwei XIA; Lianfeng SHEN

doi:10.11959/j.issn.1000-436x.2023179

您当前的位置：

首页 >

文章列表页 >

Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid

Papers | 更新时间：2024-06-06

- Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid
- Journal on Communications Vol. 44, Issue 9, Pages: 12-24(2023)
- 作者机构：
  
  1. 东南大学移动通信全国重点实验室，江苏南京 210096
  2. 东南大学软件学院，江苏南京 211100
  3. 国网山东省电力公司信息通信公司，山东济南 250001
  4. 国网山东省电力公司济南供电公司，山东济南 250012
- 作者简介：
- 基金信息：
  
  The Science and Technology Project of State Grid Corporation of China(520601220022)
- DOI：10.11959/j.issn.1000-436x.2023179
  CLC： TN92
- Online First：2023-09，
  
  Published：25 September 2023
- 稿件说明：
移动端阅览
Feng YAN, Xiaowei LIN, Zhenghao LI, et al. Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid[J]. Journal on Communications, 2023, 44(9): 12-24.
DOI：

Feng YAN, Xiaowei LIN, Zhenghao LI, et al. Spectrum allocation algorithm based on multi-agent reinforcement learning in smart grid[J]. Journal on Communications, 2023, 44(9): 12-24. DOI： 10.11959/j.issn.1000-436x.2023179.

摘要

针对智能电网中利用5G网络承载多样化电力终端的业务需求，提出了一种基于多智能体强化学习的频谱分配算法。首先，基于智能电网中部署的集成接入回程系统，考虑智能电网中轻量化和非轻量化终端业务的不同通信需求，将频谱分配问题建模为最大化系统总能效的非凸混合整数规划。其次，将前述问题构建为一个部分可观测的马尔可夫决策过程并转换为完全协作的多智能体问题，进而提出了一种集中训练分布执行框架下基于多智能体近端策略优化的频谱分配算法。最后，通过仿真验证了所提算法的性能。仿真结果表明，所提算法具有更快的收敛速度，通过有效减少层内与层间干扰、平衡接入与回程链路速率，可以将系统总速率提高25.2%。

Abstract

In view of the fact that 5G networks are used to meet the service requirements of various power terminals in smart grid

a spectrum allocation algorithm based on multi-agent reinforcement learning was proposed.Firstly

for the integrated access backhaul system deployed in smart grid

considering the different communication requirements of services in lightweight and non-lightweight terminal

the spectrum allocation problem was formulated as a non-convex mixed-integer programming aiming to maximize the overall energy efficiency.Secondly

the above problem was modeled as a partially observable Markov decision process and transformed into a fully cooperative multi-agent problem

then a spectrum allocation algorithm was proposed which was based on multi-agent proximal policy optimization under the framework of centralized training and distributed execution.Finally

the performance of the proposed algorithm was verified by simulation.The results show that the proposed algorithm has a faster convergence speed and can increase the overall transmission rate by 25.2% through effectively reducing intra-layer and inter-layer interference and balancing the access and backhaul link rates.

关键词

Keywords

references

DILEEP G . A survey on smart grid technologies and applications [J ] . Renewable Energy , 2020 , 146 : 2589 - 2625 .

CHI Y Y , ZHANG Y , LIU Y , et al . Deep reinforcement learning based edge computing network aided resource allocation algorithm for smart grid [J ] . IEEE Access , 2022 , 11 : 6541 - 6550 .

ANDREWS J G , BUZZI S , CHOI W , et al . What will 5G be？ [J ] . IEEE Journal on Selected Areas in Communications , 2014 , 32 ( 6 ): 1065 - 1082 .

KONG P Y . Radio resource allocation scheme for reliable demand response management using D2D communications in smart grid [J ] . IEEE Transactions on Smart Grid , 2020 , 11 ( 3 ): 2417 - 2426 .

YANG J J , LIU G , REN J , et al . Resource allocation for intelligent reflecting surface-assisted cooperative NOMA-URLLC networks in smart grid [C ] // Proceedings of 2022 IEEE International Conference on Communications,Control,and Computing Technologies for Smart Grids . Piscataway:IEEE Press , 2022 : 83 - 89 .

CAO Z J , LIN J , WAN C , et al . Optimal cloud computing resource allocation for demand side management in smart grid [J ] . IEEE Transactions on Smart Grid , 2017 , 8 ( 4 ): 1943 - 1955 .

SUN M Y , YUAN Y Z , MA K , et al . Spectrum allocation and computing resources optimization for demand-side cooperative communications in smart grid [J ] . IEEE Transactions on Smart Grid , 2022 , 13 ( 3 ): 1967 - 1975 .

LI Z , LIANG Q L . Capacity optimization in heterogeneous home area networks with application to smart grid [J ] . IEEE Transactions on Vehicular Technology , 2016 , 65 ( 2 ): 699 - 706 .

YIN F F , ZENG M Y , ZHANG Z L , et al . Coded caching for smart grid enabled HetNets with resource allocation and energy cooperation [J ] . IEEE Transactions on Vehicular Technology , 2020 , 69 ( 10 ): 12058 - 12071 .

LIU L L , ZHANG Z Z , WANG N , et al . Online resource management of heterogeneous cellular networks powered by grid-connected smart micro grids [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 10 ): 8416 - 8430 .

GUNGOR V C , SAHIN D , KOCAK T , et al . A survey on smart grid potential applications and communication requirements [J ] . IEEE Transactions on Industrial Informatics , 2013 , 9 ( 1 ): 28 - 42 .

3GPP . Study on integrated access and backhaul (release 16):TR 38.874 [S ] . 2018 .

BELAID M N , AUDEBERT V , DENEUVILLE B , et al . Smart grid critical traffic routing and link scheduling in 5G IAB networks [C ] // Proceedings of 2022 IEEE International Conference on Communications,Control,and Computing Technologies for Smart Grid . Piscataway:IEEE Press , 2022 : 76 - 82 .

YASHIMA T , NISHIYAMA H . Analysis of optimal bandwidth partitioning ratio in full-duplex integrated access and backhaul [C ] // Proceedings of 2022 IEEE International Conference on Communications Workshops . Piscataway:IEEE Press , 2022 : 1 - 6 .

ZHANG S M , XU X D , SUN M Y , et al . Joint spectrum and power allocation in 5G integrated access and backhaul networks at mmWave band [C ] // Proceedings of 2020 IEEE 31st Annual International Symposium on Personal,Indoor and Mobile Radio Communications . Piscataway:IEEE Press , 2020 : 1 - 7 .

PAGIN M , ZUGNO T , POLESE M , et al . Resource management for 5G NR integrated access and backhaul:a semi-centralized approach [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 2 ): 753 - 767 .

WANG X M , ZHANG Y H , SHEN R J , et al . DRL-based energy-efficient resource allocation frameworks for uplink NOMA systems [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 8 ): 7279 - 7294 .

喻鹏 , 张俊也 , 李文璟 , 等 . 移动边缘网络中基于双深度 Q 学习的高能效资源分配方法 [J ] . 通信学报 , 2020 , 41 ( 12 ): 148 - 161 .

YU P , ZHANG J Y , LI W J , et al . Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning [J ] . Journal on Communications , 2020 , 41 ( 12 ): 148 - 161 .

ZHU X Y , WANG J , LI J M , et al . A scheme for uplink NOMA communication with intelligent resource allocation for mMTC traffic over eMBB traffic [C ] // Proceedings of 2022 IEEE 95th Vehicular Technology Conference . Piscataway:IEEE Press , 2022 : 1 - 5 .

徐思雅 , 邢逸斐 , 郭少勇 , 等 . 基于深度强化学习的能源互联网智能巡检任务分配机制 [J ] . 通信学报 , 2021 , 42 ( 5 ): 191 - 204 .

XU S Y , XING Y F , GUO S Y , et al . Deep reinforcement learning based task allocation mechanism for intelligent inspection in energy Internet [J ] . Journal on Communications , 2021 , 42 ( 5 ): 191 - 204 .

LEI W L , YE Y , XIAO M . Deep reinforcement learning-based spectrum allocation in integrated access and backhaul networks [J ] . IEEE Transactions on Cognitive Communications and Networking , 2020 , 6 ( 3 ): 970 - 979 .

CHENG Q Q , WEI Z Q , YUAN J H . Deep reinforcement learning-based spectrum allocation and power management for IAB networks [C ] // Proceedings of 2021 IEEE International Conference on Communications Workshops (ICC Workshops) . Piscataway:IEEE Press , 2021 : 1 - 6 .

CHENG Z P , MIN M H , GAO Z B , et al . Joint task offloading and resource allocation for mobile edge computing in ultra-dense network [C ] // Proceedings of 2021 IEEE Global Communications Conference . Piscataway:IEEE Press , 2021 : 1 - 6 .

MENG F , CHEN P , WU L N , et al . Power allocation in multi-user cellular networks:deep reinforcement learning approaches [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 10 ): 6255 - 6267 .

周凡 , 王鸿 , 宋荣方 . 密集异构蜂窝网络中基于深度强化学习的下行链路功率分配算法 [J ] . 南京邮电大学学报(自然科学版) , 2021 , 41 ( 2 ): 12 - 19 .

ZHOU F , WANG H , SONG R F . Deep reinforcement learning based downlink power allocation algorithm in dense heterogeneous cellular networks [J ] . Journal of Nanjing University of Posts and Telecommunications (Natural Science) , 2021 , 41 ( 2 ): 12 - 19 .

KIM Y , LIM H . Multi-agent reinforcement learning-based resource management for end-to-end network slicing [J ] . IEEE Access , 2021 , 9 : 56178 - 56190 .

GUO D L , TANG L , ZHANG X G , et al . Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2020 , 69 ( 11 ): 13124 - 13138 .

YU C , VELU A , VINITSKY E , et al . The surprising effectiveness of ppo in cooperative multi-agent games [J ] . Advances in Neural Information Processing Systems , 2022 , 35 : 24611 - 24624 .

LOWE R , WU Y , TAMAR A , et al . Multi-agent actor-critic for mixed cooperative-competitive environments [J ] . arXiv Preprint,arXiv:1706.02275 , 2017 .

HOU W J , WEN H , SONG H H , et al . Multiagent deep reinforcement learning for task offloading and resource allocation in cybertwin-based networks [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 22 ): 16256 - 16268 .

Views

1326

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Advances in time-deterministic network technology and application research

Cellular-free RAN slicing resource allocation algorithm based on multi-timescale collaboration

Secure and efficient power data sharing platform construction approach based on blockchain oracle

Multi-agent caching distribution strategy for content freshness guarantee in IoV

Network slicing resource allocation algorithm based on bipartite graph matching in smart grids

Related Author

LYU Ping

CAO Zhipeng

ZHANG Wenjian

LIANG Ruoyu

SONG Ke

WANG Boye

XIA Weiwei

XIA Yaxing

Related Institution

National Digital Switching System Engineering & Research Center, Cyberspace Force Information Engineering University

State Grid Suzhou Power Supply Company

Information & Telecommunication Branch, State Grid Jiangsu Electric Power Co., Ltd.

School of Cyber Engineering, Xidian University

Shenyang Power Supply Company of State Grid Liaoning Province Electric Power Co., Ltd.

AI问答

⁰