基于深度强化学习的转发效能感知流量调度算法

沙宗轩; 霍如; 孙闯; 汪硕; 黄韬

doi:10.11959/j.issn.1000-436x.2022148

您当前的位置：

首页 >

文章列表页 >

基于深度强化学习的转发效能感知流量调度算法

学术论文 | 更新时间：2024-06-05

- 基于深度强化学习的转发效能感知流量调度算法
- Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning
- 通信学报 2022年43卷第8期页码：30-40
- 作者机构：
  
  1. 北京工业大学信息学部，北京 100124
  2. 网络通信与安全紫金山实验室, 江苏南京 211111
  3. 清华大学自动化系，北京 100084
  4. 北京邮电大学网络与交换国家重点实验室，北京 100876
- 作者简介：
  
  [ "沙宗轩（1990- ），男，回族，安徽蚌埠人，北京工业大学博士生，主要研究方向为未来网络、网络人工智能深度学习、强化学习等" ]
  [ "霍如（1988- ），女，黑龙江哈尔滨人，博士，北京工业大学讲师，主要研究方向为未来网络、工业互联网、边缘计算、网络资源管理、区块链等" ]
  [ "孙闯（1989- ），男，黑龙江哈尔滨人，博士，清华大学在站博士后，主要研究方向为传感器测量与测试技术、初始仪器精密标定方法和惯性导航技术等" ]
  [ "汪硕（1991- ），男，河南灵宝人，博士，北京邮电大学讲师，主要研究方向为数据中心网络、软件定义网络、网络流量调度等" ]
  [ "黄韬（1980- ），男，重庆人，博士，北京邮电大学教授，主要研究方向为未来网络体系架构、软件定义网络、网络虚拟化等" ]
- 基金信息：
  
  2020年工业互联网创新发展工程基金资助项目（工业互联网标识资源搜索系统）;The MIIT of China 2020 (Identification Resources Search System for Industrial Internet of Things);中国通信学会青年人才托举计划-托举培养协议基金资助项目;Young Elite Scientist Sponsorship Program by CAST and China-CIC(YESS20200287)
- DOI：10.11959/j.issn.1000-436x.2022148
  中图分类号： TN911.5
- 网络首发：2022-08，
  
  纸质出版：2022-08-25
- 稿件说明：
移动端阅览
沙宗轩, 霍如, 孙闯, 等. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022,43(8):30-40.

Zongxuan SHA, Ru HUO, Chuang SUN, et al. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning[J]. Journal on Communications, 2022, 43(8): 30-40.
沙宗轩, 霍如, 孙闯, 等. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022,43(8):30-40. DOI： 10.11959/j.issn.1000-436x.2022148.

Zongxuan SHA, Ru HUO, Chuang SUN, et al. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning[J]. Journal on Communications, 2022, 43(8): 30-40. DOI： 10.11959/j.issn.1000-436x.2022148.

摘要

软件定义网络（SDN）通过将控制平面与数据平面分离，可实现灵活的流量调度，更有效地利用网络资源。但是，流表项数量、设备负载率及连接主机数量增加等因素的共同作用会导致 SDN 交换机的转发效能降低，进而影响端到端的数据传输时延。为了解决上述问题，提出了基于深度强化学习的转发效能感知流量调度算法。首先，将交换机状态信息统一到感知模型中，通过神经网络建立交换机状态信息和转发效能的映射关系。然后，结合网络状态和流量信息，通过深度强化学习产生流量调度策略。最后，通过由最短路径和负载均衡算法产生的专家样本引导模型训练，不仅使模型学习到专家样本的知识以提升性能，同时提升模型训练效率。实验结果表明，与其他算法相比，所提算法不仅使端到端的平均传输时延降低了15.31%，而且保证了网络整体的负载均衡。

Abstract

The software defined network separates the control plane from the data plane to achieve flexible traffic scheduling

which can use network resources more efficiently.However

with the increase of the number of flow entries

load rate

the number of connected hosts

and other factors

the forwarding efficiency of the SDN switch will be reduced

which will affect the end-to-end transmission delay.To solve the above problems

the forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning was proposed.First

the switch state was integrated into the perception model

and the mapping relationship between switch state information and forwarding efficiency was established based on neural network.Then

combined with network state and traffic information

traffic scheduling policy was generated through deep reinforcement learning.Finally

the expert samples generated by the shortest path and load balance algorithms could guide the model training

which enabled the model to learn knowledge from expert samples to improve performance and accelerated the training process.The experimental results show that the proposed algorithm not only reduces the average end-to-end transmission delay by 15.31%

but also ensures the overall load balance of the network

compared with other algorithms.

关键词

Keywords

references

周桐庆 , 蔡志平 , 夏竟 , 等 . 基于软件定义网络的流量工程 [J ] . 软件学报 , 2016 , 27 ( 2 ): 394 - 417 .

ZHOU T Q , CAI Z P , XIA J , et al . Traffic engineering for software defined networks [J ] . Journal of Software , 2016 , 27 ( 2 ): 394 - 417 .

MENDIOLA A , ASTORGA J , JACOB E , et al . A survey on the contributions of software-defined networking to traffic engineering [J ] . IEEE Communications Surveys ＆ Tutorials , 2017 , 19 ( 2 ): 918 - 953 .

杨思锦 , 庄雷 , 宋玉 , 等 . 多模态网络中时间敏感网络模态的智能调度机制 [J ] . 通信学报 , 2022 , 43 ( 5 ): 82 - 91 .

YANG S J , ZHUANG L , SONG Y , et al . Intelligent scheduling mechanism of time-sensitive network modal in polymorphic network [J ] . Journal on Communications , 2022 , 43 ( 5 ): 82 - 91 .

SARASWAT S , AGARWAL V , GUPTA H P , et al . Challenges and solutions in software defined networking:a survey [J ] . Journal of Network and Computer Applications , 2019 , 141 : 23 - 58 .

黄韬 , 刘江 , 汪硕 , 等 . 未来网络技术与发展趋势综述 [J ] . 通信学报 , 2021 , 42 ( 1 ): 130 - 150 .

HUANG T , LIU J , WANG S , et al . Survey of the future network technology and trend [J ] . Journal on Communications , 2021 , 42 ( 1 ): 130 - 150 .

LECUN Y , BENGIO Y , HINTON G . Deep learning [J ] . Nature , 2015 , 521 ( 7553 ): 436 - 444 .

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [M ] . Massachusetts : MIT Press , 1998 .

XIAO Y , LIU J , WU J W , et al . Leveraging deep reinforcement learning for traffic engineering:a survey [J ] . IEEE Communications Surveys ＆ Tutorials , 2021 , 23 ( 4 ): 2064 - 2097 .

XIE J F , YU F R , HUANG T , et al . A survey of machine learning techniques applied to software defined networking (SDN):research issues and challenges [J ] . IEEE Communications Surveys ＆ Tutorials , 2019 , 21 ( 1 ): 393 - 430 .

HARTMAN T , HASSIDIM A , KAPLAN H , et al . How to split a flow? [C ] // Proceedings of IEEE INFOCOM . Piscataway:IEEE Press , 2012 : 828 - 836 .

ZHANG J J , YE M H , GUO Z H , et al . CFR-RL:traffic engineering with reinforcement learning in SDN [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 10 ): 2249 - 2259 .

ZHANG J J , GUO Z H , YE M H , et al . SmartEntry:mitigating routing update overhead with reinforcement learning for traffic engineering [C ] // Proceedings of the Workshop on Network Meets AI ＆ ML . Piscataway:IEEE Press , 2020 : 1 - 7 .

HUANG X H , YUAN T T , QIAO G H , et al . Deep reinforcement learning for multimedia traffic control in software defined networking [J ] . IEEE Network , 2018 , 32 ( 6 ): 35 - 41 .

HUANG X H , ZENG M , XIE K . Intelligent traffic control for QoS optimization in hybrid SDNs [J ] . Computer Networks , 2021 , 189 : 107877 .

ZHANG Q Y , WANG X W , LV J H , et al . Intelligent content-aware traffic engineering for SDN:an AI-driven approach [J ] . IEEE Network , 2020 , 34 ( 3 ): 186 - 193 .

XU Z Y , TANG J , MENG J S , et al . Experience-driven networking:a deep reinforcement learning based approach [C ] // Proceedings of IEEE Conference on Computer Communications . Piscataway:IEEE Press , 2018 : 1871 - 1879 .

KATO N , FADLULLAH Z M , MAO B M , et al . The deep learning vision for heterogeneous network traffic control:proposal,challenges,and future perspective [J ] . IEEE Wireless Communications , 2017 , 24 ( 3 ): 146 - 153 .

ZHANG X C , WANG Y L , GENG G G , et al . Delay-optimized multicast tree packing in software-defined networks [J ] . IEEE Transactions on Services Computing , 2021 , PP ( 99 ): 1 .

WU T , ZHOU P , WANG B H , et al . Joint traffic control and multi-channel reassignment for core backbone network in SDN-IoT:a multi-agent deep reinforcement learning approach [J ] . IEEE Transactions on Network Science and Engineering , 2021 , 8 ( 1 ): 231 - 245 .

SAHA N , MISRA S , BERA S . Q-flag:QoS-aware flow-rule aggregation in software-defined IoT networks [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 7 ): 4899 - 4906 .

FILALI A , KOBBANE A , ELMACHKOUR M , et al . SDN controller assignment and load balancing with minimum quota of processing capacity [C ] // Proceedings of IEEE International Conference on Communications (ICC) . Piscataway:IEEE Press , 2018 : 1 - 6 .

SAVAS S S , TORNATORE M , DIKBIYIK F , et al . RASCAR:recovery-aware switch-controller assignment and routing in SDN [J ] . IEEE Transactions on Network and Service Management , 2018 , 15 ( 4 ): 1222 - 1234 .

WANG T , LIU F M , XU H . An efficient online algorithm for dynamic SDN controller assignment in data center networks [J ] . IEEE/ACM Transactions on Networking , 2017 , 25 ( 5 ): 2788 - 2801 .

BERA S , MISRA S , SAHA N . Traffic-aware dynamic controller assignment in SDN [J ] . IEEE Transactions on Communications , 2020 , 68 ( 7 ): 4375 - 4382 .

BOUZIDI E H , OUTTAGARTS A , LANGAR R , et al . Dynamic clustering of software defined network switches and controller placement using deep reinforcement learning [J ] . Computer Networks , 2022 ,207:108852.

LIN N , ZHAO Q , ZHAO L , et al . A novel cost-effective controller placement scheme for software-defined vehicular networks [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 18 ): 14080 - 14093 .

MAITY I , MISRA S , MANDAL C . CORE:prediction-based control plane load reduction in software-defined IoT networks [J ] . IEEE Transactions on Communications , 2021 , 69 ( 3 ): 1835 - 1844 .

浏览量

910

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

融合SDN与目标导向分层强化学习的覆盖组播路由方法

基于DDPG强化学习的多集群算力资源调度算法

时间确定性网络技术与应用研究进展

智能反射表面辅助的车载边缘任务卸载和资源分配策略

数字孪生辅助下基于D3QN的车载网络协同卸载算法