Flink环境下基于负载预测的弹性资源调度策略

李梓杨; 于炯; 王跃飞; 卞琛; 蒲勇霖; 张译天; 刘宇

doi:10.11959/j.issn.1000-436x.2020195

您当前的位置：

首页 >

文章列表页 >

Flink环境下基于负载预测的弹性资源调度策略

学术论文 | 更新时间：2024-06-05

- Flink环境下基于负载预测的弹性资源调度策略
- Load prediction based elastic resource scheduling strategy in Flink
- 通信学报 2020年41卷第10期页码：92-108
- 作者机构：
  
  1. 新疆大学软件学院，新疆乌鲁木齐 830008
  2. 新疆大学信息科学与工程学院，新疆乌鲁木齐 830046
  3. 成都大学计算机学院，四川成都 610106
  4. 广东金融学院互联网金融与信息工程学院，广东广州 510521
- 作者简介：
  
  [ "李梓杨（1993- ），男，新疆乌鲁木齐人，新疆大学博士生，主要研究方向为分布式系统、内存计算、流式计算" ]
  [ "于炯（1964- ），男，北京人，博士，新疆大学教授、博士生导师，主要研究方向为网格计算、并行计算、分布式系统" ]
  [ "王跃飞（1991- ），男，新疆乌鲁木齐人，博士，成都大学讲师，主要研究方向为数据挖掘、机器学习" ]
  [ "卞琛（1981- ），男，江苏南京人，博士，广东金融学院副教授，主要研究方向为分布式系统、内存计算、绿色计算" ]
  [ "蒲勇霖（1991- ），男，山东淄博人，新疆大学博士生，主要研究方向为内存计算、流式计算、绿色计算" ]
  [ "张译天（1995- ），男，河南商丘人，新疆大学硕士生，主要研究方向为云计算、实时计算、分布式计算" ]
  [ "刘宇（1996- ），男，新疆克拉玛依人，新疆大学硕士生，主要研究方向为云计算、分布式计算" ]
- 基金信息：
  
  国家自然科学基金资助项目(61862060);国家自然科学基金资助项目(61462079);国家自然科学基金资助项目(61562086);国家自然科学基金资助项目(61562078);新疆维吾尔自治区自然科学基金资助项目(2017D01A20);新疆大学博士生科技创新基金资助项目(XJUBSCX-201902)
- DOI：10.11959/j.issn.1000-436x.2020195
  中图分类号： TP311
- 网络首发：2020-10，
  
  纸质出版：2020-10-25
- 稿件说明：
移动端阅览
李梓杨, 于炯, 王跃飞, 等. Flink环境下基于负载预测的弹性资源调度策略[J]. 通信学报, 2020,41(10):92-108.

Ziyang LI, Jiong YU, Yuefei WANG, et al. Load prediction based elastic resource scheduling strategy in Flink[J]. Journal on Communications, 2020, 41(10): 92-108.
李梓杨, 于炯, 王跃飞, 等. Flink环境下基于负载预测的弹性资源调度策略[J]. 通信学报, 2020,41(10):92-108. DOI： 10.11959/j.issn.1000-436x.2020195.

Ziyang LI, Jiong YU, Yuefei WANG, et al. Load prediction based elastic resource scheduling strategy in Flink[J]. Journal on Communications, 2020, 41(10): 92-108. DOI： 10.11959/j.issn.1000-436x.2020195.

摘要

为了解决大数据流式计算平台中存在计算负载剧烈波动，但集群因资源不足而遇到性能瓶颈的问题，提出了Flink环境下基于负载预测的弹性资源调度（LPERS-Flink）策略。首先，建立负载预测模型并在此基础上提出负载预测算法，预测集群负载的变化趋势；其次，建立资源判定模型，以判定集群出现资源瓶颈与资源过剩的问题，由此提出弹性资源调度算法，制定弹性资源调度计划；最后，通过在线负载迁移算法执行调度计划，实现高效的节点间负载迁移。实验结果表明，该策略在负载剧烈波动的应用场景中有较好的优化效果，实现了集群规模和资源配置对负载变化的及时响应，降低了负载迁移的通信开销。

Abstract

In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources

the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly

the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly

the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally

the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively.

关键词

Keywords

references

彭安妮 , 周威 , 贾岩 , 等 . 物联网操作系统安全研究综述 [J ] . 通信学报 , 2018 , 39 ( 3 ): 22 - 34 .

PENG A N , ZHOU W , JIA Y , et al . Survey of the Internet of things operating system security [J ] . Journal on Communications , 2018 , 39 ( 3 ): 22 - 34 .

DEAN J , GHEMAWAT S . MapReduce:simplified data processing on large clusters [J ] . Communications of the ACM , 2008 , 51 ( 1 ): 107 - 113 .

卞琛 , 于炯 , 修位蓉 , 等 . 基于分配适应度的 Spark 渐进填充分区映射算法 [J ] . 通信学报 , 2017 , 38 ( 9 ): 133 - 147 .

BIAN C , YU J , XIU W R , et al . Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree [J ] . Journal on Communications , 2017 , 38 ( 9 ): 133 - 147 .

卞琛 , 于炯 , 修位蓉 , 等 . 内存计算框架局部数据优先拉取策略 [J ] . 计算机研究与发展 , 2017 , 54 ( 4 ): 787 - 803 .

BIAN C , YU J , XIU W R , et al . Partial data shuffled first strategy for in-memory computing framework [J ] . Journal of Computer Research and Development , 2017 , 54 ( 4 ): 787 - 803 .

孙大为 , 张广艳 , 郑纬民 . 大数据流式计算:关键技术及系统实例 [J ] . 软件学报 , 2014 , 25 ( 4 ): 839 - 862 .

SUN D W , ZHANG G Y , ZHENG W M . Big data stream computing:technologies and instances [J ] . Journal of Software , 2014 , 25 ( 4 ): 839 - 862 .

ALEXANDROVE A , BERGMANN R , EWEN S , et al . The stratosphere platform for big data analytics [J ] . The VLDB Journal , 2014 , 23 ( 6 ): 939 - 964 .

CARBONE P , KATSIFODIMOS A , EWEN S , et al . Apache Flink:stream and batch processing in a single engine [J ] . Bulletin of the IEEE Computer Society Technical Committee on Data Engineering , 2015 , 36 ( 4 ): 28 - 38 .

TOSHNIWAL A , TANEJA S , SHUKLA A , et al . Storm @Twitter [C ] // The 2014 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2014 : 147 - 156 .

CARBONE P , EWEN S , FÓRA G , et al . State management in Apache Flink®:consistent stateful distributed stream processing [J ] . Proceedings of the VLDB Endowment , 2017 , 10 ( 12 ): 1718 - 1729 .

PARIS C , GYULA F , STEPHAN E , et al . Lightweight asynchronous snapshots for distributed dataflows [J ] . Computer Science,arXiv Preprint,arXiv:1506.08603 , 2015

KULKARNI S , BHAGAT N , FU M , et al . Twitter Heron:stream processing at scale [C ] // Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2015 : 239 - 250 .

FLORATOU A , AGRAWAL A . Self-regulating streaming systems:challenges and opportunities [C ] // Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics . New York:ACM Press , 2017 : 1 - 5 .

SUN D , ZHANG G , YANG S , et al . Re-stream:real-time and energy-efficient resource scheduling in big data stream computing environments [J ] . Information Sciences , 2015 , 319 : 92 - 112 .

蒲勇霖 , 于炯 , 鲁亮 , 等 . Storm平台下工作节点的内存电压调控节能策略 [J ] . 通信学报 , 2018 , 39 ( 10 ): 97 - 117 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for work node by DRAM voltage regulation in Storm [J ] . Journal on Communications , 2018 , 39 ( 10 ): 97 - 117 .

SUN D , FU G , LIU X , et al . Optimizing data stream graph for big data stream computing in cloud datacenter environments [J ] . International Journal of Advancements in Computing Technology , 2014 , 6 ( 5 ):53.

蒲勇霖 , 于炯 , 鲁亮 , 等 . 基于 Storm 平台的数据迁移合并节能策略 [J ] . 通信学报 , 2019 , 40 ( 12 ): 68 - 85 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for data migration and merging in Storm [J ] . Journal on Communications , 2019 , 40 ( 12 ): 68 - 85 .

ZHANG C , CHEN X , LI Z , et al . An on-the-fly scheduling strategy for distributed stream processing platform [C ] // IEEE International Conference on Parallel ＆ Distributed Processing with Applications,Ubiquitous Computing ＆ Communications,Big Data ＆ Cloud Computing,Social Computing ＆ Networking,Sustainable Computing ＆ Communications . Piscataway:IEEE Press , 2018 : 773 - 780 .

SHUKLA A , SIMMHAN Y . Model-driven scheduling for distributed stream processing systems [J ] . Journal of Parallel and Distributed Computing , 2018 , 117 : 98 - 114 .

CARDELLINI V , MENCAGLI G , TALIA D , et al . New landscapes of the data stream processing in the era of fog computing [J ] . Future Generation Computer Systems , 2019 , 99 : 646 - 650 .

TANTALAKI N , SOURAVLAS S , ROUMELIOTIS M , et al . Linear scheduling of big data streams on multiprocessor sets in the cloud [C ] // IEEE/WIC/ACM International Conference on Web Intelligence . New York:ACM Press , 2019 : 107 - 115 .

ESKANDARI L , MAIR J , HUANG Z , et al . T3-Scheduler:a topology and traffic aware two-level Scheduler for stream processing systems in a heterogeneous cluster [J ] . Future Generation Computer Systems , 2018 , 89 : 617 - 632 .

SILVA V A , DE-SOUZA F R , DE-ASSUNÇÃO M D , et al . Multi-objective reinforcement learning for reconfiguring data stream analytics on edge computing [C ] // Proceedings of the 48th International Conference on Parallel Processing . New York:ACM Press , 2019 :106.

LOUKOPOULOS T , TZIRITAS N , KOZIRI M , et al . A pareto-efficient algorithm for data stream processing at network edges [C ] // 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom) . Piscataway:IEEE Press , 2018 : 159 - 162 .

PAGLIARI A , HUET F , URVOY-KELLER G . On the cost of acking in data stream processing systems [C ] // 19th IEEE/ACM International Symposium on Cluster,Cloud,and Grid Computing . Piscataway:IEEE Press , 2019 : 14 - 17 .

ZHOU S , ZHANG F , CHEN H , et al . Fastjoin:a skewness-aware distributed stream join system [C ] // 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . Piscataway:IEEE Press , 2019 : 1042 - 1052 .

RÖGER H , MAYER R . A comprehensive survey on parallelization and elasticity in stream processing [J ] . ACM Computing Surveys (CSUR) , 2019 , 52 ( 2 ):36.

LIU S , WENG J , WANG J H , et al . An adaptive online scheme for scheduling and resource enforcement in Storm [J ] . IEEE/ACM Transactions on Networking , 2019 , 27 ( 4 ): 1373 - 1386 .

RUSSO G R , CARDELLINI V , PRESTI F L . Reinforcement learning based policies for elastic stream processing on heterogeneous resources [C ] // Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems . New York:ACM Press , 2019 : 31 - 42 .

RUSSO G , NARDELLI M , CARDELLINI V , et al . Multi-level elasticity for wide-area data streaming systems:a reinforcement learning approach [J ] . Algorithms , 2018 , 11 ( 9 ):134.

CARDELLINI V , PRESTI F L , NARDELLI M , et al . Towards hierarchical autonomous control for elastic data stream processing in the fog [C ] // European Conference on Parallel Processing . Berlin:Springer , 2017 : 106 - 117 .

MEHDI B , CÉDRIC T , . A fully decentralized autoscaling algorithm for stream processing applications [C ] // Auto-DaSP 2019-Third International Workshop on Autonomic Solutions for Parallel and Distributed Data Stream Processing . Berlin:Springer , 2019 : 1 - 12 .

RUSSO G R , . Self-adaptive data stream processing in geo-distributed computing environments [C ] // Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems . New York:ACM Press , 2019 : 276 - 279 .

MENCAGLI G , TORQUATI M , DANELUTTO M . Elastic-PPQ:a two-level autonomic system for spatial preference query processing over dynamic data streams [J ] . Future Generation Computer Systems , 2018 , 79 : 862 - 877 .

HIDALGO N , WLADDIMIRO D , ROSAS E . Self-adaptive processing graph with operator fission for elastic stream processing [J ] . Journal of Systems and Software , 2017 , 127 : 205 - 216 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的Flink平台弹性资源调度策略 [J ] . 通信学报 , 2019 , 40 ( 8 ): 85 - 101 .

LI Z Y , YU J , BIAN C , et al . Flow-network based auto rescale strategy for Flink [J ] . Journal on Communications , 2019 , 40 ( 8 ): 85 - 101 .

LOHRMANN B , JANACIK P , KAO O . Elastic stream processing with latency guarantees [C ] // 2015 IEEE 35th International Conference on Distributed Computing Systems . Piscataway:IEEE Press , 2015 : 399 - 410 .

SUN D , GAO S , LIU X , et al . State and runtime-aware scheduling in elastic stream computing systems [J ] . Future Generation Computer Systems , 2019 , 97 : 194 - 209 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的流式计算动态任务调度策略 [J ] . 计算机应用 , 2018 , 38 ( 9 ): 2560 - 2567 .

LI Z Y , YU J , BIAN C , et al . Dynamic task dispatching strategy for stream processing based on flow network [J ] . Journal of Computer Application , 2018 , 38 ( 9 ): 2560 - 2567 .

浏览量

1541

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于流网络的Flink平台弹性资源调度策略

流式大数据平台下的弹性数据迁移能效优化策略

基于DDPG强化学习的多集群算力资源调度算法

基于深度强化学习的分离式数据中心存储资源调度优化方法

基于多智能体深度强化学习的低轨星座跳波束资源调度研究