Load prediction based elastic resource scheduling strategy in Flink

Ziyang LI; Jiong YU; Yuefei WANG; Chen BIAN; Yonglin PU; Yitian ZHANG; Yu LIU

doi:10.11959/j.issn.1000-436x.2020195

您当前的位置：

首页 >

文章列表页 >

Load prediction based elastic resource scheduling strategy in Flink

Papers | 更新时间：2024-06-05

- Load prediction based elastic resource scheduling strategy in Flink
- Journal on Communications Vol. 41, Issue 10, Pages: 92-108(2020)
- 作者机构：
  
  1. 新疆大学软件学院，新疆乌鲁木齐 830008
  2. 新疆大学信息科学与工程学院，新疆乌鲁木齐 830046
  3. 成都大学计算机学院，四川成都 610106
  4. 广东金融学院互联网金融与信息工程学院，广东广州 510521
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61862060);The National Natural Science Foundation of China(61462079);The National Natural Science Foundation of China(61562086);The National Natural Science Foundation of China(61562078);The Natural Science Foundation of Xinjiang Uygur Autonomous Region(2017D01A20);The Doctoral Innovation Program of Xinjiang University(XJUBSCX-201902)
- DOI：10.11959/j.issn.1000-436x.2020195
  CLC： TP311
- Online First：2020-10，
  
  Published：25 October 2020
- 稿件说明：
移动端阅览
Ziyang LI, Jiong YU, Yuefei WANG, et al. Load prediction based elastic resource scheduling strategy in Flink[J]. Journal on Communications, 2020, 41(10): 92-108.
DOI：

Ziyang LI, Jiong YU, Yuefei WANG, et al. Load prediction based elastic resource scheduling strategy in Flink[J]. Journal on Communications, 2020, 41(10): 92-108. DOI： 10.11959/j.issn.1000-436x.2020195.

摘要

为了解决大数据流式计算平台中存在计算负载剧烈波动，但集群因资源不足而遇到性能瓶颈的问题，提出了Flink环境下基于负载预测的弹性资源调度（LPERS-Flink）策略。首先，建立负载预测模型并在此基础上提出负载预测算法，预测集群负载的变化趋势；其次，建立资源判定模型，以判定集群出现资源瓶颈与资源过剩的问题，由此提出弹性资源调度算法，制定弹性资源调度计划；最后，通过在线负载迁移算法执行调度计划，实现高效的节点间负载迁移。实验结果表明，该策略在负载剧烈波动的应用场景中有较好的优化效果，实现了集群规模和资源配置对负载变化的及时响应，降低了负载迁移的通信开销。

Abstract

In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources

the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly

the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly

the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally

the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively.

关键词

Keywords

references

彭安妮 , 周威 , 贾岩 , 等 . 物联网操作系统安全研究综述 [J ] . 通信学报 , 2018 , 39 ( 3 ): 22 - 34 .

PENG A N , ZHOU W , JIA Y , et al . Survey of the Internet of things operating system security [J ] . Journal on Communications , 2018 , 39 ( 3 ): 22 - 34 .

DEAN J , GHEMAWAT S . MapReduce:simplified data processing on large clusters [J ] . Communications of the ACM , 2008 , 51 ( 1 ): 107 - 113 .

卞琛 , 于炯 , 修位蓉 , 等 . 基于分配适应度的 Spark 渐进填充分区映射算法 [J ] . 通信学报 , 2017 , 38 ( 9 ): 133 - 147 .

BIAN C , YU J , XIU W R , et al . Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree [J ] . Journal on Communications , 2017 , 38 ( 9 ): 133 - 147 .

卞琛 , 于炯 , 修位蓉 , 等 . 内存计算框架局部数据优先拉取策略 [J ] . 计算机研究与发展 , 2017 , 54 ( 4 ): 787 - 803 .

BIAN C , YU J , XIU W R , et al . Partial data shuffled first strategy for in-memory computing framework [J ] . Journal of Computer Research and Development , 2017 , 54 ( 4 ): 787 - 803 .

孙大为 , 张广艳 , 郑纬民 . 大数据流式计算:关键技术及系统实例 [J ] . 软件学报 , 2014 , 25 ( 4 ): 839 - 862 .

SUN D W , ZHANG G Y , ZHENG W M . Big data stream computing:technologies and instances [J ] . Journal of Software , 2014 , 25 ( 4 ): 839 - 862 .

ALEXANDROVE A , BERGMANN R , EWEN S , et al . The stratosphere platform for big data analytics [J ] . The VLDB Journal , 2014 , 23 ( 6 ): 939 - 964 .

CARBONE P , KATSIFODIMOS A , EWEN S , et al . Apache Flink:stream and batch processing in a single engine [J ] . Bulletin of the IEEE Computer Society Technical Committee on Data Engineering , 2015 , 36 ( 4 ): 28 - 38 .

TOSHNIWAL A , TANEJA S , SHUKLA A , et al . Storm @Twitter [C ] // The 2014 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2014 : 147 - 156 .

CARBONE P , EWEN S , FÓRA G , et al . State management in Apache Flink®:consistent stateful distributed stream processing [J ] . Proceedings of the VLDB Endowment , 2017 , 10 ( 12 ): 1718 - 1729 .

PARIS C , GYULA F , STEPHAN E , et al . Lightweight asynchronous snapshots for distributed dataflows [J ] . Computer Science,arXiv Preprint,arXiv:1506.08603 , 2015

KULKARNI S , BHAGAT N , FU M , et al . Twitter Heron:stream processing at scale [C ] // Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2015 : 239 - 250 .

FLORATOU A , AGRAWAL A . Self-regulating streaming systems:challenges and opportunities [C ] // Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics . New York:ACM Press , 2017 : 1 - 5 .

SUN D , ZHANG G , YANG S , et al . Re-stream:real-time and energy-efficient resource scheduling in big data stream computing environments [J ] . Information Sciences , 2015 , 319 : 92 - 112 .

蒲勇霖 , 于炯 , 鲁亮 , 等 . Storm平台下工作节点的内存电压调控节能策略 [J ] . 通信学报 , 2018 , 39 ( 10 ): 97 - 117 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for work node by DRAM voltage regulation in Storm [J ] . Journal on Communications , 2018 , 39 ( 10 ): 97 - 117 .

SUN D , FU G , LIU X , et al . Optimizing data stream graph for big data stream computing in cloud datacenter environments [J ] . International Journal of Advancements in Computing Technology , 2014 , 6 ( 5 ):53.

蒲勇霖 , 于炯 , 鲁亮 , 等 . 基于 Storm 平台的数据迁移合并节能策略 [J ] . 通信学报 , 2019 , 40 ( 12 ): 68 - 85 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for data migration and merging in Storm [J ] . Journal on Communications , 2019 , 40 ( 12 ): 68 - 85 .

ZHANG C , CHEN X , LI Z , et al . An on-the-fly scheduling strategy for distributed stream processing platform [C ] // IEEE International Conference on Parallel ＆ Distributed Processing with Applications,Ubiquitous Computing ＆ Communications,Big Data ＆ Cloud Computing,Social Computing ＆ Networking,Sustainable Computing ＆ Communications . Piscataway:IEEE Press , 2018 : 773 - 780 .

SHUKLA A , SIMMHAN Y . Model-driven scheduling for distributed stream processing systems [J ] . Journal of Parallel and Distributed Computing , 2018 , 117 : 98 - 114 .

CARDELLINI V , MENCAGLI G , TALIA D , et al . New landscapes of the data stream processing in the era of fog computing [J ] . Future Generation Computer Systems , 2019 , 99 : 646 - 650 .

TANTALAKI N , SOURAVLAS S , ROUMELIOTIS M , et al . Linear scheduling of big data streams on multiprocessor sets in the cloud [C ] // IEEE/WIC/ACM International Conference on Web Intelligence . New York:ACM Press , 2019 : 107 - 115 .

ESKANDARI L , MAIR J , HUANG Z , et al . T3-Scheduler:a topology and traffic aware two-level Scheduler for stream processing systems in a heterogeneous cluster [J ] . Future Generation Computer Systems , 2018 , 89 : 617 - 632 .

SILVA V A , DE-SOUZA F R , DE-ASSUNÇÃO M D , et al . Multi-objective reinforcement learning for reconfiguring data stream analytics on edge computing [C ] // Proceedings of the 48th International Conference on Parallel Processing . New York:ACM Press , 2019 :106.

LOUKOPOULOS T , TZIRITAS N , KOZIRI M , et al . A pareto-efficient algorithm for data stream processing at network edges [C ] // 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom) . Piscataway:IEEE Press , 2018 : 159 - 162 .

PAGLIARI A , HUET F , URVOY-KELLER G . On the cost of acking in data stream processing systems [C ] // 19th IEEE/ACM International Symposium on Cluster,Cloud,and Grid Computing . Piscataway:IEEE Press , 2019 : 14 - 17 .

ZHOU S , ZHANG F , CHEN H , et al . Fastjoin:a skewness-aware distributed stream join system [C ] // 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . Piscataway:IEEE Press , 2019 : 1042 - 1052 .

RÖGER H , MAYER R . A comprehensive survey on parallelization and elasticity in stream processing [J ] . ACM Computing Surveys (CSUR) , 2019 , 52 ( 2 ):36.

LIU S , WENG J , WANG J H , et al . An adaptive online scheme for scheduling and resource enforcement in Storm [J ] . IEEE/ACM Transactions on Networking , 2019 , 27 ( 4 ): 1373 - 1386 .

RUSSO G R , CARDELLINI V , PRESTI F L . Reinforcement learning based policies for elastic stream processing on heterogeneous resources [C ] // Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems . New York:ACM Press , 2019 : 31 - 42 .

RUSSO G , NARDELLI M , CARDELLINI V , et al . Multi-level elasticity for wide-area data streaming systems:a reinforcement learning approach [J ] . Algorithms , 2018 , 11 ( 9 ):134.

CARDELLINI V , PRESTI F L , NARDELLI M , et al . Towards hierarchical autonomous control for elastic data stream processing in the fog [C ] // European Conference on Parallel Processing . Berlin:Springer , 2017 : 106 - 117 .

MEHDI B , CÉDRIC T , . A fully decentralized autoscaling algorithm for stream processing applications [C ] // Auto-DaSP 2019-Third International Workshop on Autonomic Solutions for Parallel and Distributed Data Stream Processing . Berlin:Springer , 2019 : 1 - 12 .

RUSSO G R , . Self-adaptive data stream processing in geo-distributed computing environments [C ] // Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems . New York:ACM Press , 2019 : 276 - 279 .

MENCAGLI G , TORQUATI M , DANELUTTO M . Elastic-PPQ:a two-level autonomic system for spatial preference query processing over dynamic data streams [J ] . Future Generation Computer Systems , 2018 , 79 : 862 - 877 .

HIDALGO N , WLADDIMIRO D , ROSAS E . Self-adaptive processing graph with operator fission for elastic stream processing [J ] . Journal of Systems and Software , 2017 , 127 : 205 - 216 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的Flink平台弹性资源调度策略 [J ] . 通信学报 , 2019 , 40 ( 8 ): 85 - 101 .

LI Z Y , YU J , BIAN C , et al . Flow-network based auto rescale strategy for Flink [J ] . Journal on Communications , 2019 , 40 ( 8 ): 85 - 101 .

LOHRMANN B , JANACIK P , KAO O . Elastic stream processing with latency guarantees [C ] // 2015 IEEE 35th International Conference on Distributed Computing Systems . Piscataway:IEEE Press , 2015 : 399 - 410 .

SUN D , GAO S , LIU X , et al . State and runtime-aware scheduling in elastic stream computing systems [J ] . Future Generation Computer Systems , 2019 , 97 : 194 - 209 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的流式计算动态任务调度策略 [J ] . 计算机应用 , 2018 , 38 ( 9 ): 2560 - 2567 .

LI Z Y , YU J , BIAN C , et al . Dynamic task dispatching strategy for stream processing based on flow network [J ] . Journal of Computer Application , 2018 , 38 ( 9 ): 2560 - 2567 .

Views

1641

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Flow-network based auto rescale strategy for Flink

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

Multi-cluster computing power resource scheduling algorithm based on DDPG reinforcement learning

Storage resource scheduling optimization method for separated data center based on deep reinforcement learning

Research on low earth orbit constellation beam hopping resource scheduling based on multi-agent deep reinforcement learning

Related Author

Yuefei WANG

Liang LU

Yonglin PU

Xiaolong XU

Jiong YU

Ziyang LI

Binglei GUO

HONG Yuchen

Related Institution

School of Computer Science and Technology, Civil Aviation University of China

School of Software, Nanjing University of Information Science and Technology

School of Computer Engineering, Hubei University of Arts and Science, Xiangyang

CHN Energy Zhi Shen Control Technology Co., Ltd.

Computer Network information Center, Chinese Academy of Sciences

AI问答

⁰