Energy-efficient strategy for data migration and merging in Storm

Yonglin PU; Jiong YU; Liang LU; Ziyang LI; Chen BIAN; Bin LIAO

doi:10.11959/j.issn.1000-436x.2019226

您当前的位置：

首页 >

文章列表页 >

Energy-efficient strategy for data migration and merging in Storm

Papers | 更新时间：2024-06-05

- Energy-efficient strategy for data migration and merging in Storm
- Journal on Communications Vol. 40, Issue 12, Pages: 68-85(2019)
- 作者机构：
  
  1. 新疆大学信息科学与工程学院，新疆乌鲁木齐 830046
  2. 中国民航大学计算机科学与技术学院，天津 300300
  3. 广东金融学院互联网金融与信息工程学院，广东广州 510521
  4. 新疆财经大学统计与信息学院，新疆乌鲁木齐 830012
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61862060);The National Natural Science Foundation of China(61462079);The National Natural Science Foundation of China(61562086);The National Natural Science Foundation of China(61562078);The Research Innovation Project of Graduate Student in Xinjiang Uygur Autonomous Region(XJGRI2016028);The Doctoral Innovation Program of Xinjiang University(XJUBSCX-201902)
- DOI：10.11959/j.issn.1000-436x.2019226
  CLC： TP311
- Online First：2019-12，
  
  Published：25 December 2019
- 稿件说明：
移动端阅览
Yonglin PU, Jiong YU, Liang LU, et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on Communications, 2019, 40(12): 68-85.
DOI：

Yonglin PU, Jiong YU, Liang LU, et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on Communications, 2019, 40(12): 68-85. DOI： 10.11959/j.issn.1000-436x.2019226.

摘要

针对Storm存在低效率、高能耗的问题，通过分析Storm平台的基本框架与拓扑结构，设计了资源约束模型、最优线程数据重组原则和节点降压原则，并在此基础上提出了基于 Storm 平台的数据迁移合并节能策略（DMM-Storm），包括资源约束算法、数据迁移合并算法和节点降压算法。其中资源约束算法根据资源约束模型，判断工作节点是否允许数据的迁移；数据迁移合并算法根据最优线程数据重组原则，设计了最优的线程数据迁移方法；节点降压算法根据节点降压限制条件，降低了工作节点的电压。实验结果表明，与现有的节能策略相比，执行DMM-Storm在不影响集群性能的前提下，有效降低了能耗。

Abstract

Storm is suffering the problems of high energy consumption but low efficiency.Aiming at this problem

the resource constraint model

the optimal principle of data reorganization in executors and node voltage reduction principle were proposed based on the analysis of the architecture and topology of Storm

and further the energy-efficient strategy for data migration and merging was put forward in Storm(DMM-Storm)

which was composed of resource constraint algorithm

data migration and merging algorithm as well as node voltage reduction algorithm.The resource constraint algorithm estimates whether work nodes are appropriate for data migration according to the resource constraint model.The data migration and merging algorithm designs an optimal method to migrate data according to the the optimal principle of data reorganization in executors.The node voltage reduction algorithm reduces voltage of work nodes according to node voltage reduction principle.The experimental results show that the DMM-Storm can reduce energy consumption efficiently without affecting the performance of cluster compared with the existing researches.

关键词

Keywords

references

孟小峰 , 慈祥 . 大数据管理:概念、技术与挑战 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 146 - 169 .

MENG X F , CI X . Big data management:concepts,techniques and challenges [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 146 - 169 .

RANJAN R . Streaming big data processing in datacenter clouds [J ] . IEEE Cloud Computing , 2014 , 1 ( 1 ): 78 - 83 .

CHEN C L P , ZHANG C Y . Data-intensive applications,challenges,techniques and technologies:a survey on big data [J ] . Information Sciences , 2014 , 275 ( 11 ): 314 - 347 .

孙大为 . 大数据流式计算:应用特征和技术挑战 [J ] . 大数据 , 2015 , 1 ( 3 ): 99 - 105 .

SUN D W . Big data stream computing:features and challenges [J ] . Big Data Research , 2015 , 1 ( 3 ): 99 - 105 .

KAMBATLA K , KOLLIAS G , KUMAR V , et al . Trends in big data analytics [J ] . Journal of Parallel and Distributed Computing , 2014 , 74 ( 7 ): 2561 - 2573 .

杨挺 , 王萌 , 张亚健 , 等 . 云计算数据中心 HDFS 差异性存储节能优化算法 [J ] . 计算机学报 , 2019 ( 4 ): 721 - 735 .

YANG T , WANG M , ZHANG Y J , et al . HDFS differential storage energy-saving optimal algorithm in cloud data center [J ] . Chinese Journal of Computers , 2019 ( 4 ): 721 - 735 .

余晓晖 , . 数据中心能效测评指南 [R ] . “云计算发展与政策论坛”技术报告,(2012-03-16) [2019-07-04 ] .

YU X H . Data center energy efficiency assessment guide [R ] . Cloud Computing Development and Policy Forum Technical Report,(2012-03-16) [2019-07-04 ] .

陈小燕 , 干丽萍 , 郭文平 . 大数据可视化工具比较及应用 [J ] . 计算机教育 , 2018 , 282 ( 6 ): 100 - 105 .

CHEN X Y , GAN L P , GUO W P . Comparison and application of big data visualization tools [J ] . Computer Education , 2018 , 282 ( 6 ): 100 - 105 .

SUN D , ZHANG G , YANG S , et al . Re-Stream:real-time and energy-efficient resource scheduling in big data stream computing environments [J ] . Information Sciences , 2015 , 319 : 92 - 112 .

鲁亮 , 于炯 , 卞琛 , 等 . 大数据流式计算框架 Storm 的任务迁移策略 [J ] . 计算机研究与发展 , 2018 , 55 ( 1 ): 71 - 92 .

LU L , YU J , BIAN C , et al . A task migration strategy in big data stream computing with Storm [J ] . Journal of Computer Research and Development , 2018 , 55 ( 1 ): 71 - 92 .

BORTHAKUR D , GRAY J , SARMA J S , et al . Apache Hadoop goes realtime at Facebook [C ] // The 2011 ACM SIGMOD International Conference on Management of Data . ACM , 2011 : 1071 - 1080 .

NEUMEYER L , ROBBINS B , NAIR A , KESARI A . S4:distributed stream computing platform [C ] // The 10th IEEE International Conference on Data Mining Workshops (ICDMW 2010) . IEEE , 2010 : 170 - 177 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的Flink平台弹性资源调度策略 [J ] . 通信学报 , 2019 , 40 ( 8 ): 85 - 101 .

LI Z Y , YU J , BIAN C , et al . Flow-network based auto rescale strategy for Flink [J ] . Journal on Communications , 2019 , 40 ( 8 ): 85 - 101 .

卞琛 , 于炯 , 修位蓉 , 等 . 基于分配适应度的 Spark 渐进填充分区映射算法 [J ] . 通信学报 , 2017 , 38 ( 9 ): 133 - 147 .

BIAN C , YU J , XIU W R , et al . Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree [J ] . Journal on Communications , 2017 , 38 ( 9 ): 133 - 147 .

KULKARNI S , BHAGAT N , FU M , et al . Twitter heron:Stream processing at scale [C ] // The 2015 ACM SIGMOD International Conference on Management of Data . ACM , 2015 : 239 - 250 .

ANDERSON Q . Storm real-time processing cookbook [M ] . Birmingham : Packt PublishingPress , 2013 : 4 - 8 .

TA V D , LIU C M , NKABINDE G W . Big data stream computing in healthcare real-time analytics [C ] // The 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA) . IEEE , 2016 : 37 - 42 .

MISHNE G , DALTON J , LI Z , et al . Fast data in the era of big data:Twitter’s real-time related query suggestion architecture [C ] // The 2013 ACM SIGMOD International Conference on Management of Data . ACM , 2013 : 1147 - 1158 .

DING W L , HAN Y B , ZHAO Z F , et al . Stream-oriented availability services for endpoint-to-endpoint data transmission [C ] // The 2012 In ternational Conference on Cloud and Service Computing . IEEE , 2012 : 212 - 218 .

SHIN D J , PARK S K , KIM S M , et al . Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory [C ] // ACM Research in Applied Computation Symposium . ACM , 2012 : 395 - 402 .

BONAMY R , BILAVARN S , MULLER F . An energy-aware scheduler for dynamically reconfigurable multi-core systems [C ] // International Symposium on Reconfigurable Communication-Centric Systems-On-Chip . IEEE , 2015 : 1 - 6 .

KIM H S , SHIN D I , YU Y J , et al . Towards energy proportional cloud for data processing frameworks [M ] . San Jose : USENIX AssociationPress , 2010 : 1 - 8 .

FAISAL S M , TZIANTZIOULIS G , GOK A M , et al . Edge importance identification for energy efficient graph processing [C ] // IEEE International Conference on Big Data . IEEE , 2015 : 347 - 354 .

SONG J , MA Z , THOMAS R , et al . Energy efficiency optimization in big data processing platform by improving resources utilization [J ] . Sustainable Computing:Informatics and Systems , 2019 , 21 : 80 - 89 .

MU J , PEI Y , LI W , et al . Research on energy saving optimization strategy of substation operation based on big data technology [C ] // 2018 Chinese Control And Decision Conference (CCDC) . IEEE , 2018 : 3567 - 3571 .

DE MATTEIS T , MENCAGLI G . Keep calm and react with foresight:strategies for low-latency and energy-efficient elastic data stream processing [J ] . Journal of Systems and Software , 2016 , 51 ( 8 ): 1 - 12 .

LEVERICH J , KOZYRAKIS C . On the energy (in) efficiency of Hadoop clusters [J ] . ACM SIGOPS Operating Systems Review , 2010 , 44 ( 1 ): 61 - 65 .

LANG W , PATEL J M . Energy management for MapReduce clusters [J ] . Proceedings of the VLDB Endowment , 2010 , 3 ( 1-2 ): 129 - 139 .

宋杰 , 李甜甜 , 朱志良 , 等 . 云数据管理系统能耗基准测试与分析 [J ] . 计算机学报 , 2017 , 36 ( 7 ): 1485 - 1499 .

SONG J , LI T T , ZHU Z L , et al . Benchmarking and analyzing the energy consumption of cloud data management system [J ] . Chinese Journal of Computers , 2013 , 36 ( 7 ): 1485 - 1499 .

廖彬 , 张陶 , 于炯 , 等 . MapReduce 能耗建模及优化分析 [J ] . 计算机研究与发展 , 2016 , 53 ( 9 ): 2107 - 2131 .

LIAO B , ZHANG T , YU J , et al . Energy consumption modeling and optimization analysis for MapReduce [J ] . Journal of Computer Research and Development , 2016 , 53 ( 9 ): 2107 - 2131 .

LIAO B , YU J , ZHANG T , et al . Energy-efficient algorithms for distributed storage system based on block storage structure reconfiguration [J ] . Journal of Computer Research ＆ Development , 2015 , 48 ( 2 ): 71 - 86 .

SHIN D J , PARK S K , KIM S M , et al . Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory [C ] // ACM Research in Applied Computation Symposium . ACM , 2012 : 395 - 402 .

ZHOU S , CHELMIS C , PRASANNA V K . High-Throughput and Energy-Efficient Graph Processing on FPGA [C ] // International Symposium on Field-Programmable Custom Computing Machines . IEEE , 2016 : 103 - 110 .

廖彬 , 张陶 , 于炯 , 等 . 温度感知的MapReduce节能任务调度策略 [J ] . 通信学报 , 2016 , 37 ( 1 ): 61 - 75 .

LIAO B , ZHANG T , YU J . Temperature aware energy-efficient task scheduling strategies for MapReduce [J ] . Journal on Communications , 2016 , 37 ( 1 ): 61 - 75 .

VASUDEVAN V , FRANKLIN J , ANDERSEN D . FAWN damentally power-efficient clusters [C ] // The 12th Workshop on Hot Topics in Operating Systems (HotOS 09Ү) . Usenix Association , 2009 : 1 - 5 .

廖彬 , 于炯 , 孙华 , 等 . 基于存储结构重配置的分布式存储系统节能算法 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 3 - 18 .

LIAO B , YU J , SUN H , et al . Energy-efficient algorithms for distributed storage system based on data storage structure reconfiguration [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 3 - 18 .

GUO B , YU J , LIAO B , et al . A green framework for DBMS based on energy-aware query optimization and energy-efficient query processing [J ] . Journal of Network and Computer Applications , 2017 , 84 : 118 - 130 .

WANG Z , WANG H , ZHAO W , et al . Energy optimization of parallel programs in a heterogeneous system by combining processor core-shutdown and dynamic voltage scaling [J ] . Future Generation Computer Systems , 2019 , 92 : 198 - 209 .

CORDESCHI N , SHOJAFAR M , AMENDOLA D , et al . Energy-efficient adaptive networked datacenters for the QoS support of real-time applications [J ] . The Journal of Supercomputing , 2014 , 71 ( 2 ): 448 - 478 .

PANDA A , CHATHA K S . An embedded architecture for energy-efficient stream computing [J ] . IEEE Embedded Systems Letters , 2014 , 6 ( 3 ): 57 - 60 .

ZONG Z , MANZANARES A , RUAN X , et al . EAD and PEBD:two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters [J ] . IEEE Transactions on Computers , 2010 , 60 ( 3 ): 360 - 374 .

蒲勇霖 , 于炯 , 鲁亮 , 等 . Storm平台下工作节点的内存电压调控节能策略 [J ] . 通信学报 , 2018 , 39 ( 10 ): 101 - 121 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for work node by DRAM voltage regulation in Storm [J ] . Journal on Communications , 2018 , 39 ( 10 ): 101 - 121 .

Views

2077

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Energy-efficient strategy for work node by DRAM voltage regulation in storm

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

Storage resource scheduling optimization method for separated data center based on deep reinforcement learning

Joint optimization strategy of age of information and energy consumption for offloading and scheduling in WBAN

Joint scheduling algorithm for correlative tasks in multi-access edge computing

Related Author

Yonglin PU

Jiong YU

Liang LU

Chen BIAN

Ziyang LI

Yonglin PU

Xiaolong XU

Jiong YU

Related Institution

Xinjiang Runwu Network Limited Company

School of Software, Xinjiang University

School of Computer Engineering, Hubei University of Arts and Science, Xiangyang

School of Software, Nanjing University of Information Science and Technology

cole de Technologie Supérieure (ETS), University of Quebec

AI问答

⁰