基于Storm平台的数据迁移合并节能策略

蒲勇霖; 于炯; 鲁亮; 李梓杨; 卞琛; 廖彬

doi:10.11959/j.issn.1000-436x.2019226

您当前的位置：

首页 >

文章列表页 >

基于Storm平台的数据迁移合并节能策略

学术论文 | 更新时间：2024-06-05

- 基于Storm平台的数据迁移合并节能策略
- Energy-efficient strategy for data migration and merging in Storm
- 通信学报 2019年40卷第12期页码：68-85
- 作者机构：
  
  1. 新疆大学信息科学与工程学院，新疆乌鲁木齐 830046
  2. 中国民航大学计算机科学与技术学院，天津 300300
  3. 广东金融学院互联网金融与信息工程学院，广东广州 510521
  4. 新疆财经大学统计与信息学院，新疆乌鲁木齐 830012
- 作者简介：
  
  [ "蒲勇霖（1991- ），男，山东淄博人，新疆大学博士生，主要研究方向为流式计算、绿色计算、内存计算等" ]
  [ "于炯（1964- ），男，新疆乌鲁木齐人，博士，新疆大学教授、博士生导师，主要研究方向为并行计算、分布式系统、绿色计算等" ]
  [ "鲁亮（1990- ），男，天津人，博士，中国民航大学讲师，主要研究方向为分布式系统、内存计算、绿色计算" ]
  [ "李梓杨（1993- ），男，新疆乌鲁木齐人，新疆大学博士生，主要研究方向为流式计算、内存计算等" ]
  [ "卞琛（1981- ），男，江苏南京人，博士，广东金融学院副教授，主要研究方向为分布式系统、内存计算、绿色计算等" ]
  [ "廖彬（1986- ），男，新疆乌鲁木齐人，博士，新疆财经大学副教授、硕士生导师，主要研究方向为分布式系统、数据库理论与技术、绿色计算等" ]
- 基金信息：
  
  国家自然科学基金资助项目(61862060);国家自然科学基金资助项目(61462079);国家自然科学基金资助项目(61562086);国家自然科学基金资助项目(61562078);新疆维吾尔自治区研究生科研创新项目(XJGRI2016028);新疆大学博士生科技创新基金资助项目(XJUBSCX-201902)
- DOI：10.11959/j.issn.1000-436x.2019226
  中图分类号： TP311
- 网络出版日期：2019-12，
  
  纸质出版日期：2019-12-25
- 稿件说明：
移动端阅览
蒲勇霖, 于炯, 鲁亮, 等. 基于Storm平台的数据迁移合并节能策略[J]. 通信学报, 2019,40(12):68-85.

Yonglin PU, Jiong YU, Liang LU, et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on communications, 2019, 40(12): 68-85.
蒲勇霖, 于炯, 鲁亮, 等. 基于Storm平台的数据迁移合并节能策略[J]. 通信学报, 2019,40(12):68-85. DOI： 10.11959/j.issn.1000-436x.2019226.

Yonglin PU, Jiong YU, Liang LU, et al. Energy-efficient strategy for data migration and merging in Storm[J]. Journal on communications, 2019, 40(12): 68-85. DOI： 10.11959/j.issn.1000-436x.2019226.

摘要

针对Storm存在低效率、高能耗的问题，通过分析Storm平台的基本框架与拓扑结构，设计了资源约束模型、最优线程数据重组原则和节点降压原则，并在此基础上提出了基于 Storm 平台的数据迁移合并节能策略（DMM-Storm），包括资源约束算法、数据迁移合并算法和节点降压算法。其中资源约束算法根据资源约束模型，判断工作节点是否允许数据的迁移；数据迁移合并算法根据最优线程数据重组原则，设计了最优的线程数据迁移方法；节点降压算法根据节点降压限制条件，降低了工作节点的电压。实验结果表明，与现有的节能策略相比，执行DMM-Storm在不影响集群性能的前提下，有效降低了能耗。

Abstract

Storm is suffering the problems of high energy consumption but low efficiency.Aiming at this problem

the resource constraint model

the optimal principle of data reorganization in executors and node voltage reduction principle were proposed based on the analysis of the architecture and topology of Storm

and further the energy-efficient strategy for data migration and merging was put forward in Storm(DMM-Storm)

which was composed of resource constraint algorithm

data migration and merging algorithm as well as node voltage reduction algorithm.The resource constraint algorithm estimates whether work nodes are appropriate for data migration according to the resource constraint model.The data migration and merging algorithm designs an optimal method to migrate data according to the the optimal principle of data reorganization in executors.The node voltage reduction algorithm reduces voltage of work nodes according to node voltage reduction principle.The experimental results show that the DMM-Storm can reduce energy consumption efficiently without affecting the performance of cluster compared with the existing researches.

关键词

Keywords

references

孟小峰 , 慈祥 . 大数据管理:概念、技术与挑战 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 146 - 169 .

MENG X F , CI X . Big data management:concepts,techniques and challenges [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 146 - 169 .

RANJAN R . Streaming big data processing in datacenter clouds [J ] . IEEE Cloud Computing , 2014 , 1 ( 1 ): 78 - 83 .

CHEN C L P , ZHANG C Y . Data-intensive applications,challenges,techniques and technologies:a survey on big data [J ] . Information Sciences , 2014 , 275 ( 11 ): 314 - 347 .

孙大为 . 大数据流式计算:应用特征和技术挑战 [J ] . 大数据 , 2015 , 1 ( 3 ): 99 - 105 .

SUN D W . Big data stream computing:features and challenges [J ] . Big Data Research , 2015 , 1 ( 3 ): 99 - 105 .

KAMBATLA K , KOLLIAS G , KUMAR V , et al . Trends in big data analytics [J ] . Journal of Parallel and Distributed Computing , 2014 , 74 ( 7 ): 2561 - 2573 .

杨挺 , 王萌 , 张亚健 , 等 . 云计算数据中心 HDFS 差异性存储节能优化算法 [J ] . 计算机学报 , 2019 ( 4 ): 721 - 735 .

YANG T , WANG M , ZHANG Y J , et al . HDFS differential storage energy-saving optimal algorithm in cloud data center [J ] . Chinese Journal of Computers , 2019 ( 4 ): 721 - 735 .

余晓晖 , . 数据中心能效测评指南 [R ] . “云计算发展与政策论坛”技术报告,(2012-03-16) [2019-07-04 ] .

YU X H . Data center energy efficiency assessment guide [R ] . Cloud Computing Development and Policy Forum Technical Report,(2012-03-16) [2019-07-04 ] .

陈小燕 , 干丽萍 , 郭文平 . 大数据可视化工具比较及应用 [J ] . 计算机教育 , 2018 , 282 ( 6 ): 100 - 105 .

CHEN X Y , GAN L P , GUO W P . Comparison and application of big data visualization tools [J ] . Computer Education , 2018 , 282 ( 6 ): 100 - 105 .

SUN D , ZHANG G , YANG S , et al . Re-Stream:real-time and energy-efficient resource scheduling in big data stream computing environments [J ] . Information Sciences , 2015 , 319 : 92 - 112 .

鲁亮 , 于炯 , 卞琛 , 等 . 大数据流式计算框架 Storm 的任务迁移策略 [J ] . 计算机研究与发展 , 2018 , 55 ( 1 ): 71 - 92 .

LU L , YU J , BIAN C , et al . A task migration strategy in big data stream computing with Storm [J ] . Journal of Computer Research and Development , 2018 , 55 ( 1 ): 71 - 92 .

BORTHAKUR D , GRAY J , SARMA J S , et al . Apache Hadoop goes realtime at Facebook [C ] // The 2011 ACM SIGMOD International Conference on Management of Data . ACM , 2011 : 1071 - 1080 .

NEUMEYER L , ROBBINS B , NAIR A , KESARI A . S4:distributed stream computing platform [C ] // The 10th IEEE International Conference on Data Mining Workshops (ICDMW 2010) . IEEE , 2010 : 170 - 177 .

李梓杨 , 于炯 , 卞琛 , 等 . 基于流网络的Flink平台弹性资源调度策略 [J ] . 通信学报 , 2019 , 40 ( 8 ): 85 - 101 .

LI Z Y , YU J , BIAN C , et al . Flow-network based auto rescale strategy for Flink [J ] . Journal on Communications , 2019 , 40 ( 8 ): 85 - 101 .

卞琛 , 于炯 , 修位蓉 , 等 . 基于分配适应度的 Spark 渐进填充分区映射算法 [J ] . 通信学报 , 2017 , 38 ( 9 ): 133 - 147 .

BIAN C , YU J , XIU W R , et al . Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree [J ] . Journal on Communications , 2017 , 38 ( 9 ): 133 - 147 .

KULKARNI S , BHAGAT N , FU M , et al . Twitter heron:Stream processing at scale [C ] // The 2015 ACM SIGMOD International Conference on Management of Data . ACM , 2015 : 239 - 250 .

ANDERSON Q . Storm real-time processing cookbook [M ] . Birmingham : Packt PublishingPress , 2013 : 4 - 8 .

TA V D , LIU C M , NKABINDE G W . Big data stream computing in healthcare real-time analytics [C ] // The 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA) . IEEE , 2016 : 37 - 42 .

MISHNE G , DALTON J , LI Z , et al . Fast data in the era of big data:Twitter’s real-time related query suggestion architecture [C ] // The 2013 ACM SIGMOD International Conference on Management of Data . ACM , 2013 : 1147 - 1158 .

DING W L , HAN Y B , ZHAO Z F , et al . Stream-oriented availability services for endpoint-to-endpoint data transmission [C ] // The 2012 In ternational Conference on Cloud and Service Computing . IEEE , 2012 : 212 - 218 .

SHIN D J , PARK S K , KIM S M , et al . Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory [C ] // ACM Research in Applied Computation Symposium . ACM , 2012 : 395 - 402 .

BONAMY R , BILAVARN S , MULLER F . An energy-aware scheduler for dynamically reconfigurable multi-core systems [C ] // International Symposium on Reconfigurable Communication-Centric Systems-On-Chip . IEEE , 2015 : 1 - 6 .

KIM H S , SHIN D I , YU Y J , et al . Towards energy proportional cloud for data processing frameworks [M ] . San Jose : USENIX AssociationPress , 2010 : 1 - 8 .

FAISAL S M , TZIANTZIOULIS G , GOK A M , et al . Edge importance identification for energy efficient graph processing [C ] // IEEE International Conference on Big Data . IEEE , 2015 : 347 - 354 .

SONG J , MA Z , THOMAS R , et al . Energy efficiency optimization in big data processing platform by improving resources utilization [J ] . Sustainable Computing:Informatics and Systems , 2019 , 21 : 80 - 89 .

MU J , PEI Y , LI W , et al . Research on energy saving optimization strategy of substation operation based on big data technology [C ] // 2018 Chinese Control And Decision Conference (CCDC) . IEEE , 2018 : 3567 - 3571 .

DE MATTEIS T , MENCAGLI G . Keep calm and react with foresight:strategies for low-latency and energy-efficient elastic data stream processing [J ] . Journal of Systems and Software , 2016 , 51 ( 8 ): 1 - 12 .

LEVERICH J , KOZYRAKIS C . On the energy (in) efficiency of Hadoop clusters [J ] . ACM SIGOPS Operating Systems Review , 2010 , 44 ( 1 ): 61 - 65 .

LANG W , PATEL J M . Energy management for MapReduce clusters [J ] . Proceedings of the VLDB Endowment , 2010 , 3 ( 1-2 ): 129 - 139 .

宋杰 , 李甜甜 , 朱志良 , 等 . 云数据管理系统能耗基准测试与分析 [J ] . 计算机学报 , 2017 , 36 ( 7 ): 1485 - 1499 .

SONG J , LI T T , ZHU Z L , et al . Benchmarking and analyzing the energy consumption of cloud data management system [J ] . Chinese Journal of Computers , 2013 , 36 ( 7 ): 1485 - 1499 .

廖彬 , 张陶 , 于炯 , 等 . MapReduce 能耗建模及优化分析 [J ] . 计算机研究与发展 , 2016 , 53 ( 9 ): 2107 - 2131 .

LIAO B , ZHANG T , YU J , et al . Energy consumption modeling and optimization analysis for MapReduce [J ] . Journal of Computer Research and Development , 2016 , 53 ( 9 ): 2107 - 2131 .

LIAO B , YU J , ZHANG T , et al . Energy-efficient algorithms for distributed storage system based on block storage structure reconfiguration [J ] . Journal of Computer Research ＆ Development , 2015 , 48 ( 2 ): 71 - 86 .

SHIN D J , PARK S K , KIM S M , et al . Adaptive page grouping for energy efficiency in hybrid PRAM-DRAM main memory [C ] // ACM Research in Applied Computation Symposium . ACM , 2012 : 395 - 402 .

ZHOU S , CHELMIS C , PRASANNA V K . High-Throughput and Energy-Efficient Graph Processing on FPGA [C ] // International Symposium on Field-Programmable Custom Computing Machines . IEEE , 2016 : 103 - 110 .

廖彬 , 张陶 , 于炯 , 等 . 温度感知的MapReduce节能任务调度策略 [J ] . 通信学报 , 2016 , 37 ( 1 ): 61 - 75 .

LIAO B , ZHANG T , YU J . Temperature aware energy-efficient task scheduling strategies for MapReduce [J ] . Journal on Communications , 2016 , 37 ( 1 ): 61 - 75 .

VASUDEVAN V , FRANKLIN J , ANDERSEN D . FAWN damentally power-efficient clusters [C ] // The 12th Workshop on Hot Topics in Operating Systems (HotOS 09Ү) . Usenix Association , 2009 : 1 - 5 .

廖彬 , 于炯 , 孙华 , 等 . 基于存储结构重配置的分布式存储系统节能算法 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 3 - 18 .

LIAO B , YU J , SUN H , et al . Energy-efficient algorithms for distributed storage system based on data storage structure reconfiguration [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 3 - 18 .

GUO B , YU J , LIAO B , et al . A green framework for DBMS based on energy-aware query optimization and energy-efficient query processing [J ] . Journal of Network and Computer Applications , 2017 , 84 : 118 - 130 .

WANG Z , WANG H , ZHAO W , et al . Energy optimization of parallel programs in a heterogeneous system by combining processor core-shutdown and dynamic voltage scaling [J ] . Future Generation Computer Systems , 2019 , 92 : 198 - 209 .

CORDESCHI N , SHOJAFAR M , AMENDOLA D , et al . Energy-efficient adaptive networked datacenters for the QoS support of real-time applications [J ] . The Journal of Supercomputing , 2014 , 71 ( 2 ): 448 - 478 .

PANDA A , CHATHA K S . An embedded architecture for energy-efficient stream computing [J ] . IEEE Embedded Systems Letters , 2014 , 6 ( 3 ): 57 - 60 .

ZONG Z , MANZANARES A , RUAN X , et al . EAD and PEBD:two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters [J ] . IEEE Transactions on Computers , 2010 , 60 ( 3 ): 360 - 374 .

蒲勇霖 , 于炯 , 鲁亮 , 等 . Storm平台下工作节点的内存电压调控节能策略 [J ] . 通信学报 , 2018 , 39 ( 10 ): 101 - 121 .

PU Y L , YU J , LU L , et al . Energy-efficient strategy for work node by DRAM voltage regulation in Storm [J ] . Journal on Communications , 2018 , 39 ( 10 ): 101 - 121 .

浏览量

569

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

storm平台下工作节点的内存电压调控节能策略

流式大数据平台下的弹性数据迁移能效优化策略

信息年龄和能耗联合优化的无线体域网卸载和调度策略

多接入边缘计算中相关性任务的联合调度算法

基于信息熵与遗传算法的并行关联规则增量挖掘算法