温度感知的MapReduce节能任务调度策略

廖彬; 张陶; 于炯; 刘继; 尹路通; 郭刚

doi:10.11959/j.issn.1000-436x.2016008

您当前的位置：

首页 >

文章列表页 >

温度感知的MapReduce节能任务调度策略

学术论文 | 更新时间：2024-06-05

- 温度感知的MapReduce节能任务调度策略
- Temperature aware energy-efficient task scheduling strategies for mapreduce
- 通信学报 2016年37卷第1期页码：61-75
- 作者机构：
  
  1. 新疆财经大学统计与信息学院，新疆乌鲁木齐 830012
  2. 新疆医科大学医学工程技术学院，新疆乌鲁木齐 830011
  3. 新疆大学软件学院，新疆乌鲁木齐 830008
- 作者简介：
  
  [ "廖彬（1986-），男，四川内江人，博士，新疆财经大学副教授，主要研究方向为绿色计算、数据库系统理论及数据挖掘等。" ]
  [ "张陶（1986-），女，新疆乌鲁木齐人，新疆医科大学讲师，主要研究方向为云计算、医学大数据分析等。" ]
  [ "于炯（1964-），男，北京人，博士，新疆大学教授、博士生导师，主要研究方向为网格计算、大数据与云计算等。" ]
  [ "刘继（1974-），男，新疆乌鲁木齐人，博士，新疆财经大学教授，主要研究方向为信息管理及数据挖掘。" ]
  [ "尹路通（1992-），男，河南信阳人，新疆大学硕士生，主要研究方向为节能计算、大数据计算模式等。" ]
  [ "郭刚（1990-），男，新疆乌鲁木齐人，新疆大学硕士生，主要研究方向为节能计算、大数据计算模式等。" ]
- 基金信息：
  
  国家自然科学基金资助项目(61562078);国家自然科学基金资助项目(61262088);国家自然科学基金资助项目(71261025);新疆财经大学博士科研启动基金资助项目(2015BS007)
- DOI：10.11959/j.issn.1000-436x.2016008
  中图分类号： TP393.09
- 网络首发：2016-01，
  
  纸质出版：2016-01-25
- 稿件说明：
移动端阅览
廖彬, 张陶, 于炯, 等. 温度感知的MapReduce节能任务调度策略[J]. 通信学报, 2016,37(1):61-75.

Bin LIAO, Tao ZHANG, Jiong YU, et al. Temperature aware energy-efficient task scheduling strategies for mapreduce[J]. Journal on Communications, 2016, 37(1): 61-75.
廖彬, 张陶, 于炯, 等. 温度感知的MapReduce节能任务调度策略[J]. 通信学报, 2016,37(1):61-75. DOI： 10.11959/j.issn.1000-436x.2016008.

Bin LIAO, Tao ZHANG, Jiong YU, et al. Temperature aware energy-efficient task scheduling strategies for mapreduce[J]. Journal on Communications, 2016, 37(1): 61-75. DOI： 10.11959/j.issn.1000-436x.2016008.

摘要

现有的FIFO、Fair、Capacity、LATE及Deadline Constraint等MapReduce任务调度器的主要区别在于队列与作业选择策略的不同，而任务选择策略基本相同，都是将数据的本地性(data-locality)作为选择的主要因素，忽略了对TaskTracker 当前温度状态的考虑。实验表明，当TaskTracker处于高温状态时，一方面使CPU利用率变高，导致节点能耗增大，任务处理速度下降，导致任务完成时间增加；另一方面，易发的宕机现象将直接导致任务的失败，推测执行(speculative execution)机制容易使运行时任务被迫中止。继而提出温度感知的节能任务调度策略，将节点 CPU 温度纳入任务调度的决策信息，以避免少数高温任务执行节点对作业整体进度的影响。实验结果表明，算法能够避免任务分配到高温节点，从而有效地缩短作业完成时间，减小作业执行能耗，提高系统稳定性。

Abstract

The main difference among the existing MapReduce task chedulers such as FIFO

Fair

Capacity

LATE and Deadline Constraint is their choice of operation strat ue and job. On the count of the task selection strategies of these task schedulers are basically the same

taking the data-locality as the key factor of selection

they all ignore the current state of the temperature of the TaskTracker. The experiments show that when the TaskTracker is in a state of high temperature it will cause some negative results. On one hand

utilization of the CPU becomes higher

which means more energy is consumed at each node. And as a result of task processing speed dropping off

more time will be needed to complete the same task.On the other hand

the prone downtime phenomenon will ectly lead to the failure of the task

and speculative execution mechanism is easy to make the runtime task suspend. Temperature aware energy-efficient task scheduling strategy is put forward to solve the problem. CPU temperature of the node was put into the task scheduling deci-sion-making information to avoid bad impact on the overall ogress of the job form the task execution nodes with a high temperature. The experimental results show that the algorithm can avoid allocating task to high temperature nodes

which ef-fectively shorten the job completion time

reduce energy consumption of job execution and improve system stability.

关键词

Keywords

references

孟小峰，慈祥．大数据管理: 概念、技术与挑战 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 146 - 149 .

MENG X F , CI X . Big data management: concepts, techniques and challenges [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 146 - 149 .

GANTZ J , CHUTE C , MANFREDIZ A , et al . The diverse and ex-ploding digital universe: an updated forecast of worldwide information growth through [EB/OL ] . http://www.ifap.ru/library/book268.pdf http://www.ifap.ru/library/book268.pdf .

Global action plan, an inefficient truth [R/OL ] . Global action plan report , 2007 . http://globalactionplan.org.uk http://globalactionplan.org.uk .

TIMES N Y . Power, pollution and the Internet [EB/OL ] . http://www.nytimes.com/2012/09/23/technology/data-ceneters-waste-vast-amounts-of-energy-belying-industry-image.html http://www.nytimes.com/2012/09/23/technology/data-ceneters-waste-vast-amounts-of-energy-belying-industry-image.html .

BARROSO L A , HLZLE U . The datacenter as a computer: an intro-duction to the design of warehouse-scale machines [R ] . Morgan: Syn-thesis Lectures on Computer Architecture, Morgan ＆ Cla l Pub-lishers , 2009 .

BORTHAKU D . The hadoop distributed file system: architecture and design [J ] . Hadoop Project Website , 2007 , 11 ( 1 ): 1 - 10 .

GHEMAWAT S , GOBIOFF H , LEUNG S T . The google gile sys-tem [C ] // 19th ACM Symposium on Operating System Principles . New York, ACM , c 2003 : 29 - 43 .

DEAN J , GHEMAWAT S . MapReduce: simplifed data processing on large clusters [C ] // The Conference on Operating System Design and Implementation (OSDI). New York, ACM , c 2004 : 137 - 150 .

王鹏，孟丹，詹剑锋，等．数据密集型计算编程模型研究进展 [J ] . 计算机研究与发展 , 2010 , 47 ( 11 ): 1993 - 2002 .

WANG P , MENG D , ZHAN JF , et al . Review of programming models for data-Intensive computing [J ] . Journal of Computer Research a Development , 2010 , 47 ( 11 ): 1993 - 2002 .

LI D , WANG J E . Energy efficient redundant and inexpensive disk array [C ] // The ACM SIGOPS European Workshop . New York, ACM , c 2004 : 29 - 35 .

林闯，田源，姚敏．绿色网络和绿色评价: 节能机制、模型和评价 [J ] . 计算机学报 , 2011 , 34 ( 4 ): 593 - 612 .

LIN C , TIAN Y , YAO M . Green network and green evaluation: Me-chanism, modeling and evaluation [J ] . Chinese Journal f Computers , 2011 , 34 ( 4 ): 593 - 612 .

廖彬，于炯，张陶 , 等．基于分布式文件系统 HDFS 的节能算法 [J ] . 计算机学报 , 2013 , 36 ( 5 ): 1047 - 1064 .

LIAO B , YU J , ZHANG T , et al . Energy-efficient algorithms for dis-tributed file system HDFS [J ] . Chinese Journal of Computers , 2013 , 36 ( 5 ): 1047 - 1064 .

ALBERS S . Energy-efficient algorithms [J ] . Communications of the ACM , 2010 , 53 ( 5 ): 86 - 96 .

WIERMAN A , ANDREW L L , TANG A . Power-aware speed scaling in processor sharing systems [C ] // The 28th Conference on Computer Communications (INFOCOM 2009), Piscataway, NJ , c 2009 : 2007 - 2015 .

ANDREW L L , LIN M , WIERMAN A . Optimality, fairness, and robustness in speed scaling designs [C ] // ACM International Confe-rence on Measurement and Modeling of International Computer Sys-tems (SIGMETRICS 2010), New York,ACM , c 2010 : 37 - 48 .

NEUGEBAUER R , MCAULEY D . Energy is just another resource:energy accounting and energy pricing in the nemesis OS [C ] // The 8th IEEE Workshop on Hot Topics in Operating Systems . Piscataway, NJ , c 2001 : 59 - 64 .

FLINN J , SATYANARAYANAN M . Managing battery lifetime w h energy-aware adaptation [J ] . ACM Transactions on Computer Systems (TOCS), 2004 , 22 ( 2 ): 179 - 182 .

MEISNER D , GOLD B T , WENISCH T F . PowerNap: eliminatin server idle power [J ] . ACM SIGPLAN Notices , 2009 , 44 ( 3 ): 205 - 216 .

YE K , JIANG X , YE D , et al . Two optimization mechanisms to im-prove the isolation property of server consolidation i virtualized mul-ti-core server [C ] // The 12th IEEE International Conference on High Performance Computing and Communications . Melbourne, Australia , c 2010 : 281 - 288 .

CHOI J , GOVINDAN S , JEONG J , et al . Power consumption predic-tion and power-aware packing in consolidated environments [J ] . IEEE Transactions on Computers , c 2010 , 59 ( 12 ): 1640 - 1654 .

YE K , JIANG X , HUANG D , et al . Live migration of multiple virtual machines with resource reservation in cloud computing environ-ments [C ] // The 4th IEEE International Conference on Cloud Compu-ting . Washington,USA , c 2011 : 267 - 274 .

LIAO X , JIN H , LIU H . Towards a green cluster through ynamic remapping of virtual machines [J ] . Future Generation Co ter Sys-tems , 2012 , 28 ( 2 ): 469 - 477 .

JANG J W , JEON M , KIM H S , et al . Energy reduction in consolidat-ed servers through memory-aware virtual machine scheduling [J ] . IEEE Transactions on Computers , 2011 , 99 ( 1 ): 552 - 564 .

WANG X , WANG Y . Coordinating power control and performance management for virtualized server cluster [J ] . IEEE Tra ions on Parallel and Distributed Systems , 2011 , 22 ( 2 ): 245 - 259 .

WANG Y , WNAG X , CHEN M , et al . Partic: power-aware response time control for virtualized web servers [J ] . IEEE Transactions on Pa-rallel and Distributed Systems , 2011 , 22 ( 2 ): 323 - 336 .

DASGUPTA G , SHARMA A , VERMA A , et al . Workload manage-ment for power efficiency in virtualized data-centers [J ] . Communica-tions of the ACM , 2011 , 54 ( 7 ): 131 - 141 .

SRIKANTAIAH S , KANSAL A , ZHAO F . Energy aware consolida-tion for cloud computing [J ] . Cluster Computing , 2009 , 12 ( 1 ): 1 - 15 .

GARG S K , YEO C S , ANANDASIVAM A , et al . Environ-ment-conscious scheduling of HPC applications on distributed cloud-oriented data centers [J ] . Journal of Parallel and Distributed Computing , 2010 , 71 ( 6 ): 732 - 749 .

KUSIC D , KEPHART J O , HANSON J E , et al . Power and perfor-mance management of virtualized computing environments ia loo-kahead control [J ] . Cluster Computing , 2009 , 12 ( 1 ): 1 - 15 .

SONG Y , WANG H , LI Y , et al . Multi-tiered on-demand resource. scheduling for VM-based data center [C ] // Proceedings of the 9th IEEE/ACM International Symposium on Cluster Computing nd the Grid (CCGrid 2009). Piscataway, NJ , c 2009 : 148 - 155 .

GMACH D , ROLIA J , CHERKASOVA L , et al . resource pool man-agement: Reactive versus proactive or let's be friends [J ] . Computer Networks , 2009 , 53 ( 17 ): 2905 - 2922 .

BUYYA R , BELOGLAZOV A , ABAWAJY J . Energy-efficient man-agement of data center resources for cloud computing: a vision, archi-tectural elements, and open challenges [J ] . Eprint Arxiv , 2010 , 12 ( 4 ): 6 - 17 .

KIM K H , BELOGLAZOV A , BUYYA R . Power-aware provisioning of cloud resources for real-time services [C ] // The 7th International Work-shop on Middleware for Grids . New York , c 2009 : 1 - 6 .

王意洁，孙伟东，周松 , 等．云计算环境下分布存储关键技术 [J ] . 软件学报 , 2012 , 23 ( 4 ): 962 - 986 .

WANG Y J , SUN W D , ZHOU S , et al . Key technologies of distri-buted storage for cloud computing [J ] . Journal of Software , 2012 , 23 ( 4 ): 962 - 986 .

GREENAN K H , LONG D D E , MILLER E L , et al . A spin-up saved is energy earned: achieving power-efficient, erasure-coded sto-rage [C ] // The HotDep 2008. Berkeley: USENIX Association , c 2008 : 4 .

WEDDLE C , OLDHAM M , QIAN J , et al . A gear-shifting pow-er-aware raid [J ] . ACM Transactions on Storage , 2007 , 3 ( 3 ): 1553 - 1569 .

LI D , WANG J . Conserving energy in conventional disk based RAID systems [C ] // The 3rd Int Workshop on Storage Network Architecture and Parallel I/Os (SNAPI 2005). Piscataway, NJ , c 2005 : 65 - 72 .

YAO X , WANG J . Rimac: a novel redundancy-based hierarchical cache architecture for energy efficient, high performa storage sys-tems [C ] // The EuroSys . New York,ACM , c 2006 : 249 - 262 .

PINHEIRO E , BIANCHINI R , DUBNICKI C . Exploiting redundancy to conserve energy in storage systems [C ] // The SIGMetrics Perfor-mance 2006 . New York,ACM , c 2006 : 15 - 26 .

NARAYANAN D , DONNELLY A , ROWSTRON A . Write off-loading:practical power management for enterprise storage [J ] . ACM Transac-tions on Storage (TOS), 2008 , 4 ( 3 ): 253 - 267 .

STORER M , GREENAN K , MILLER E , et al . Replacing tape with energy efficient, reliable, disk-based archival storage [C ] // The FAST 2008 . New York,ACM , c 2008 : 1 - 16 .

ZHU Q , CHEN Z , TAN L , et al . Hibernator: Helping disk arrays sleep through the winter [C ] // The 20th ACM Symposium on Operating Sys-tems Principles (SOSP). New York,ACM , c 2005 : 177 - 190 .

VASIC N , BARISITS M , SALZGEBER V . Making cluster applications energy-aware [C ] // The ACDC 2009 . New York,ACM , c 2009 : 37 - 42 .

廖彬，于炯，孙华，等．基于存储结构重配置的分布式存储系统节能算法 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 3 - 18 .

LIAO B , YU J , SUN H , et al . Energy-efficient algorithms for distri-buted storage system based on data storage structure reconfiguration [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 3 - 18 .

LIAO B , YU J , SUN H , et al . A QoS-aware dynamic data replica deletion strategy for distributed storage systems under cloud compu-ting environments [C ] // Cloud and Green Computing (CGC), 2012 Second International Conference. IEEE . c 2012 : 219 - 225 .

廖彬，于炯，钱育蓉，等．基于可用性度量的分布式文件系统节点失效恢复算法 [J ] . 计算机科学 , 2013 , 40 ( 1 ): 144 - 149 .

LIAO B , YU J , QIAN Y R , et al . The node failure recovery algorithm for distributed file system based on measurement of data availabili-ty [J ] . Computer Sicence , 2013 , 40 ( 1 ): 144 - 149 .

ZHU Q , DAVID F M , DEVARAJ C F , et al . Reducing energy con-sumption of disk storage using power-aware cache manage-ment [C ] // The HPCA 2004 . Piscataway,NJ , c 2004 : 118 - 129 .

LEVERICH J , KOZYRAKIS C . On the energy (in)efficiency f ha-doop clusters [J ] . ACM SIGOPS Operating Systems Review , 2010 , 44 ( 1 ): 61 - 65 .

LANG W , PATEL J M . Energy management for mapreduce clusters [J ] . Proceedings of the VLDB Endowment , 2010 , 3 ( 1 - 2 ): 129 - 139 .

CHEN Y , KEYS L , KATZ R H . Towards energy effcient mapre-duce [R ] . Berkeley: EECS Department, University of California , 2009 - 10 - 9 .

WIRTZ T , GE R . Improving MapReduce energy efficiency for com-putation intensive workloads [C ] // Green Computing Conference and Workshops (IGCC). IEEE , c 2011 : 1 - 8 .

GOIRI Í , LE K , NGUYEN T D , et al . GreenHadoop: leveraging green energy in data-processing frameworks [C ] // The 7th ACM european con-ference on Computer Systems. ACM , c 2012 : 57 - 70 .

CARDOSA M , SINGH A , PUCHA H , et al . Exploiting Spa-tio-Temporal Tradeoffs for Energy Efficient MapReduce in t Cloud [D ] . Department of Computer Science and Engineering, Univer-sity of Minnesota , 2010 .

CHEN Y , GANAPATHI A , KATZ R H . To compress or not to com-press-compute vs. IO tradeoffs for mapreduce energy efficien-cy [C ] // The First ACM SIGCOMM Workshop on Green Networking . New Delhi, India , c 2010 : 23 - 28 .

宋杰，李甜甜，朱志良，等．云数据管理系统能耗基准测试与分析 [J ] . 计算机学报 , 2013 , 36 ( 7 ): 1485 - 1499 .

SONG J , LI T T , ZHU Z L , et al . Benchmarking and analyzing the energy consumption of cloud data management system [J ] . Chinese Journal of Computers , 2013 , 36 ( 7 ): 1485 - 1492 .

ANDREWS M , ANTA A F , ZHANG L , et al . Routing for energy minimization in the speed scaling model [C ] // The 29th IEEE Confe-rence on Computer Communications (INFOCOM'10). San Diego, USA , c 2010 : 1 - 9 .

BARROSO L A , HOLZLE U . The case for energy-proportional com-puting [J ] . Computer , 2007 , 40 ( 12 ): 33 - 37 .

林闯，田源，姚敏．绿色网络和绿色评价:节能机制、模型和评价 [J ] . 计算机学报 , 2011 , 34 ( 4 ): 593 - 612 .

LIN C , TIAN Y , YAO M . Green network and green evaluation: Me-chanism, modeling and evaluation [J ] . Chinese Journal of Computers , 2011 , 34 ( 4 ): 593 - 612 .

浏览量

1710

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向绿色计算的算网能一体化协同任务调度机制研究

基于改进量子遗传和QoS感知方法的车联网云雾计算系统任务调度策略

边缘辅助实时应用中信息年龄感知的任务调度

算力网络中面向计算重用的任务调度优化