Compression strategies selection method based on classification of HBase data

Hai-yan WANG; Cai-hang FU

doi:10.11959/j.issn.1000-436x.2016068

您当前的位置：

首页 >

文章列表页 >

Compression strategies selection method based on classification of HBase data

academic paper | 更新时间：2024-06-05

- Compression strategies selection method based on classification of HBase data
- Journal of Communications Vol. 37, Issue 4, Pages: 12-22(2016)
- 作者机构：
  
  1. 南京邮电大学计算机学院，江苏南京210023
  2. 南京邮电大学江苏省无线传感网高技术研究重点实验室，江苏南京210003
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61201163)
- DOI：10.11959/j.issn.1000-436x.2016068
  CLC： TP301.6
- Online First：2016-04，
  
  Published：25 April 2016
- 稿件说明：
移动端阅览
Hai-yan WANG, Cai-hang FU. Compression strategies selection method based on classification of HBase data[J]. Journal of Communications, 2016, 37(4): 12-22.
DOI：

Hai-yan WANG, Cai-hang FU. Compression strategies selection method based on classification of HBase data[J]. Journal of Communications, 2016, 37(4): 12-22. DOI： 10.11959/j.issn.1000-436x.2016068.

摘要

为解决现有的HBase数据压缩策略选择方法未考虑数据的冷热性，以及在选择过程中存在片面性和不可靠性的缺陷，提出了基于HBase数据分类的压缩策略选择方法。依据数据文件的访问频度将HBase数据划分为冷热数据，并限定具体的访问级别；在此基础上增加评估层，综合考虑基于相邻区和统计列的选择方法，提出基于数据访问级别的压缩策略选择方法。仿真实验及结果表明，提出的压缩策略选择方法不仅节省了存储空间，还大大提高了数据查询的性能。

Abstract

Most of the current compression strategies selection methods for HBase data did not consider whether the data was cold or hot. Besides

problem of incompleteness and unreliability existed during selection process. To address the problems above

a compression strategies selection method based on classification of HBase data was put forward. HBase data was classified into cold and hot data according to the access frequency of each data file and an access level would be designated to each file. On this base

an evaluation layer was added and a compression strategies selection method based on access level with integration of neighbor sector and statistic column based selection methods. Simulation experiments and results demonstrate that the proposed compression strategies selection method based on classification of HBase data can not only save storage space but also greatly improve the query performance of HBase data.

关键词

Keywords

references

程学旗，靳小龙，王元卓，等．大数据系统和分析技术综述 [J ] ．软件学报， 2014 , 25 ( 9 ): 1889 - 1908 .

CHENG X Q , JIN X L , WANG Y Z , et al . Survey on big data system and analytic technology [J ] . Journal of Software , 2014 , 25 ( 9 ): 1889 - 1908 .

郭嘉凯．如何存储“冷数据” [J ] ．软件和信息服务， 2013 , 23 ( 10 ): 58 - 59 .

GUO J K . How to store“cloud data” [J ] . Software and Information Service , 2013 , 23 ( 10 ): 58 - 59 .

王振玺，乐嘉锦，王梅，等．列存储数据区级压缩模式与压缩策略选择方法 [J ] ．计算机学报， 2010 , 33 ( 8 ): 1524 - 1530 .

WANG Z X , LE J J , WANG M , et al . Sector-based compression and compression strategy selection method for column stores [J ] . Chinese Journal of Computers , 2010 , 33 ( 8 ): 1524 - 1530 .

TRONDHEIM N , STONEBRAKER M , ABADI D J , et al . C-store-A column-oriented DBMS [C ] // The 31st VLDB Conference . Trondheim, Norway , c 2005 : 553 - 564 .

YAN K , XIE M Y , ZHU H . Fixed-length string compression for direct operations in column-oriented databases [C ] // 2013 Ninth International Conference on Natural Computation (ICNC). Shenyang,China , c 2013 : 1171 - 1176 .

TOMMY S , SANJAY M . On compressing data in wireless sensor networks for energy efficiency and real time delivery [J ] . Distributed and Parallel Databases , 2013 , 31 ( 2 ): 151 - 182 .

MEHLA U S , DASGUPTA K S . Hamming distance based reordering and columnwise bit stuffing with difference vector: a etter scheme for test data compression with run length based codes [C ] // VLSID 23rd In-ternational Conference on VLSI Design . India,Bangalore , c 2010 : 33 - 38 .

丘建平，张广艳，舒继武． DMStone：一个分级存储系统性能测试工具 [J ] ．软件学报， 2012 , 23 ( 4 ): 987 - 995 .

QIU J P , ZHANG G Y , SHU J W . DMStone: a tool for evaluating hie-rarchical storage management systems [J ] . Journal of Software , 2012 , 23 ( 4 ): 987 - 995 .

LEVANDOSKI J J , LARSON P A , STOICA R . Identifying hot and cold data in main-memory databases [C ] // 2013 IEEE 29th International Conference on Data Engineering (ICDE). Australia, Brisbane , c 2013 : 26 - 37 .

GAO H B , WANG D F . LSD2H: a novel storage method of linked sensor data based on HBase [C ] // 2014 10th International Conference on Semantics, Knowledge and Grids (SKG). Beijing, China , c 2014 : 116 - 119 .

朱敏，程佳，柏文阳．一种基于HBase的RDF 数据存储模型 [J ] ．计算机研究与发展， 2013 , 50 ( Suppl. ): 23 - 31 .

ZHU M , CHENG J , BAI W Y . A storage model for RDF data based on HBase [J ] . Journal of Computer Research and Development , 2013 , 50 ( Suppl. ): 23 - 31 .

葛微，罗圣美，周文辉，等． HiBase：一种基于分层式索引的高效HBase查询技术与系统 [C ] // 2014中国大数据技术大会．中国，北京， c 2014

GE W , LUO S M , ZHOU W H , et al . HiBase: a hierarchical ndexing mechanism and system for efficient HBase query [C ] // Big Data Tech-nology Conference 2014 , China,Beijing , c 2014

IDREOSS.Self-organizing tuple reconstruction in column-stores [C ] // Proceedings of the SIGMOD . Providence, Rhode Island,USA , c 2009 : 297 - 308 .

宁正元，王李近．统计与决策常用算法及其实现 [M ] ．北京：清华大学出版社， 2009 : 260 - 345 .

NING Z Y , WANG L J . Statistical and decision algorithm and its rea-lization [M ] . Beijing ： Tsinghua University Press ， 2009 : 260 - 345 .

李光，王亚东，苏小红．隐私保持的决策树分类挖掘 [J ] ．电子学报， 2010 , 38 ( 1 ): 204 - 212 .

LI G , WANG Y D , SU X H . Privacy preserving data mining on deci-sion tree [J ] . Acta Electronica Sinica , 2010 , 38 ( 1 ): 204 - 212 .

崔颖安，李雪，王志晓，等．在线社交媒体数据抽样方法的比较研究 [J ] ．计算机学报． 2014 , 37 ( 8 ): 1859 - 1876 .

CUI Y A , LI X , WANG Z X , et al . A comparison on methodologies of sampling online social media [J ] . Chinese Journal of Computer , 2014 , 37 ( 8 ): 1859 - 1876 .

WEI Y , YONG W F . A Cost-effective and reliable cloud storage [C ] // 2014 IEEE International Conference on Cloud Computing . Anchorage, AK , c 2014 : 938 - 939 .

TPC [EB/OL ] . http://www.tpc.org/tpch/ http://www.tpc.org/tpch/ , 2011 .

龙礡涛．列存储数据仓库中压缩技术的研究与实现 [D ] ．上海：东华大学， 2013 .

LONG B T . Research and implementation of compression technology in column-oriented data warehouse [D ] . Shanghai : Donghua University , 2013 .

NYAMAGWA M , LIU J , UEHARA T . Cloud foren: a novel framework for digital forensics in cloud computing [J ] . Journal of Harbin Institute of Technology , 2014 , 21 ( 6 ): 39 - 45 .

Views

1774

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Communication-efficient distributed precoding design for Massive MIMO

Parallel association rules incremental mining algorithm based on information entropy and genetic algorithm

DTW-based multi-wavelet data compression algorithm for wireless sensor networks

Adaptive multiple-modalities data compression algorithm using wavelet for wireless sensor networks

HitIct:Chinese corpus for the evaluation of lossless compression algorithms

Related Author

Yang LI

Zonghui ZHANG

Qingjiang SHI

Mian LI

Zhigang CHEN

Qianhu DENG

Yimin MAO

Lan LI

Related Institution

School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)

Shenzhen Research Institute of Big Data

Pengcheng Laboratory

Pazhou Laboratory (Huangpu)

School of Software Engineering, Tongji University

AI问答

⁰