Research on a real-time receiving scheme of streaming data

Xiaoyan ZHANG; Zhihao LIU; Xiaofeng DU; Tianbo LU

doi:10.11959/j.issn.1000-436x.2022080

您当前的位置：

首页 >

文章列表页 >

Research on a real-time receiving scheme of streaming data

Papers | 更新时间：2024-06-05

- Research on a real-time receiving scheme of streaming data
- Journal on Communications Vol. 43, Issue 4, Pages: 154-163(2022)
- 作者机构：
  
  北京邮电大学计算机学院（国家示范性软件学院），北京 100876
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62162060)
- DOI：10.11959/j.issn.1000-436x.2022080
  CLC： TP3-0
- Online First：2022-04，
  
  Published：25 April 2022
- 稿件说明：
移动端阅览
Xiaoyan ZHANG, Zhihao LIU, Xiaofeng DU, et al. Research on a real-time receiving scheme of streaming data[J]. Journal on Communications, 2022, 43(4): 154-163.
DOI：

Xiaoyan ZHANG, Zhihao LIU, Xiaofeng DU, et al. Research on a real-time receiving scheme of streaming data[J]. Journal on Communications, 2022, 43(4): 154-163. DOI： 10.11959/j.issn.1000-436x.2022080.

摘要

针对现代数据仓库系统中常见的需接收大量流数据，且其与磁盘上已有的数据做连接后再入库的场景进行了探讨。通过合理设置磁盘分页和应用缓存模块，分散磁盘I/O压力，在已有研究的基础上提出了一种具有更高效率的数据接收方案，并引入一致性哈希函数将其扩展到分布式环境，提出一种应用于分布式环境的D-CACHEJOIN算法。通过理论计算算法的成本模型，并使用服从Zipfian分布的数据进行模拟实验。实验结果表明，在接近现实的实际应用场景下，所提算法拥有比现有算法更高的效率，同时能够快速方便地扩展到分布式环境。

Abstract

Discussing the common scenarios in modern data warehouse systems that need to receive a large amount of streaming data

connect it with the existing data on the disk

and then store it in the warehouse.By rationally setting disk paging and applying cache modules to disperse the disk I/O pressure

a more efficient data receiving scheme was proposed based on the existing research

and a consistent Hash function was introduced and extended to distributed environment and a D-CACHEJOIN algorithm applied to distributed environment was proposed.The cost model of the algorithm was calculated by theory and simulation experiment was performed using data that obey the Zipfian distribution.The experiment results show that the proposed algorithm has higher efficiency than existing algorithms in practical application scenarios close to reality

and can be quickly and easily extended to distributed environments.

关键词

Keywords

references

POLYZOTIS N , SKIADOPOULOS S , VASSILIADIS P , et al . Supporting streaming updates in an active data warehouse [C ] // Proceedings of 2007 IEEE 23rd International Conference on Data Engineering . Piscataway:IEEE Press , 2007 : 476 - 485 .

林子雨 , 林琛 , 冯少荣 , 等 . MESHJOIN*:实时数据仓库环境下的数据流更新算法 [J ] . 计算机科学与探索 , 2010 , 4 ( 10 ): 927 - 939 .

LIN Z Y , LIN C , FENG S R , et al . MESHJOIN*:an algorithm supporting streaming updates in a real-time data warehouse [J ] . Journal of Frontiers of Computer Science ＆ Technology , 2010 , 4 ( 10 ): 927 - 939 .

KIM H J , LEE K H . Semi-stream similarity join processing in a distributed environment [J ] . IEEE Access , 2020 , 8 : 130194 - 130204 .

熊超 . 多路数据流等值连接中独立元素问题的研究 [D ] . 深圳:中国科学院大学(中国科学院深圳先进技术研究院) , 2020 .

XIONG C . The distinct element problem in equi-join for multiple data streams [D ] . Shenzhen:Shenzhen Institutes of Advanced Technology,Chinese Academy of Sciences , 2020 .

魏星贝 , 李陶深 , 许嘉 , 等 . QJoin：质量驱动的乱序数据流连接处理技术 [J ] . 广西科学 , 2020 , 27 ( 3 ): 266 - 275 .

WEI X B , LI T S , XU J , et al . QJoin:quality-driven join processing technique over out-of-order data streams [J ] . Guangxi Sciences , 2020 , 27 ( 3 ): 266 - 275 .

POLYZOTIS N , SKIADOPOULOS S , VASSILIADIS P , et al . Meshing streaming updates with persistent data in an active data warehouse [J ] . IEEE Transactions on Knowledge and Data Engineering , 2008 , 20 ( 7 ): 976 - 991 .

ZULFIKAR A F , LESLIE HENDRIC SPITS WARNAS H , GAOL F L , et al . Query optimization for distributed databases uses a semi-join based approach (SBA) with the SDD-1 algorithm [C ] // Proceedings of 2019 International Conference on Information and Communications Technology (ICOIACT) . Piscataway:IEEE Press , 2019 : 619 - 623 .

VAIDEHI V , DEVI D S . Distributed database management and join of multiple data streams in wireless sensor network using querying techniques [C ] // Proceedings of 2011 International Conference on Recent Trends in Information Technology (ICRTIT) . Piscataway:IEEE Press , 2011 : 583 - 588 .

陈付梅 , 韩德志 , 毕坤 , 等 . 大数据环境下的分布式数据流处理关键技术探析 [J ] . 计算机应用 , 2017 , 37 ( 3 ): 620 - 627 .

CHEN F M , HAN D Z , BI K , et al . Key technologies of distributed data stream processing based on big data [J ] . Journal of Computer Applications , 2017 , 37 ( 3 ): 620 - 627 .

ABDELHAFIZ B M , . Distributed database using sharding database architecture [C ] // Proceedings of 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering . Piscataway:IEEE Press , 2020 : 1 - 17 .

NAEEM M A , DOBBIE G , WEBER G . A lightweight stream-based join with limited resource consumption [C ] // Proceedings of the 14th international conference on Data Warehousing and Knowledge Discovery . Berlin:Springer , 2012 : 431 - 442 .

JEON Y H , LEE K H , KIM H J . Distributed join processing between streaming and stored big data under the micro-batch model [J ] . IEEE Access , 2019 , 7 : 34583 - 34598 .

YUAN J , WANG Y H , CHEN H H , et al . Eunomia:efficiently eliminating abnormal results in distributed stream join systems [C ] // Proceedings of 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS) . Piscataway:IEEE Press , 2021 : 1 - 11 .

狄程 , 杨中国 , 韩燕波 , 等 . 面向流数据的实时处理及服务化系统 [J ] . 重庆大学学报 , 2020 , 43 ( 7 ): 75 - 83 .

DI C , YANG Z G , HAN Y B , et al . View-driven flow data oriented real-time processing and service system [J ] . Journal of Chongqing University , 2020 , 43 ( 7 ): 75 - 83 .

DAVID K , KEHMAN E , LEIGHTON T , et al . Consistent hashing and random trees:distributed caching protocols for relieving hot spots on the world wide web [C ] // Proceedings of the 29th ACM Symposium on Theory of Computing . New York:ACM Press , 1997 : 654 - 663 .

KNUTH D E . The art of computer programming,vol.3:sorting and searching,2nd edition [M ] . Redwood City : Addison Wesley Longman Publishing Co.,Inc. , 1998 .

ARMSTRONG R . The long tail:why the future of business is selling less of more [J ] . Canadian Journal of Communication , 2008 , 33 ( 1 ): 274 - 276 .

WORRELL J , . Real-time model checking:algorithms and complexity [C ] // Proceedings of 2008 15th International Symposium on Temporal Representation and Reasoning . Piscataway:IEEE Press , 2008 :19.

NAEEM M A , BAJWA I S , JAMIL N . A cached-based approach to enrich Stream data with master data [C ] // Proceedings of 2015 Tenth International Conference on Digital Information Management (ICDIM) . Piscataway:IEEE Press , 2015 : 57 - 62 .

NIAN H , CHEN L , XU Y Y , et al . Sequences domain impedance modeling of three-phase grid-connected converter using harmonic transfer matrices [J ] . IEEE Transactions on Energy Conversion , 2018 , 33 ( 2 ): 627 - 638 .

MOTWANI R , VASSILVITSKII S . Distinct values estimators for power law distributions [C ] // Proceedings of the 3rd Workshop on Analytic Algorithmics and Combinatorics (ANALCO) . Philadelphia:Society for Industrial and Applied Mathematics , 2006 : 230 - 237 .

Views

1182

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on the backbone network cache method based on application-awareness

Caching strategy based on node’s importance to community in information-centric networks

Hierarchical data grid resource allocation based on quotient space theory

Survey of cache technology for VoiceXML-based voice platform

Distributed network management system with load balancing

Related Author

Yun-jie LIU

Chun-jing HAN

Ye YANG

Hong-lei LYU

Jing-guo GE

Tong LI

Jun CAI

Shun-zheng YU

Related Institution

Institute of Computing Technology，Chinese Academy of Science

University of Chinese Academy of Sciences

Institute of Information Engineering，Chinese Academy of Science

School of Electronic and Information,Guangdong Polytechnic Normal University

Department of Electronics and Communication Engineering,Sun Yat-Sen University

AI问答

⁰