Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery

XIN Jie; CUI Zhi-ming; ZHAO Peng-peng; ZHANG Guang-ming; XIAN Xue-feng

您当前的位置：

首页 >

文章列表页 >

Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery

更新时间：2024-10-14

- Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery
- Vol. 32, Issue 7, Pages: 189-195(2011)
- 作者机构：
  
  苏州大学智能信息处理及应用研究所
- 作者简介：
- 基金信息：
- DOI：
  CLC： TP393.09
- Published：2011
- 稿件说明：
移动端阅览
XIN Jie, CUI Zhi-ming, ZHAO Peng-peng, et al. Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery[J]. 2011, 32(7): 189-195.
DOI：

XIN Jie, CUI Zhi-ming, ZHAO Peng-peng, et al. Applying MapReduce frameworks to a virtualization platform for Deep Web data source discovery[J]. 2011, 32(7): 189-195. DOI：

摘要

为了提高Deep Web爬虫发现和搜集数据源的效率

提出了一种融合MapReduce和虚拟化技术实现DeepWeb海量数据挖掘的并行计算方法。基于MapReduce架构提出了一个Deep Web爬虫模型

通过链接过滤分类、页面过滤分类、表单过滤分类等3个MapReduce过程找到Deep Web数据源接口

并利用虚拟机构建单机集群进行性能测试。实验结果显示该方法可以实现大规模数据的并行处理

有效提高爬虫数据源发现的效率

避免网络及物理资源的浪费

验证了云计算技术在Deep Web数据挖掘方面的可行性。

Abstract

In order to improve the performance of Deep Web crawler in discovering and searching data sources interfaces

a new method was raised to parallel processing the mass data within the Deep Web compromising MapReduce program-ming model and virtualization technology.The new crawling architecture was designed with three producers

the link classified MapReduce

the page classified MapReduce and the form classified MapReduce.Server virtualization was adopted to simulate the cluster environment in order to test the performance.Experiment results indicate that this method is capable for large-scale data parallel computing

can improve the crawling efficiency and avoid wasteful expenditure

which prove the feasibility of applying cloudy technologies into Deep Web data mining field.

关键词

Keywords

references

Views

1146

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Stochastic algorithm for HDFS data theft detection based on MapReduce

Stochastic gradient descent algorithm preserving differential privacy in MapReduce framework

Information extraction from massive Web pages based on node property and text content

k-means clustering method preserving differential privacy in MapReduce framework

Temperature aware energy-efficient task scheduling strategies for mapreduce

Related Author

Xingyuan CHEN

Binglong LI

Yuanzhao GAO

Xiaoping WU

Yu FU

Yihan YU

Hai-yan WANG

Pan CAO

Related Institution

Third Academy,Information Engineering University

State Key Laboratory of Cryptology

Department of Information Security, Naval University of Engineering

School of Computer Science and Technology, Nanjing University of Posts and Telecommunications

Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks

AI问答

⁰