基于神经网络的文本表示模型新方法

曾谁飞; 张笑燕; 杜晓峰; 陆天波

doi:10.11959/j.issn.1000-436x.2017088

您当前的位置：

首页 >

文章列表页 >

基于神经网络的文本表示模型新方法

学术论文 | 更新时间：2024-06-05

- 基于神经网络的文本表示模型新方法
- New method of text representation model based on neural network
- 通信学报 2017年38卷第4期页码：86-98
- 作者机构：
  
  1. 北京邮电大学软件学院，北京 100876
  2. 北京邮电大学计算机学院，北京 100876
- 作者简介：
  
  [ "曾谁飞（1978-），男，江西广昌人，北京邮电大学博士生，主要研究方向为智能信息处理、机器学习、深度学习和神经网络等。" ]
  [ "张笑燕（1973-），女，山东烟台人，博士，北京邮电大学教授，主要研究方向为软件工程理论、移动互联网软件、ad hoc和无线传感器网络。" ]
  [ "杜晓峰（1973-），男，陕西韩城人，北京邮电大学讲师，主要研究方向为云计算与大数据分析。" ]
  [ "陆天波（1977-），男，贵州毕节人，博士，北京邮电大学副教授，主要研究方向为网络与信息安全、安全软件工程、P2P计算。" ]
- 基金信息：
- DOI：10.11959/j.issn.1000-436x.2017088
  中图分类号： TP183
- 网络出版日期：2017-04，
  
  纸质出版日期：2017-04-25
- 稿件说明：
移动端阅览
曾谁飞, 张笑燕, 杜晓峰, 等. 基于神经网络的文本表示模型新方法[J]. 通信学报, 2017,38(4):86-98.

Shui-fei ZENG, Xiao-yan ZHANG, Xiao-feng DU, et al. New method of text representation model based on neural network[J]. Journal on communications, 2017, 38(4): 86-98.
曾谁飞, 张笑燕, 杜晓峰, 等. 基于神经网络的文本表示模型新方法[J]. 通信学报, 2017,38(4):86-98. DOI： 10.11959/j.issn.1000-436x.2017088.

Shui-fei ZENG, Xiao-yan ZHANG, Xiao-feng DU, et al. New method of text representation model based on neural network[J]. Journal on communications, 2017, 38(4): 86-98. DOI： 10.11959/j.issn.1000-436x.2017088.

摘要

提出了一种改进的文本表示模型提取文本特征词向量方法。首先构建基于词典索引和所对应的词性索引的double word-embedding列表的word-embedding词向量，其次，利用在此基础上Bi-LSTM循环神经网络对生成后的词向量进一步进行特征提取，最后，通过mean-pooling层处理句子向量后且使用了softmax层进行文本分类。实验验证了Bi-LSTM和double word-embedding神经网络相结合的模型训练效果与提取情况。实验结果表明，该模型不但能较好地处理高质量的文本特征向量提取和表达序列，而且比LSTM、LSTM+context window和Bi-LSTM这3种神经网络有较明显的表达效果。

Abstract

Method of text representation model was proposed to extract word-embedding from text feature.Firstly

the word-embedding of the dual word-embedding list based on dictionary index and the corresponding part of speech index was created.Then

feature vectors was obtained further from these extracted word-embeddings by using Bi-LSTM recurrent neural network.Finally

the sentence vectors were processed by mean-pooling layer and text categorization was classified by softmax layer.The training effects and extraction performance of the combination model of Bi-LSTM and double word-embedding neural network were verified.The experimental results show that this model not only performs well in dealing with the high-quality text feature vector and the expression sequence

but also significantly outperforms other three kinds of neural networks

which includes LSTM

LSTM+context window and Bi-LSTM.

关键词

Keywords

references

XU W D , AULI M , CLARK S . CCG supertagging with a recurrent neural network [C ] // The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers) . 2015 : 250 - 255 .

GEOFFREY E,HINTON , . Learning distributed representations of concepts [C ] // The Eighth Annual Conference of the Cognitive Science Society . Amherst,Mass , 1986 : 1 - 12 .

YAO Y S , HUANG Z . Bi-directional LSTM recurrent neural network for Chinese word segmentation [J ] . arXiv:1602.04874v1[cs.LG] , 2016 .

CHIU J P C , NICHOLS E . Named entity recognition with bidirectional LSTM-CNNs [J ] . Transactions of the Association for Computational Linguistics , 2016 , 4 : 357 - 370 .

HOCHREITER S , SCHMIDHUBER J . Long short-term memory [J ] . Neural Computation , 1997 , 9 ( 8 ): 1735 .

TAN M , XIANG B , ZHOU B W . LSTM-based deep learning models for non-factoid answer selection [C ] // ICLR 2016 .

KIPERWASSER E , GOLDBERG Y . Simple and accurate dependency parsing using Bidirectional LSTM feature representations [J ] . Transactions of the Association for Computational Linguistics , 2016 , 4 : 313 - 327 .

ZEILER M D . ADADELTA:an adaptive learning rate method [J ] . arXir:1212.5701v1[cs.LG] , 2012 .

MIKOLOV T , SUTSKEVER I , CHEN K , et al . Distributed representations of words and phrases and their compositionality [J ] . Advances in Neural Information Processing Systems , 2013 , 26 : 3111 - 3119 .

MAAS A L , DALY R E , PHAM P T , et al . Learning word vectors for sentiment analysis [C ] // The 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies . 2011 : 142 - 150 .

CHO K . Natural language understanding with distributed representation [J ] . Nato Asi , 2015 , 147 : 139 - 155 .

SANTOS C D , TAN M , XIANG B . Attentive pooling networks [J ] . arXir:1602.03609v1[cs.CL] , 2016 .

SEVERYN A , MOSCHITTI A . Modeling relational information in question-answer pairs with convolutional neural networks [J ] . arXiv:1604.01178v1[cs.CL] , 2016 .

CROSS J , HUANG L . Incremental parsing with minimal features using bi-directional LSTM [C ] // The 54th Annual Meeting of the Association for Computational Linguistics . 2016 : 32 - 37 .

TURIAN J , RATINOV L , BENGIO Y . Word representations:a simple and general method for semi-supervised learning [C ] // The 48th Annual Meeting of the Association for Computational Linguistics . 2010 : 384 - 394 .

SUGAWARA H , TAKAMURA H , SASANO R , et al . Context representation with word embeddings for WSD [M ] . Computational Linguistics.Springer . Singapore , 2015 : 108 - 119 .

SUNDERMEYER M,SCHLÜTER R , NEY H . LSTM neural networks for language modeling [J ] . Interspeech , 2012 , 31 ( 43 ): 601 - 608 .

WANG P L , QIAN Y , SOONG F K , et al . A unified tagging solution:bidirectional LSTM recurrent neural network with word embedding [J ] . arXiv:1511.00215v1[cs.CL] , 2015 .

HUANG Z H , XU W , YU K . Bidirectional LSTM-CRF models for sequence tagging [J ] . arXiv:1508.01991v1[cs.CL] , 2015 .

WANG L,LUÍS T , MARUJO L , et al . Finding function in form:compositional character models for open vocabulary word representation [C ] // The 2015 Conference on Empirical Methods in Natural Language Processing . 2016 : 1520 - 1530 .

RAO A , SPASOJEVIC N . Actionable and political text classification using word embeddings and LSTM [J ] . arXiv:1607.02501v1[cs.CL] , 2016 .

浏览量

1541

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于阵列的神经网络水声通信信号多参数联合估计算法

基于神经网络的恶意DNS流量检测方法

基于可见光和射频融合的通信定位一体化系统

基于特征依赖图的源代码漏洞检测方法

基于改进CFCC特征提取的语种识别算法研究