Improving deep convolutional neural networks with mixed maxout units

Hui-zhen ZHAO; Fu-xian LIU; Long-yue LI; Chang LUO

doi:10.11959/j.issn.1000-436x.2017145

您当前的位置：

首页 >

文章列表页 >

Improving deep convolutional neural networks with mixed maxout units

Papers | 更新时间：2024-06-05

- Improving deep convolutional neural networks with mixed maxout units
- Journal on Communications Vol. 38, Issue 7, Pages: 105-114(2017)
- 作者机构：
  
  空军工程大学防空反导学院，陕西西安 710051
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61601499)
- DOI：10.11959/j.issn.1000-436x.2017145
  CLC： TP391.3
- Online First：2017-07，
  
  Published：25 July 2017
- 稿件说明：
移动端阅览
Hui-zhen ZHAO, Fu-xian LIU, Long-yue LI, et al. Improving deep convolutional neural networks with mixed maxout units[J]. Journal on Communications, 2017, 38(7): 105-114.
DOI：

Hui-zhen ZHAO, Fu-xian LIU, Long-yue LI, et al. Improving deep convolutional neural networks with mixed maxout units[J]. Journal on Communications, 2017, 38(7): 105-114. DOI： 10.11959/j.issn.1000-436x.2017145.

摘要

针对深度卷积神经网络中 maxout 单元非最大特征无法传递、特征图像子空间池化表达能力不足的局限性，提出混合maxout (mixout

mixed maxout)单元。首先，计算相同输入在不同卷积变换下所形成的特征图像子空间的指数概率分布；其次，根据概率分布计算特征图像子空间的期望；最后，利用伯努利分布对子空间的最大值与期望值加权，均衡单元模型。分别构建基于 mixout 单元的简单模型和网中网模型进行实验，结果表明 mixout单元模型性能较好。

Abstract

The maxout units have the problem of not delivering non-max features， resulting in the insufficient of pooling operation over a subspace that is composed of several linear feature mappings

when they are applied in deep convolutional neural networks.The mixed maxout (mixout) units were proposed to deal with this constrain.Firstly

the exponential probability of the feature mappings getting from different linear transformations was computed.Then

the averaging of a subspace of different feature mappings by the exponential probability was computed.Finally

the output was randomly sampled from the max feature and the mean value by the Bernoulli distribution

leading to the better utilizing of model averaging ability of dropout.The simple models and network in network models was built to evaluate the performance of mixout units.The results show that mixout units based models have better performance.

关键词

Keywords

references

周昌令 , 栾兴龙 , 肖建国 . 基于深度学习的域名查询行为向量空间嵌入 [J ] . 通信学报 , 2016 , 37 ( 3 ): 165 - 174 .

ZHOU C L , LUAN X L , XIAO J G . Vector space embedding of DNS query behaviors by deep learning [J ] . Journal on Communications , 2016 , 37 ( 3 ): 165 - 174 .

杨钊 , 陶大鹏 , 张树业 , 等 . 大数据下的基于深度神经网的相似汉字识别 [J ] . 通信学报 , 2014 , 35 ( 9 ): 184 - 189 .

YANG Z , TAO D P , ZHANG S Y , et al . Similar handwritten Chinese character recognition based on deep neural networks with big data [J ] . Journal on Communications , 2014 , 35 ( 9 ): 184 - 189 .

SPRINGENBERG J T , RIEDMILLER M . Improving deep neural networks with probabilistic maxout units [J ] . arXiv preprint arXiv:1312.6116 , 2013 .

HINTON G E , SRIVASTAVA N , KRIZHEVSKY A , et al . Improving neural networks by preventing co-adaptation of feature detectors [J ] . arXiv preprint arXiv:1207.0580 , 2012 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks [C ] // The 26th Annual Conference on Neural Information Processing Systems . 2012 : 1097 - 1105 .

WANG S I , MANNING C D . Fast dropout training [C ] // The 30th International Conference on Machine Learning . 2013 : 118 - 126 .

BA J , FREY B . Adaptive dropout for training deep neural networks [C ] // The Advances in Neural Information Processing Systems . 2013 : 3084 - 3092 .

TOMPSON J , GOROSHIN R , JAIN A , et al . Efficient object localization using convolutional networks [C ] // The IEEE Conference on Computer Vision and Pattern Recognition . 2015 : 648 - 656 .

WAN L , ZEILER M , ZHANG S , et al . Regularization of neural networks using dropconnect [C ] // The 30th International Conference on Machine Learning . 2013 : 1058 - 1066 .

GOODFELLOW I J , WARDE-FARLEY D , MIRZA M , et al . Maxout networks [C ] // The 30th International Conference on Machine Learning . 2013 : 1319 - 1327 .

ZEILER M D , FERGUS R . Stochastic pooling for regularization of deep convolutional neural networks [J ] . arXiv Preprint arXiv:1301.3557 , 2013 .

NAIR V , HINTON G E . Rectified linear units improve restricted Boltzmann machines [C ] // The 27th International Conference on Machine Learning . 2010 : 807 - 814 .

MAAS A L , HANNUN A Y , NG A Y . Rectifier nonlinearities improve neural network acoustic models [C ] // The 30th International Conference on Machine Learning . 2013 ,30(1).

HE K , ZHANG X , REN S , et al . Delving deep into rectifiers:surpassing human-level performance on ImageNet classification [C ] // The IEEE International Conference on Computer Vision . 2015 : 1026 - 1034 .

WANG S . General constructive representations for continuous piecewise-linear functions [J ] . IEEE Transactions on Circuits and Systems I:Regular Papers , 2004 , 51 ( 9 ): 1889 - 1896 .

LI J , WANG X , XU B . Understanding the dropout strategy and analyzing its effectiveness on LVCSR [C ] // 38th IEEE International Conference on Acoustics,Speech,and Signal Processing . 2013 : 7614 - 7618 .

YU D , WANG H , CHEN P , et al . Mixed pooling for convolutional neural networks [C ] // International Conference on Rough Sets and Knowledge Technology . 2014 : 364 - 375 .

LIN M , CHEN Q , YAN S . Network in network [J ] . Journal of Artificial Intelligence Research ， 2011 ， 41 （ 2 ）： 297 - 327 .

KRIZHEVSKY A , HINTON G E . Learning multiple layers of features from tiny images [R ] . Computer Science Department,University of Toronto,Tech . 2009 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [J ] . arxiv Prepraint arXiv:1512.03385 , 2015 .

NETZER Y , WANG T , COATES A , et al . Reading digits in natural images with unsupervised feature learning [R ] . NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011 .

CHANG J R , CHEN Y S . Batch-normalized maxout network in network [J ] . arXiv Preprint arXiv:1511.02583 , 2015 .

JIN X , XU C , FENG J , et al . Deep learning with s-shaped rectified linear activation units [J ] . arXiv Preprint arXiv:1512.07030 , 2015 .

LIAO Z , CARNEIRO G . On the importance of normalisation layers in deep learning with piecewise linear activation units [C ] // The 2016 IEEE Winter Conference on Applications of Computer Vision . 2016 : 1 - 8 .

XU B , WANG N , CHEN T , et al . Empirical evaluation of rectified activations in convolutional network [J ] . arXiv Preprint arXiv:1505.00853 , 2015 .

SRIVASTAVA N , HINTON G , KRIZHEVSKY A , et al . Dropout:a simple way to prevent neural networks from overfitting [J ] . The Journal of Machine Learning Research , 2014 , 15 ( 1 ): 1929 - 1958 .

CLEVERT D A , UNTERTHINER T , HOCHREITER S . Fast and accurate deep network learning by exponential linear units (ELUS) [J ] . arXiv preprint arXiv:1511.07289 , 2015 .

Views

1813

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Road vehicle detection based on improved YOLOv3-SPP algorithm

Research progress of deep learning-based object detection of optical remote sensing image

Semantic segmentation of 3D point cloud based on contextual attention CNN

Radio signal recognition based on image deep learning

Study on traffic scene semantic segmentation method based on convolutional neural network

Related Author

Tao WANG

Hao FENG

Rongxin MI

Lin LI

Zhenxue HE

Yiming FU

Shu WU

Yurong LIAO

Related Institution

School of Information and Communication Engineering, Beijing Information Science ＆Technology University

Key Laboratory of Optoelectronic Testing Technology and Instrument, Ministry of Education, Beijing Information Science ＆Technology University

National Computer Network Emergency Response Technical Team/Coordination Center of China

Hebei Key Laboratory of Agricultural Big Data, Hebei Agricultural University

Department of Electronic and Optical Engineering, Space Engineering University

AI问答

⁰