卷积神经网络的损失最小训练后参数量化方法

张帆; 黄赟; 方子茁; 郭威

doi:10.11959/j.issn.1000-436x.2022068

您当前的位置：

首页 >

文章列表页 >

卷积神经网络的损失最小训练后参数量化方法

学术论文 | 更新时间：2024-06-05

- 卷积神经网络的损失最小训练后参数量化方法
- Lost-minimum post-training parameter quantization method for convolutional neural network
- 通信学报 2022年43卷第4期页码：114-122
- 作者机构：
  
  1. 国家数字交换系统工程技术研究中心，河南郑州 450002
  2. 信息工程大学，河南郑州 450001
  3. 紫金山实验室，江苏南京 211111
  4. 东南大学网络空间安全学院，江苏南京 211189
- 作者简介：
  
  [ "张帆（1981- ），男，河南郑州人，博士，国家数字交换系统工程技术研究中心副研究员，主要研究方向为主动防御、人工智能等" ]
  [ "黄赟（1993- ），男，江西新余人，信息工程大学硕士生，主要研究方向为神经网络模型量化压缩、网络内生安全等" ]
  [ "方子茁（1997- ），男，河南郑州人，东南大学硕士生，主要研究方向为网络内生安全、数据库安全、人工智能安全等" ]
  [ "郭威（1990- ），男，北京人，博士，国家数字交换系统工程技术研究中心副研究员，主要研究方向为主动防御、人工智能安全等" ]
- 基金信息：
  
  国家自然科学基金创新群体基金资助项目(61521003)
- DOI：10.11959/j.issn.1000-436x.2022068
  中图分类号： TP391
- 网络首发：2022-04，
  
  纸质出版：2022-04-25
- 稿件说明：
移动端阅览
张帆, 黄赟, 方子茁, 等. 卷积神经网络的损失最小训练后参数量化方法[J]. 通信学报, 2022,43(4):114-122.

Fan ZHANG, Yun HUANG, Zizhuo FANG, et al. Lost-minimum post-training parameter quantization method for convolutional neural network[J]. Journal on Communications, 2022, 43(4): 114-122.
张帆, 黄赟, 方子茁, 等. 卷积神经网络的损失最小训练后参数量化方法[J]. 通信学报, 2022,43(4):114-122. DOI： 10.11959/j.issn.1000-436x.2022068.

Fan ZHANG, Yun HUANG, Zizhuo FANG, et al. Lost-minimum post-training parameter quantization method for convolutional neural network[J]. Journal on Communications, 2022, 43(4): 114-122. DOI： 10.11959/j.issn.1000-436x.2022068.

摘要

针对数据敏感性场景下模型量化存在数据集不可用的问题，提出了一种不需要使用数据集的模型量化方法。首先，依据批归一化层参数及图像数据分布特性，通过误差最小化方法获得模拟输入数据；然后，通过研究数据舍入特性，提出基于损失最小化的因子动态舍入方法。通过对GhostNet等分类模型及M2Det等目标检测模型进行量化实验，验证了所提量化方法对图像分类及目标检测模型的有效性。实验结果表明，所提量化方法能够使模型大小减少75%左右，在基本保持原有模型准确率的同时有效地降低功耗损失、提高运算效率。

Abstract

To solve the problem that that no dataset is available for model quantization in data-sensitive scenarios

a model quantization method without using data sets was proposed.Firstly

according to the parameters of batch normalized layer and the distribution characteristics of image data

the simulated input data was obtained by error minimization method.Then

by studying the characteristics of data rounding

a factor dynamic rounding method based on loss minimization was proposed.Through quantitative experiments on classification models such as GhostNet and target detection models such as M2Det

the effectiveness of the proposed quantification method for image classification and target detection models was verified.The experimental results show that the proposed quantization method can reduce the model size by about 75%

effectively reduce the power loss and improve the computing efficiency while basically maintaining the accuracy of the original model.

关键词

Keywords

references

郭璠 , 张泳祥 , 唐琎 , 等 . YOLOv3-A:基于注意力机制的交通标志检测网络 [J ] . 通信学报 , 2021 , 42 ( 1 ): 87 - 99 .

GUO F , ZHANG Y X , TANG J , et al . YOLOv3-A:a traffic sign detection network based on attention mechanism [J ] . Journal on Communications , 2021 , 42 ( 1 ): 87 - 99 .

黄志清 , 曲志伟 , 张吉 , 等 . 基于深度强化学习的端到端无人驾驶决策 [J ] . 电子学报 , 2020 , 48 ( 9 ): 1711 - 1719 .

HUANG Z Q , QU Z W , ZHANG J , et al . End-to-end autonomous driving decision based on deep reinforcement learning [J ] . Acta Electronica Sinica , 2020 , 48 ( 9 ): 1711 - 1719 .

IOFFE S , SZEGEDY C . Batch normalization:accelerating deep network training by reducing internal covariate shift [C ] // Proceedings of the 32nd International Conference on International Conference on Machine Learning .[S.l. ] : JMLR.org , 2015 : 448 - 456 .

CHOUKROUN Y , KRAVCHIK E , YANG F , et al . Low-bit quantization of neural networks for efficient inference [C ] // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway:IEEE Press , 2019 : 3009 - 3018 .

QIN H T , GONG R H , LIU X L , et al . Forward and backward information retention for accurate binary neural networks [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2020 : 2247 - 2256 .

ESSER S K , MCKINSTRY J L , BABLANI D , et al . Learned step size quantization [J ] . arxiv Preprint,arxiv:1902.08153 , 2019 .

NAGEL M , BAALEN M V , BLANKEVOORT T , et al . Data-free quantization through weight equalization and bias correction [C ] // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway:IEEE Press , 2019 : 1325 - 1334 .

LIU Y A , ZHANG W , WANG J . Zero-shot adversarial quantization [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2021 : 1512 - 1521 .

CAI Y H , YAO Z W , DONG Z , et al . ZeroQ:a novel zero shot quantization framework [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2020 : 13166 - 13175 .

NAGEL M , AMJAD R A , BAALEN M V , et al . Up or down? adaptive rounding for post-training quantization [C ] // Proceedings of 2020 International Conference on Machine Learning . New York:ACM Press , 2020 : 7197 - 7206 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 770 - 778 .

DENG J , DONG W , SOCHER R , et al . ImageNet:a large-scale hierarchical image database [C ] // Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2009 : 248 - 255 .

SZEGEDY C , VANHOUCKE V , IOFFE S , et al . Rethinking the inception architecture for computer vision [C ] // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 2818 - 2826 .

HAN D , YUN S , HEO B , et al . Rethinking channel dimensions for efficient model design [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2021 : 732 - 741 .

SANDLER M , HOWARD A , ZHU M L , et al . MobileNetV2:inverted residuals and linear bottlenecks [C ] // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2018 : 4510 - 4520 .

RADOSAVOVIC I , KOSARAJU R P , GIRSHICK R , et al . Designing network design spaces [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2020 : 10425 - 10433 .

HAN K , WANG Y H , TIAN Q , et al . GhostNet:more features from cheap operations [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2020 : 1577 - 1586 .

LIN T Y , MAIRE M , BELONGIE S J , et al . Microsoft COCO:common objects in context [C ] // Proceedings of 2014 European Conference on Computer Vision . Berlin:Springer , 2014 : 740 - 755 .

ZHANG S F , WEN L Y , BIAN X , et al . Single-shot refinement neural network for object detection [C ] // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2018 : 4203 - 4212 .

ZHAO Q J , SHENG T , WANG Y T , et al . M2Det:a single-shot object detector based on multi-level feature pyramid network [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . New York:ACM Press , 2019 : 9259 - 9266 .

浏览量

790

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于CNN-BiLSTM-Attention融合模型的差分隐私轨迹重构攻击

基于多尺度卷积融合编码网络的调制识别方法

基于神经网络的恶意DNS流量检测方法

基于同态密文转换的隐私保护卷积神经网络推理方案

基于函数加密的密文卷积神经网络模型