面向异构流式数据的高性能联邦持续学习算法

姜慧; 何天流; 刘敏; 孙胜; 王煜炜

doi:10.11959/j.issn.1000-436x.2023102

您当前的位置：

首页 >

文章列表页 >

面向异构流式数据的高性能联邦持续学习算法

学术论文 | 更新时间：2024-06-06

- 面向异构流式数据的高性能联邦持续学习算法
- High-performance federated continual learning algorithm for heterogeneous streaming data
- 通信学报 2023年44卷第5期页码：123-136
- 作者机构：
  
  1. 中国科学院计算技术研究所，北京 100190
  2. 中国科学院大学计算机科学与技术学院，北京 100190
  3. 中关村实验室，北京 100084
- 作者简介：
  
  [ "姜慧（1995- ），女，江苏扬州人，中国科学院大学博士生，主要研究方向为联邦学习、边缘智能、分布式机器学习等" ]
  [ "何天流（1999- ），男，江西吉安人，中国科学院大学硕士生，主要研究方向为联邦学习、边缘智能、分布式机器学习等" ]
  [ "刘敏（1976- ），女，河南偃师人，博士，中国科学院计算技术研究所研究员、博士生导师，主要研究方向为移动计算和边缘智能" ]
  [ "孙胜（1990- ），女，河北衡水人，博士，中国科学院计算技术研究所助理研究员，主要方向为联邦学习、移动计算和边缘智能" ]
  [ "王煜炜（1980- ），男，河北唐山人，博士，中国科学院计算技术研究所高级工程师、硕士生导师，主要研究方向为联邦学习、移动边缘计算和下一代网络架构" ]
- 基金信息：
  
  国家重点研发计划基金资助项目(2021YFB2900102);国家自然科学基金资助项目(62072436)
- DOI：10.11959/j.issn.1000-436x.2023102
  中图分类号： TP302
- 网络首发：2023-05，
  
  纸质出版：2023-05-25
- 稿件说明：
移动端阅览
姜慧, 何天流, 刘敏, 等. 面向异构流式数据的高性能联邦持续学习算法[J]. 通信学报, 2023,44(5):123-136.

Hui JIANG, Tianliu HE, Min LIU, et al. High-performance federated continual learning algorithm for heterogeneous streaming data[J]. Journal on Communications, 2023, 44(5): 123-136.
姜慧, 何天流, 刘敏, 等. 面向异构流式数据的高性能联邦持续学习算法[J]. 通信学报, 2023,44(5):123-136. DOI： 10.11959/j.issn.1000-436x.2023102.

Hui JIANG, Tianliu HE, Min LIU, et al. High-performance federated continual learning algorithm for heterogeneous streaming data[J]. Journal on Communications, 2023, 44(5): 123-136. DOI： 10.11959/j.issn.1000-436x.2023102.

摘要

为了缓解提供智能服务的 AI 模型训练流式数据存在模型性能差、训练效率低等问题，在具有隐私数据的分布式终端系统中，提出了一种面向异构流式数据的高性能联邦持续学习算法（FCL-HSD）。为了缓解当前模型遗忘旧数据问题，在本地训练阶段引入结构可动态扩展模型，并设计扩展审核机制，以较小的存储开销来保障AI模型识别旧数据的能力；考虑到终端的数据异构性，在中央节点侧设计了基于数据分布相似度的全局模型定制化策略，并为模型的不同模块执行分块聚合方式。在不同数据集下多种数据增量场景中验证了所提算法的可行性和有效性。实验结果证明，相较于现有工作，所提算法在保证模型对新数据具有分类能力的前提下，可以有效提升模型对旧数据的分类能力。

Abstract

Aiming at the problems of poor model performance and low training efficiency in training streaming data of AI models that provide intelligent services

a high-performance federated continual learning algorithm for heterogeneous streaming data (FCL-HSD) was proposed in the distributed terminal system with privacy data.In order to solve the problem of the current model forgetting old data

a model with dynamically extensible structure was introduced in the local training stage

and an extension audit mechanism was designed to ensure the capability of the AI model to recognize old data at the cost of small storage overhead.Considering the heterogeneity of terminal data

a customized global model strategy based on data distribution similarity was designed at the central server side

and an aggregation-by-block manner was implemented for different modules of the model.The feasibility and effectiveness of the proposed algorithm were verified under various data increment scenarios with different data sets.Experimental results show that

compared with existing works

the proposed algorithm can effectively improve the model performance to classify old data on the premise of ensuring the capability to classify new data.

关键词

Keywords

references

麻省理工科技评论 . 2021 年中国数字经济时代人工智能生态白皮书 [R ] . 2022 .

MIT Technology Review . 2021 white paper on artificial intelligence ecology in China's digital economy era [R ] . 2022 .

BILOGREVIC I , JADLIWALA M , KALKAN K , et al . Privacy in mobile computing for location-sharing-based services [C ] // Inter national Symposium on Privacy Enhancing Technologies Symposium . Berlin:Springer , 2011 : 77 - 96 .

LIANG X H , LI X , LUAN T H , et al . Morality-driven data forwarding with privacy preservation in mobile social networks [J ] . IEEE Transactions on Vehicular Technology , 2012 , 61 ( 7 ): 3209 - 3222 .

KONEČNÝ J , MCMAHAN H B , RAMAGE D , et al . Federated optimization:distributed machine learning for on-device intelligence [J ] . arXiv Preprint,arXiv:1610.02527 , 2016 .

LE J Q , LEI X Y , MU N K , et al . Federated continuous learning with broad network architecture [J ] . IEEE Transactions on Cybernetics , 2021 , 51 ( 8 ): 3874 - 3888 .

SERRA J , SURIS D , MIRON M , et al . Overcoming catastrophic forgetting with hard attention to the task [C ] // Proceedings of the International Conference on Machine Learning . New York:ACM Press , 2018 : 4548 - 4557 .

WU Y , CHEN Y P , WANG L J , et al . Large scale incremental learning [C ] // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2020 : 374 - 382 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2016 : 770 - 778 .

CASTRO F M , MARÍN-JIMÉNEZ M J , GUIL N , et al . End-to-end incremental learning [C ] // European Conference on Computer Vision . Berlin:Springer , 2018 : 241 - 257 .

MASANA M , LIU X L , TWARDOWSKI B , et al . Class-incremental learning:survey and performance evaluation on image classification [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 5 ): 5513 - 5533 .

LI Z Z , HOIEM D . Learning without forgetting [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018 , 40 ( 12 ): 2935 - 2947 .

REBUFFI S A , KOLESNIKOV A , SPERL G , et al . iCaRL:incremental classifier and representation learning [C ] // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2017 : 5533 - 5542 .

YAN S P , XIE J W , HE X M . D:dynamically expandable representation for class incremental learning [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2021 : 3013 - 3022 .

MUN H , LEE Y . Internet traffic classification with federated learning [J ] . Electronics , 2020 , 10 ( 1 ): 27 .

DONG J H , WANG L X , FANG Z , et al . Federated class-incremental learning [C ] // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2022 : 10154 - 10163 .

LI T , SANJABI M , BEIRAMI A , et al . Fair resource allocation in federated learning [J ] . arXiv Preprint,arXiv:1905.10497 , 2019 .

ABAD M S H , OZFATURA E , GUNDUZ D , et al . Hierarchical federated learning ACROSS heterogeneous cellular networks [C ] // Proceed ings of 2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP) . Piscataway:IEEE Press , 2020 : 8866 - 8870 .

LONG G , XIE M , SHEN T , et al . Multi-center federated learning:clients clustering for better personalization [J ] . arXiv Preprint,arXiv:2005.01026 , 2020 .

KRIZHEVSKY A , SUTSKEVER I , GEOFFREY E H . Learning multiple layers of features from tiny images [J ] . Communications of the ACM , 2012 , 60 ( 6 ): 84 - 90 .

DRAPER-GIL G , LASHKARI A H , MAMUN M S I , et al . Characterization of encrypted and VPN traffic using time-related features [C ] // Pro ceedings of the 2nd International Conference on Information Systems Security and Privacy . [S.l]:Scite Press , 2016 : 407 - 414 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks [J ] . Communications of the ACM , 2017 , 60 ( 6 ): 84 - 90 .

骆子铭 , 许书彬 , 刘晓东 . 基于机器学习的TLS恶意加密流量检测方案 [J ] . 网络与信息安全学报 , 2020 , 60 ( 1 ): 77 - 83 .

LUO Z M , XU S B , LIU X D . Scheme for identifying malware traffic with TLS data based on machine learning [J ] . Chinese Journal of Network and Information Security , 2020 , 60 ( 1 ): 77 - 83 .

PACHECO F , EXPOSITO E , GINESTE M , et al . Towards the deployment of machine learning solutions in network traffic classification:a systematic survey [J ] . IEEE Communications Surveys ＆ Tutorials , 2019 , 21 ( 2 ): 1988 - 2014 .

XIE G R , LI Q , JIANG Y . Self-attentive deep learning method for online traffic classification and its interpretability [J ] . Computer Networks , 2021 ,196:108267.

浏览量

1605

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向动态算力节点的联邦学习差分隐私重校准方法

HFLGuard：基于聚类和自编码器的异构联邦中毒防御方法

基于多方计算的安全拜占庭弹性联邦学习

空天地一体化网络中基于联邦深度强化学习的边缘协作缓存策略

基于零集中差分隐私的联邦学习激励方案