A Federated Distillation Method Based on Feature Projection and Adaptive Enhancement

Chen Ningjiang; Zhang Dehua

doi:10.11959/j.issn.1000-436x.TXXB260116

您当前的位置：

首页 >

文章列表页 >

A Federated Distillation Method Based on Feature Projection and Adaptive Enhancement

更新时间：2026-05-14

- A Federated Distillation Method Based on Feature Projection and Adaptive Enhancement
- Journal on Communications (2026)
- 作者机构：
  
  1.广西大学计算机与电子信息学院，广西南宁 530004
  2.广西智能数字服务技术创新中心，广西南宁 530004
  3.广西高校并行分布与智能计算重点实验室，广西南宁 530004
  4.广西人工智能学院，广西南宁 530004
- 作者简介：
- 基金信息：
  
  The Guangxi Key Research and Development Program Project(GuikeAB25069258;GuikeAB25069130)
- DOI：10.11959/j.issn.1000-436x.TXXB260116
  CLC： TP309
- Received：09 March 2026，
  
  Revised：2026-05-13，
  
  Accepted：14 May 2026，
- 稿件说明：
移动端阅览
Chen Ningjiang, Zhang Dehua. A Federated Distillation Method Based on Feature Projection and Adaptive Enhancement[J/OL]. Journal on Communications, 2026.
DOI：

Chen Ningjiang, Zhang Dehua. A Federated Distillation Method Based on Feature Projection and Adaptive Enhancement[J/OL]. Journal on Communications, 2026. DOI： 10.11959/j.issn.1000-436x.TXXB260116.

摘要

为了解决现有联邦蒸馏方法难以处理异构模型间表征空间不一致与特征分布不均的问题，提出一种基于特征投影与自适应增强的联邦异构知识蒸馏框架，实现了跨异构客户端模型的知识高效融合。该框架在服务器端通过蒸馏方式整合客户端输出，在客户端设计轻量化多出口分支，将异构模型的中间特征投影到对齐的logits空间，以突破异构蒸馏中的特征对齐瓶颈。实验结果表明，该方法在标准数据集上取得良好效果，减少了通信轮次和数据传输量，同时增强了系统在异构环境下的鲁棒性，为联邦异构知识融合提供了一种有效的新方案。

Abstract

To address the problem that existing federated distillation methods struggle to handle inconsistent representation spaces and uneven feature distributions across heterogeneous models

a federated heterogeneous knowledge distillation framework based on feature projection and adaptive enhancement was proposed

achieving efficient knowledge fusion across heterogeneous client models. On the server side

client outputs were integrated via distillation; on the client side

lightweight multi-exit branches were employed to project intermediate features into an aligned logits space to overcome feature alignment bottlenecks. Promising performance was demonstrated on benchmark datasets

with reduced communication rounds and data transmission as well as enhanced system robustness

thus providing an effective new solution for heterogeneous model knowledge fusion in federated learning.

关键词

Keywords

references

Verbraeken J , Wolting M , Katzy J , et al . A survey on distributed machine learning [J ] . ACM Computing Surveys , 2020 , 53 ( 2 ): 1 - 33 .

Kairouz P , Mcmahan H B , Avent B , et al . Advances and open problems in federated learning [J ] . Foundations and Trends in Machine Learning , 2021 , 14 ( 1-2 ): 1 - 210 .

Mcmahan H B , Moore E , Ramage D , et al . Communication efficient learning of deep networks from decentralized data [C ] // Proc of the 20th Int Conf on Artificial Intelligence and Statistics . Fort Lauderdale , 2017 : 1273 - 1282 .

Jeong E , Oh S , Kim H , et al . Communication-efficient on device machine learning: Federated distillation and augmentation under non-IID private data [J/OL ] . arXiv preprint , 2018 : arXiv: 1811.11479 .

Li L , Gou J P , Yu B S , et al . Federated Distillation: A Survey [J/OL ] . arXiv preprint , 2024 : arXiv: 2404.08564 .

Ge H H , Pokhrel S R , Liu Z Y , et al . PFL-DKD: Modeling decoupled knowledge fusion with distillation for improving personalized federated learning [J ] . Computer Networks , 2024 , 254 : 110758 .

Liu W , Chen J Y , Wang B , et al . FedDM: A discrepancy-aware federated learning method based on multibranch feature fusion for non-IID data environments [J ] . IEEE Internet of Things Journal , 2025 , 12 ( 21 ): 45825 - 45835 .

Gou J P , Yu B , Maybank S J , et al . Knowledge distillation: a survey [J ] . International Journal of Computer Vision , 2021 , 129 ( 6 ): 1789 - 1819 .

Huang Y L , Hu K , Zhang Y , et al . Distilling knowledge from heterogeneous architectures for semantic segmentation [C ] // Proc of the AAAI Conference on Artificial Intelligence . Vancouver , 2024 , 38 : 3824 - 3832 .

Li Y C , Wang X Y , Xu W C , et al . Feature distillation is the better choice for model-heterogeneous federated learning [C ] // Proc of the 39th Advances in Neural Information Processing Systems . Vancouver , 2025 : 104726 - 104744 .

Lin T , Kong L J , Stich S U , et al . Ensemble distillation for robust model fusion in federated learning [C ] // Proc of NeurIPS 2020 . Vancouver , 2020 : 2351 - 2363 .

郑晶晶 , 赖金山 , 张凤荔 , 等 . FedASHP：高性能通信的自适应稀疏高效联邦学习方法 [J/OL ] . 小型微型计算机系统 , 1 - 9 [ 2026-04-15 ] . https://doi.org/10.20009/j.cnki.21-1106/TP.2025-0342 https://doi.org/10.20009/j.cnki.21-1106/TP.2025-0342 .

Zheng J J , Lai J S , Zhang F L , et al . FedASHP: An adaptive sparse efficient federated learning method for high-performance communication [J/OL ] . Journal of Chinese Computer Systems , 1 - 9 [ 2026-04-15 ] . https://doi.org/10.20009/j.cnki.21-1106/TP.2025-0342 https://doi.org/10.20009/j.cnki.21-1106/TP.2025-0342 .

刘佳瑜 , 王勇 , 杨静 , 等 . FedDW：基于一致性优化的权重蒸馏异构联邦学习方法 [J ] . 计算机研究与发展 , 2026 , 63 ( 4 ): 998 - 1009 .

Liu J Y , Wang Y , Yang J , et al . FedDW: Weight distillation heterogeneous federated learning method based on consensus optimization [J ] . Journal of Computer Research and Development , 2026 , 63 ( 4 ): 998 - 1009 .

王瑞锦 , 王金波 , 张凤荔 , 等 . 联邦原型学习的特征图中毒攻击和双重防御机制 [J ] . 软件学报 , 2025 , 36 ( 3 ): 1355 - 1374 .

Wang R J , Wang J B , Zhang F L , et al . Feature graph poisoning attack and double defense mechanism for federated prototype learning [J ] . Journal of Software , 2025 , 36 ( 3 ): 1355 - 1374 .

Siddika F , Hossen M A , Zhang W S , et al . Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes [C ] // Proc of the 42nd Int Conf on Machine Learning . Vienna , 2025 .

Song W S , Yan M W , Li X Z , et al . Bidirectional decoupled distillation for heterogeneous federated learning [J ] . Entropy , 2024 , 26 ( 9 ): 762 .

Li K , Ding Y , Zhu Z Q , et al . Transforming gaps into gains: bridging model and data heterogeneity in federated learning via knowledge weak-aware zones [C ] // Proc of the 39th Advances in Neural Information Processing Systems . Vancouver , 2025 .

Yu Y , Kim N . Heterogeneous knowledge distillation using conceptual learning [J ] . IEEE Access , 2024 , 12 : 52803 - 52814 .

Li T , Sahu A K , Talwalkar A , et al . Federated learning: challenges, methods, and future directions [J ] . IEEE Signal Processing Magazine , 2020 , 37 ( 3 ): 50 - 60 .

Bottou L , Curtis F E , Nocedal J . Optimization methods for large-scale machine learning [J ] . SIAM Review , 2018 , 60 ( 2 ): 223 - 311 .

Hsu T M H , Qi H , Brown M . Measuring the effects of non-identical data distribution for federated visual classification [J/OL ] . arXiv preprint arXiv: 1909.06335 , 2019 .

Li T , Sahu A K , Zaheer M , et al . FedProx: Federated optimization in heterogeneous networks [C ] // Proc of MLSys 2020 . Austin , 2020 : 429 - 450 .

Li Q B , He B S , Song D . Model-contrastive federated learning [C ] // Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition . Nashville , 2021 : 10713 - 10722 .

Wu C , Wu F , Lyu L , et al . Communication-efficient federated learning via knowledge distillation [J ] . Nature Communications , 2022 , 13 ( 1 ): 2032 .

Tan Y , Long G D , Liu L , et al . FedProto: Federated prototype learning across heterogeneous clients [C ] // Proc of AAAI 2022 . Vancouver , 2022 : 8042 - 8050 .

Chen H Y , Chao W L . On bridging generic and personalized federated learning for image classification [C ] // Proc of the 10th Int Conf on Learning Representations . Virtual , 2022 .

Yi L P , Yu H , Ren C , et al . Federated Model Heterogeneous Matryoshka Representation Learning [C ] // Proc of the 38th Advances in Neural Information Processing Systems . Vancouver , 2024 : 66431 - 66454 .

Zhang J Q , Liu Y , Hua Y , et al . FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning [C ] // Proc of the 38th AAAI Conference on Artificial Intelligence . AAAI Press , 2024 : 16768 - 16776 .

Du H Z , Xiang Y R , Cai Y W , et al . FedFree: Breaking Knowledge-sharing Barriers through Layer-wise Alignment in Heterogeneous Federated Learning [C ] // The 39th Annual Conference on Neural Information Processing Systems . Vancouver, Canada : Neural Information Processing Systems Foundation , 2025 .

Han P , Xiao H , Zheng S , et al . FedMKD: Hybrid Feature Guided Multilayer Fusion Knowledge Distillation in Heterogeneous Federated Learning [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2026 , 37 ( 3 ): 1463 - 1476 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Dynamic compression method for federated learning based on feature information in ISAC networks

DPBR-Adapt: a hierarchically adaptive differential privacy defence scheme for federated learning

Personalized differential privacy federated learning method for collaborative spectrum sensing

Federated learning with differential privacy recalibration for dynamic computing nodes

HFLGuard: clustering and autoencoder-based defense method against heterogeneous federated poisoning

Related Author

Deng Bingguang

Peng Jiayin

Hu Ronglei

Bai Chenyang

Wei Zhanzhen

Han Yanyan

Duan Xiaoyi

Zhang Hao

Related Institution

School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications

Department of Electronic and Communication Engineering, Beijing Electronic Science and Technology Institute

School of Information Engineering, Minzu University of China

School of Automation, Guangdong University of Technology

School of Cyberspace Security, Beijing Jiaotong University

AI问答

⁰