Unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features

Zeyu WANG; Shuhui BU; Wei HUANG; Yuanpan ZHENG; Qinggang WU; Huawen CHANG; Xu ZHANG

doi:10.11959/j.issn.1000-436x.2022212

您当前的位置：

首页 >

文章列表页 >

Unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features

Papers | 更新时间：2024-06-05

- Unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features
- Journal on Communications Vol. 43, Issue 12, Pages: 157-171(2022)
- 作者机构：
  
  1. 郑州轻工业大学计算机与通信工程学院，河南郑州 450000
  2. 西北工业大学航空学院，陕西西安 710072
- 作者简介：
- 基金信息：
  
  The Science and Technology Project of Henan Province(222102210021);The Plan Support for Key Scientific Research Project of Higher Education in Henan Province(21A520049);The Plan Support for Key Scientific Research Project of Higher Education in Henan Province(23A520004)
- DOI：10.11959/j.issn.1000-436x.2022212
  CLC： TP391
- Online First：2022-12，
  
  Published：25 December 2022
- 稿件说明：
移动端阅览
Zeyu WANG, Shuhui BU, Wei HUANG, et al. Unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features[J]. Journal on Communications, 2022, 43(12): 157-171.
DOI：

Zeyu WANG, Shuhui BU, Wei HUANG, et al. Unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features[J]. Journal on Communications, 2022, 43(12): 157-171. DOI： 10.11959/j.issn.1000-436x.2022212.

摘要

为了解决领域自适应中存在领域间视觉、空间以及语义特征分布差异的问题，提出了基于多模态特征的无监督领域自适应多级对抗语义分割网络。首先，设计3层结构的注意力融合语义分割网络来分别从源域和目标域学习上述三类特征。然后，在单级对抗学习中引入联合分布置信度和语义置信度的自监督学习方法，从而在领域间所学特征的分布距离最小化过程中实现更多目标域像素的分布对齐。最后，通过基于多模态特征的多级对抗学习方法对3路对抗分支与3个自适应子网进行联合优化，从而能够有效学习各子网所提取特征的域间不变表示。实验结果表明，与当前先进方法相比，所提网络在GTA5到Cityscapes、SYNTHIA到Cityscapes和SUN-RGBD到NYUD-v2的数据集上分别取得最优的平均交并比62.2%、66.9%和59.7%。

Abstract

In order to solve the problem of the distribution differences of visual

spatial

and semantic features between domains in domain adaptation

an unsupervised domain adaptation multi-level adversarial network for semantic segmentation based on multi-modal features was proposed.Firstly

an attentive fusion semantic segmentation network with three-layer structure was designed to learn the above three types of features from the source domain and target domain

respectively.Secondly

a self-supervised learning method jointing distribution confidence and semantic confidence was introduced into the single-level adversarial learning

so as to achieve the distribution alignment of more target domain pixels in the process of minimizing the distribution distance of the learnt features between domains.Finally

three adversarial branches and three adaptive sub-networks were jointly optimized by the multi-level adversarial learning method based on multi-modal features

which could effectively learn the invariant representation between domains for the features extracted from each sub-network.The experimental results show that compared with existing state-of-the-art methods

on the datasets of GTA5 to Cityscapes

SYNTHIA to Cityscapes

and SUN-RGBD to NYUD-v2 the proposed network achieves the best mean intersection over union of 62.2%

66.9%

and 59.7%

respectively.

关键词

Keywords

references

徐英姿 , 刘原 , 时梦然 , 等 . 语义在通信中的应用综述 [J ] . 电信科学 , 2022 , 38 ( Z1 ): 43 - 59 .

XU Y Z , LIU Y , SHI M R , et al . A survey of semantic applications in communications [J ] . Telecommunications Science , 2022 , 38 ( Z1 ): 43 - 59 .

AGIA C , JATAVALLABHULA K M , KHODEIR M , et al . Taskography:evaluating robot task planning over large 3D scene graphs [C ] // Proceedings of Conference on Robot Learning . Cambridge:JMLR , 2022 : 46 - 58 .

CAESAR H , BANKITI V , LANG A H , et al . nuScenes:a multimodal dataset for autonomous driving [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2020 : 11621 - 11631 .

YU C , LIU Z X , LIU X J , et al . DS-SLAM:a semantic visual SLAM towards dynamic environments [C ] // Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . Piscataway:IEEE Press , 2018 : 1168 - 1174 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // Proceedings of IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 770 - 778 .

LI Z , GAN Y , LIANG X , et al . LSTM-CF:unifying context modeling and fusion with LSTMS for RGB-D scene labeling [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2016 : 541 - 557 .

YUAN Y H , CHEN X L , WANG J D . Object-contextual representations for semantic segmentation [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2020 : 173 - 190 .

RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks [J ] . arXiv Preprint,arXiv:1511.06434 , 2015 .

HOFFMAN J , WANG D , YU F , et al . FCNs in the wild:Pixel-level adversarial and constraint-based adaptation [J ] . arXiv Preprint,arXiv:1612.02649 , 2016 .

TSAI Y H , HUNG W C , SCHULTER S , et al . Learning to adapt structured output space for semantic segmentation [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2018 : 7472 - 7481 .

ZHU J Y , PARK T , ISOLA P , et al . Unpaired image-to-image translation using cycle-consistent adversarial networks [C ] // Proceedings of the IEEE International Conference on Computer Vision . Piscataway:IEEE Press , 2017 : 2223 - 2232 .

LI Z Y , TOGO R , OGAWA T , et al . Learning intra-domain style-invariant representation for unsupervised domain adaptation of semantic segmentation [J ] . Pattern Recognition , 2022 , 132 ( 12 ): 108911 .

LI Y S , YUAN L , VASCONCELOS N . Bidirectional learning for domain adaptation of semantic segmentation [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2019 : 6936 - 6945 .

YANG J , AN W , WANG S , et al . Label-driven reconstruction for domain adaptation in semantic segmentation [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2020 : 480 - 498 .

CHENG Y , WEI F , BAO J , et al . Dual path learning for domain adaptation of semantic segmentation [C ] // Proceedings of the IEEE/CVF International Conference on Computer Vision . Piscataway:IEEE Press , 2021 : 9082 - 9091 .

LEE S , HYUN J , SEONG H , et al . Unsupervised domain adaptation for semantic segmentation by content transfer [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto:AAAI Press , 2021 : 8306 - 8315 .

SHIN I , WOO S , PAN F , et al . Two-phase pseudo label densification for self-training based domain adaptation [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2020 : 532 - 548 .

PAN F , SHIN I , RAMEAU F , et al . Unsupervised intra-domain adaptation for semantic segmentation through self-supervision [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2020 : 3764 - 3773 .

PENG C L , MA J Y . Domain adaptive semantic segmentation via entropy-ranking and uncertain learning-based self-training [J ] . IEEE/CAA Journal of Automatica Sinica , 2022 , 9 ( 8 ): 1524 - 1527 .

YANG J , AN W , YAN C , et al . Context-aware domain adaptation in semantic segmentation [C ] // Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . Piscataway:IEEE Press , 2021 : 514 - 524 .

HUANG J , LU S , GUAN D , et al . Contextual-relation consistent domain adaptation for semantic segmentation [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2020 : 705 - 722 .

RICHTER S R , VINEET V , ROTH S , et al . Playing for data:ground truth from computer games [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2016 : 102 - 118 .

ROS G , SELLART L , MATERZYNSKA J , et al . The SYNTHIA dataset:a large collection of synthetic images for semantic segmentation of urban scenes [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 3234 - 3243 .

CORDTS M , OMRAN M , RAMOS S , et al . The cityscapes dataset for semantic urban scene understanding [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 3213 - 3223 .

SONG S R , LICHTENBERG S P , XIAO J X . SUN RGB-D:a RGB-D scene understanding benchmark suite [C ] // Proceedings of IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2015 : 567 - 576 .

SILBERMAN N , HOIEM D , KOHLI P , et al . Indoor segmentation and support inference from RGBD images [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2012 : 746 - 760 .

PASZKE A , GROSS S , MASSA F , et al . PyTorch:an imperative style,high-performance deep learning library [J ] . Advances in Neural Information Processing Systems , 2019 , 32 ( 12 ): 8024 - 8035 .

GUO X Q , YANG C , LI B P , et al . MetaCorrection:domain-aware meta loss correction for unsupervised domain adaptation in semantic segmentation [C ] // Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway:IEEE Press , 2021 : 3926 - 3935 .

ARASLANOV N , ROTH S . Self-supervised augmentation consistency for adapting semantic segmentation [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2021 : 15384 - 15394 .

JIANG Z K , LI Y X , YANG C Y , et al . Prototypical contrast adaptation for domain adaptive semantic segmentation [C ] // Proceedings of European Conference on Computer Vision . Berlin:Springer , 2022 : 36 - 54 .

ZHANG F , KOLTUN V , TORR P , et al . Unsupervised contrastive domain adaptation for semantic segmentation [J ] . arXiv Preprint,arXiv:2204.08399 , 2022 .

YANG J Y , LI C Y , AN W Z , et al . Exploring robustness of unsupervised domain adaptation in semantic segmentation [C ] // Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway:IEEE Press , 2021 : 9174 - 9183 .

Views

452

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Fusion reconstruction mechanism and contrast learning method for WSN abnormal node detection

Lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution

Self-supervised speech representation learning based on positive sample comparison and masking reconstruction

YOLOv3-A: a traffic sign detection network based on attention mechanism

Semantic segmentation of 3D point cloud based on contextual attention CNN

Related Author

WANG Yong

JIANG Qiuxiang

HUANG Yuan

CHENG Jin

YE Miao

Quanyi WANG

Gang XIE

Xinlin XIE

Related Institution

Guangxi Key Laboratory of Wireless Wideband Communication and Signal Processing, Guilin University of Electronic Technology

School of Electronic and Information Engineering, Taiyuan University of Science and Technology

Shanxi Key Laboratory of Advanced Control and Equipment Intelligence

College of Information System Engineering, Information Engineering University

School of Automation, Central South University

AI问答

⁰