Study on video action recognition based on augment negative example multi-granularity discrimination model

LIU Liangzhen; YANG Yang; XIA Yingjie; KUANG Li

doi:10.11959/j.issn.1000-436x.2024268

您当前的位置：

首页 >

文章列表页 >

Study on video action recognition based on augment negative example multi-granularity discrimination model

Papers | 更新时间：2025-01-14

- Study on video action recognition based on augment negative example multi-granularity discrimination model
- Journal on Communications Vol. 45, Issue 12, Pages: 28-43(2024)
- 作者机构：
  
  1.中南大学计算机学院，湖南长沙 410083
  2.杭州电子科技大学微电子研究院，浙江杭州 310005
  3.浙江大学计算机科学与技术学院，浙江杭州 310012
- 作者简介：
- 基金信息：
  
  The National Key Research and Development Program of China(2022YFF0902500)
- DOI：10.11959/j.issn.1000-436x.2024268
  CLC： TP391.41
- Received：29 July 2024，
  
  Revised：2024-12-05，
  
  Published：25 December 2024
- 稿件说明：
移动端阅览
刘良振,杨阳,夏莹杰等.基于增强负例多粒度区分模型的视频动作识别研究[J].通信学报,2024,45(12):28-43.

LIU Liangzhen,YANG Yang,XIA Yingjie,et al.Study on video action recognition based on augment negative example multi-granularity discrimination model[J].Journal on Communications,2024,45(12):28-43.
刘良振,杨阳,夏莹杰等.基于增强负例多粒度区分模型的视频动作识别研究[J].通信学报,2024,45(12):28-43. DOI： 10.11959/j.issn.1000-436x.2024268.

LIU Liangzhen,YANG Yang,XIA Yingjie,et al.Study on video action recognition based on augment negative example multi-granularity discrimination model[J].Journal on Communications,2024,45(12):28-43. DOI： 10.11959/j.issn.1000-436x.2024268.

摘要

为提升模型对视频动作的细粒度区分能力，提出一种基于对比学习的增强负例区分范式。通过为每个视频生成增强负例集合，以补充最难区分的视频-文本负例对。为了进一步区分正负例，基于该范式提出一种用于视频动作识别的多粒度区分模型。在该模型中，视频表征器通过引入文本正例特征引导视频特征提取，而正负语义区分器利用自注意力机制构建正负语义之间的自相关关系。该模型既能够实现模态间视频与增强负例集的粗粒度区分，还可以实现文本模态内正例与增强负例集的细粒度区分。实验结果表明，增强负例集能显著提升模型在细粒度类别标签上的识别能力，多粒度区分模型在Kinetics-400、HMDB51和UCF101数据集上的性能均优于当前较具代表性的方法。

Abstract

An augment negative example discrimination paradigm based on contrastive learning was proposed to improve the model’s fine-grained discrimination ability of video actions. The most challenging video-text negative pairs was generated

forming an augmented negative example set for each video sample. Based on this paradigm

a multi-granularity discrimination model for video action recognition was proposed to further distinguish between positive and negative examples. In this model

video features were extracted by the video representation module guided by textual positive examples

while self-correlation relationships between positive and negative semantics were established by the semantic discriminator equipped with a self-attention mechanism. Meanwhile

a coarse-grained distinction between the video modality and the augmented negative example set was achieved

while a fine-grained distinction between positive examples and the augmented negative example set within the text modality was also accomplished. Experimental results demonstrate that the augment negative set improves the model’s recognition ability on fine-grained class labels

and the multi-granularity discrimination model outperforms current state-of-the-art methods on the Kinetics-400

HMDB51 and UCF101 datasets.

关键词

Keywords

references

顾晓丹 , 吴文甲 , 凌振 . 用户密集环境下基于边缘智能的直播视频传输优化机制 [J ] . 通信学报 , 2023 , 44 ( 11 ): 55 - 66 .

GU X D , WU W J , LING Z . Live video transmission optimization mechanism based on edge intelligence in high client-density environment [J ] . Journal on Communications , 2023 , 44 ( 11 ): 55 - 66 .

毕春艳 , 刘越 . 基于深度学习的视频人体动作识别综述 [J ] . 图学学报 , 2023 , 44 ( 4 ): 625 - 639 .

BI C Y , LIU Y . A survey of video human action recognition based on deep learning [J ] . Journal of Graphics , 2023 , 44 ( 4 ): 625 - 639 .

张伟 , 王宇 , 陈新怡 , 等 . 基于图像块码本模型的监控视频背景参考帧生成方法 [J ] . 通信学报 , 2023 , 44 ( 1 ): 129 - 141 .

ZHANG W , WANG Y , CHEN X Y , et al . Background reference frame generation method for surveillance video based on image block codebook model [J ] . Journal on Communications , 2023 , 44 ( 1 ): 129 - 141 .

许可 , 李嘉怡 , 蒋兴浩 , 等 . 一种基于轮廓稀疏对抗的视频步态隐私保护算法 [J ] . 信息网络安全 , 2024 , 24 ( 1 ): 48 - 59 .

XU K , LI J Y , JIANG X H , et al . A video gait privacy protection algorithm based on sparse adversarial attack on silhouette [J ] . Netinfo Security , 2024 , 24 ( 1 ): 48 - 59 .

NAM B T , JUNGER D , CURIO C , et al . Towards human action recognition during surgeries using de-identified video data [J ] . Current Directions in Biomedical Engineering , 2022 , 8 ( 1 ): 109 - 112 .

KARPATHY A , TODERICI G , SHETTY S , et al . Large-scale video classification with convolutional neural networks [C ] // Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2014 : 1725 - 1732 .

SIMONYAN K , ZISSERMAN A . Two-stream convolutional networks for action recognition in videos [J ] . Neural Information Processing Systems , 2014 ( 1 ): 568 - 576 .

CARREIRA J , ZISSERMAN A . Quo vadis, action recognition? a new model and the kinetics dataset [C ] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2017 : 4724 - 4733 .

LIN J , GAN C , HAN S . TSM: temporal shift module for efficient video understanding [C ] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2019 : 7082 - 7092 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // International Conference on Machine Learning . New York : ACM Press , 2021 : 8748 - 8763 .

WANG M M , XING J Z , LIU Y . ActionCLIP: a new paradigm for video action recognition [J ] . arXiv Preprint , arXiv: 2109.08472 , 2021 .

KAY W , CARREIRA J , SIMONYAN K , et al . The kinetics human action video dataset [J ] . arXiv Preprint , arXiv: 1705.06950 , 2017 .

CAI T T , FRANKLE J , SCHWAB D J , et al . Are all negatives created equal in contrastive instance discrimination? [J ] . arXiv Preprint , arXiv: 2010.06682 , 2020 .

HORN B K P , SCHUNCK B G . Determining optical flow [J ] . Artificial Intelligence , 1981 , 17 ( 1-3 ): 185 - 203 .

WANG L M , QIAO Y , TANG X O . Action recognition with trajectory-pooled deep-convolutional descriptors [C ] // Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2015 : 4305 - 4314 .

DONAHUE J , HENDRICKS L A , GUADARRAMA S , et al . Long-term recurrent convolutional networks for visual recognition and description [C ] // Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2015 : 2625 - 2634 .

WANG L M , XIONG Y J , WANG Z , et al . Temporal segment networks: towards good practices for deep action recognition [C ] // Lecture Notes in Computer Science . Berlin : Springer , 2016 : 20 - 36 .

FEICHTENHOFER C , PINZ A , WILDES R P . Spatiotemporal multiplier networks for video action recognition [C ] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2017 : 7445 - 7454 .

JI S W , YANG M , YU K . 3D convolutional neural networks for human action recognition [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2013 , 35 ( 1 ): 221 - 231 .

TRAN D , BOURDEV L , FERGUS R , et al . Learning spatiotemporal features with 3D convolutional networks [C ] // Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2015 : 4489 - 4497 .

HARA K , KATAOKA H , SATOH Y . Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and imagenet? [C ] // Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2018 : 6546 - 6555 .

XIE S N , SUN C , HUANG J , et al . Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification [C ] // Lecture Notes in Computer Science . Berlin : Springer , 2018 : 318 - 335 .

FEICHTENHOFER C , FAN H Q , MALIK J , et al . SlowFast networks for video recognition [C ] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2019 : 6201 - 6210 .

FEICHTENHOFER C . X3D: expanding architectures for efficient video recognition [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 200 - 210 .

ZHU Y , LI X Y , LIU C H , et al . A comprehensive study of deep video action recognition [J ] . arXiv Preprint , arXiv: 2012.06567 , 2020 .

PIERGIOVANNI A , ANGELOVA A , RYOO M S . Tiny video networks [J ] . arXiv Preprint , arXiv: 1910.06961 , 2019 .

JIANG B Y , WANG M M , GAN W H , et al . STM: Spatiotemporal and motion encoding for action recognition [C ] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2019 : 2000 - 2009 .

LI Y , JI B , SHI X T , et al . TEA: temporal excitation and aggregation for action recognition [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 906 - 915 .

LIU Z Y , WANG L M , WU W , et al . TAM: temporal adaptive module for video recognition [C ] // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2021 : 13688 - 13698 .

WANG L M , TONG Z , JI B , et al . TDN: temporal difference networks for efficient action recognition [C ] // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2021 : 1895 - 1904 .

ZHANG Y T , BAI Y , WANG H , et al . Look more but care less in video recognition [J ] . Neural Information Processing Systems , 2022 , 35 : 30813 - 30825 .

JU C , HAN T D , ZHENG K H , et al . Prompting visual-language models for efficient video understanding [C ] // Lecture Notes in Computer Science . Berlin : Springer , 2022 : 105 - 124 .

NI B L , PENG H W , CHEN M H , et al . Expanding language-image pretrained models for general video recognition [C ] // Lecture Notes in Computer Science . Berlin : Springer , 2022 : 1 - 18 .

PAN J T , LIN Z Y , ZHU X T , et al . ST-adapter: parameter-efficient image-to-video transfer learning [J ] . Neural Information Processing Systems , 2022 , 35 : 26462 - 26477 .

ZHAO Y C , LUO C , TANG C X , et al . Streaming video model [C ] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2023 : 14602 - 14612 .

WU W H , SUN Z , OUYANG W L . Revisiting classifier: transferring vision-language models for video recognition [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2023 , 37 ( 3 ): 2847 - 2855 .

WU W H , WANG X H , LUO H P , et al . Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models [C ] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2023 : 6620 - 6630 .

WANG M M , XING J Z , JIANG B Y , et al . A multimodal, multi-task adapting framework for video action recognition [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2024 , 38 ( 6 ): 5517 - 5525 .

KUEHNE H , JHUANG H , GARROTE E , et al . HMDB: a large video database for human motion recognition [C ] // Proceedings of the 2011 International Conference on Computer Vision . Piscataway : IEEE Press , 2011 : 2556 - 2563 .

SOOMRO K , ZAMIR A R , SHAH M . UCF101: a dataset of 101 human actions classes from videos in the wild [J ] . arXiv Preprint , arXiv: 1212.0402 , 2012 .

DENG J , DONG W , SOCHER R , et al . ImageNet: a large-scale hierarchical image database [C ] // Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2009 : 248 - 255 .

LIU Y Z , YUAN J S , TU Z G . Motion-driven visual tempo learning for video-based action recognition [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 4104 - 4116 .

WANG M M , XING J Z , SU J , et al . Learning spatiotemporal and motion features in a unified 2D network for action recognition [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 3 ): 3347 - 3362 .

SHENG X X , LI K C , SHEN Z Q , et al . A progressive difference method for capturing visual tempos on action recognition [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , 33 ( 3 ): 977 - 987 .

CHEN Y T , GE H W , LIU Y X , et al . AGPN: action granularity pyramid network for video action recognition [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , 33 ( 8 ): 3912 - 3923 .

MOU Y T , JIANG X H , XU K , et al . Compressed video action recognition with dual-stream and dual-modal transformer [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2024 , 34 ( 5 ): 3299 - 3312 .

ZHENG Z W , YANG L , WANG Y L , et al . Dynamic spatial focus for efficient compressed video action recognition [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2024 , 34 ( 2 ): 695 - 708 .

LI Z L , LI J , MA Y Q , et al . Spatio-temporal adaptive network with bidirectional temporal difference for action recognition [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , 33 ( 9 ): 5174 - 5185 .

ZHANG Y K , LI J , JIANG N , et al . Temporal transformer networks with self-supervision for action recognition [J ] . IEEE Internet of Things Journal , 2023 , 10 ( 14 ): 12999 - 13011 .

Views

1306

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Few-shot document-level event extraction method for threat intelligence

Traffic concealed data detection method based on contrastive learning and pre-trained Transformer

Sequential recommendation algorithm for long-tail users based on knowledge-enhanced contrastive learning

CLB-Defense: based on contrastive learning defense for graph neural network against backdoor attack

Method based on contrastive incremental learning for fine-grained malicious traffic classification

Related Author

Umer Nauman

TANG Mengmeng

DENG Miaolei

GUO Yuanbo

WANG Jinfang

FU Cai

HE Shuai

ZHANG Jingchao

Related Institution

School of Computer Science, Zhengzhou University of Aeronautics

National Key Laboratory of Security Communication

School of Information Science and Engineering, Henan University of Technology

School of Cyberspace Security, Hainan University

School of Cyberspace Science and Engineering, Huazhong University of Science and Technology

AI问答

⁰