基于大语言模型的多模态社交媒体信息流行度预测研究
Research on multimodal social media information popularity prediction based on large language model
- 通信学报 2024年45卷第11期 页码：141-156
- 作者机构：
首都师范大学管理学院，北京 100089
- 作者简介：
[ "王洁（1977- ），女，湖北黄石人，博士，首都师范大学教授、硕士生导师，主要研究方向为社交网络分析、数据挖掘等。" ]
[ "王子曈（2000- ），男，北京人，首都师范大学硕士生，主要研究方向为数据挖掘、机器学习等。" ]
[ "彭岩（1967- ），女，北京人，博士，首都师范大学教授、博士生导师，主要研究方向为数据挖掘、机器学习等。" ]
[ "郝博文（1995- ），男，黑龙江牡丹江人，博士，首都师范大学讲师，主要研究方向为数据挖掘、推荐系统、自然语言处理等。" ]
- 基金信息：国家自然科学基金资助项目(62172287;62402328);北京市教育委员会科研计划基金资助项目(KM202410028007)
DOI：10.11959/j.issn.1000-436x.2024193
中图分类号： TP393收稿：2024-08-14，
修回：2024-11-01，
纸质出版：2024-11-25
- 稿件说明：
移动端阅览
王洁,王子曈,彭岩等.基于大语言模型的多模态社交媒体信息流行度预测研究[J].通信学报,2024,45(11):141-156.
WANG Jie,WANG Zitong,PENG Yan,et al.Research on multimodal social media information popularity prediction based on large language model[J].Journal on Communications,2024,45(11):141-156.
王洁,王子曈,彭岩等.基于大语言模型的多模态社交媒体信息流行度预测研究[J].通信学报,2024,45(11):141-156. DOI： 10.11959/j.issn.1000-436x.2024193.
WANG Jie,WANG Zitong,PENG Yan,et al.Research on multimodal social media information popularity prediction based on large language model[J].Journal on Communications,2024,45(11):141-156. DOI： 10.11959/j.issn.1000-436x.2024193.
摘要
针对现有多模态社交媒体信息流行度预测算法对特征依赖强、泛化能力不足、面对少样本/冷启动环境表现不佳的问题，提出了一种基于大语言模型指令微调和人类对齐的多模态社交媒体流行度预测模型MultiSmpLLM。首先，定义面向冷启动用户的多模态社交媒体流行度预测任务。其次，构建多模态微调指令，并分别通过低秩适配微调（LoRA）和冻结微调（Freeze）方法对大语言基座模型（Llama3）进行指令微调。最后，提出了一种改进直接偏好优化（DPO）的算法IDPOP，通过构造偏好数据，并对DPO损失函数施加由参数调节的惩罚项，解决了基于人类反馈的强化学习（RLHF）算法训练不稳定、不收敛，以及标准DPO在社交媒体流行度预测任务中产生错误优化的问题。实验结果表明，MultiSmpLLM显著优于传统多模态预测模型和GPT-4o等多模态大语言模型。
Abstract
To address the limitations of strong feature dependency
insufficient generalization
and inadequate performance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms
a MultiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed. Firstly
the task of multimodal social media popularity prediction for cold-start users was defined. Secondly
multimodal fine-tuning instructions were constructed
and the large language model (Llama3) was instructionally fine-tuned using the low-rank adaptation (LoRA) and parameter freeze (Freeze) method. Finally
an improved direct preference optimization (DPO) algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function
resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction. Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o.
关键词
Keywords
references
SHOKEEN J , RANA C . Social recommender systems: techniques, domains, metrics, datasets and future scope [J ] . Journal of Intelligent Information Systems , 2020 , 54 ( 3 ): 633 - 667 .
ALRASHIDI M , SELAMAT A , IBRAHIM R , et al . Social recommendation for social networks using deep learning approach: a systematic review, taxonomy, issues, and future directions [J ] . IEEE Access , 2023 , 11 : 63874 - 63894 .
LI Y M , XU X K , RIAZ M , et al . Risk identification of public opinion on social media: a new approach based on cross-spatial network analysis [J ] . The Electronic Library , 2024 , 42 ( 4 ): 576 - 597 .
CRANE R , SORNETTE D . Robust dynamic classes revealed by measuring the response function of a social system [J ] . Proceedings of the National Academy of Sciences of the United States of America , 2008 , 105 ( 41 ): 15649 - 15653 .
DU N , DAI H J , TRIVEDI R , et al . Recurrent marked temporal point processes: embedding event history to vector [C ] // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . New York : ACM Press , 2016 : 1555 - 1564 .
WU B , CHENG W H , LIU P Y , et al . SMP challenge: an overview of social media prediction challenge 2019 [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York : ACM Press , 2019 : 2667 - 2671 .
BLUNDELL C , HELLER K A , BECK J M . Modelling reciprocating relationships with Hawkes processes [J ] . Advances in Neural Information Processing Systems , 2012 , 4 : 2600 - 2608 .
LIN S Y , KONG X N , YU P S . Predicting trends in social networks via dynamic activeness model [C ] // Proceedings of the 22nd ACM International Conference on Information & Knowledge Management . New York : ACM Press , 2013 : 1661 - 1666 .
KOBAYASHI R , LAMBIOTTE R . TiDeH: time-dependent Hawkes process for predicting retweet dynamics [C ] // Proceedings of the International AAAI Conference on Web and Social Media , 2021 , 10 ( 1 ): 191 - 200 .
GAO S , MA J , CHEN Z M . Modeling and predicting retweeting dynamics on microblogging platforms [C ] // Proceedings of the Eighth ACM International Conference on Web Search and Data Mining . New York : ACM Press , 2015 : 107 - 116 .
CANNEYT S V , LEROUX P , DHOEDT B , et al . Modeling and predicting the popularity of online news based on temporal and content-related features [J ] . Multimedia Tools and Applications , 2018 , 77 ( 1 ): 1409 - 1436 .
ZHOU F , XU X , TRAJCEVSKI G , et al . A survey of information cascade analysis: models, predictions, and recent advances [J ] . ACM Computing Surveys , 2022 , 54 ( 2 ): 1 - 36 .
BAO P , SHEN H W , HUANG J M , et al . Popularity prediction in microblogging network: a case study on sina weibo [C ] // Proceedings of the 22nd International Conference on World Wide Web . New York : ACM Press , 2013 : 177 - 178 .
WENG L L , MENCZER F , AHN Y Y . Predicting successful memes using network and community structure [C ] // Proceedings of the International AAAI Conference on Web and Social Media , 2014 , 8 ( 1 ): 535 - 544 .
HONG L J , DAN O , DAVISON B D . Predicting popular messages in twitter [C ] // Proceedings of the 20th International Conference Companion on World Wide Web . New York : ACM Press , 2011 : 57 - 58 .
ARORA A , HASSIJA V , BANSAL S , et al . A novel multimodal online news popularity prediction model based on ensemble learning [J ] . Expert Systems , 2023 , 40 ( 8 ): 1 - 23 .
NAVEED N , GOTTRON T , KUNEGIS J , et al . Bad news travel fast: a content-based analysis of interestingness on twitter [C ] // Proceedings of the 3rd International Web Science Conference . New York : ACM Press , 2011 : 1 - 7 .
BAKSHY E , HOFMAN J M , MASON W A , et al . Everyone’s an influencer: quantifying influence on twitter [C ] // Proceedings of the Fourth ACM International Conference on Web Search and Data Mining . New York : ACM Press , 2011 : 65 - 74 .
WANG K , WANG P H , CHEN X , et al . A feature generalization framework for social media popularity prediction [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM Press , 2020 : 4570 - 4574 .
LAI X , ZHANG Y H , ZHANG W . HyFea: winning solution to social media popularity prediction for multimedia grand challenge 2020 [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM Press , 2020 : 4565 - 4569 .
DING K Y , WANG R G , WANG S Q . Social media popularity prediction: a multiple feature fusion approach with deep neural networks [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York : ACM Press , 2019 : 2682 - 2686 .
HSU C C , KANG L W , LEE C Y , et al . Popularity prediction of social media based on multi-modal feature mining [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York : ACM Press , 2019 : 2687 - 2691 .
JIA X Q , SHANG J X , LIU D J , et al . HeDAN: heterogeneous diffusion attention network for popularity prediction of online content [J ] . Knowledge-Based Systems , 2022 , 254 : 109659 .
CHUNG H W , HOU L , LONGPRE S , et al . Scaling instruction-finetuned language models [J ] . Journal of Machine Learning Research , 2024 , 25 ( 70 ): 1 - 53 .
WEI J , BOSMA M , ZHAO V Y , et al . Finetuned language models are zero-shot learners [J ] . arXiv Preprint , arXiv: 2109.01652 , 2021 .
KIM S , KANG H , CHOI S , et al . Large language models meet collaborative filtering: an efficient all-round LLM-based recommender system [J ] . arXiv Preprint , arXiv: 2404.11343 , 2024 .
KIKKISETTI D , MUSTAFA R U , MELILLO W , et al . Using LLMs to discover emerging coded antisemitic hate-speech in extremist social media [J ] . arXiv Preprint , arXiv: 2401.10841 , 2024 .
ISLAM T , GOLDWASSER D . Discovering latent themes in social media messaging: a machine-in-the-loop approach integrating LLMs [J ] . arXiv Preprint , arXiv: 2403.10707 , 2024 .
RADWAN A , AMARNEH M , ALAWNEH H , et al . Predictive analytics in mental health leveraging LLM embeddings and machine learning models for social media analysis [J ] . International Journal of Web Services Research , 2024 , 21 ( 1 ): 1 - 22 .
WANG Y , INKPEN D , GAMAARACHCHIGE P K . Explainable depression detection using large language models on social media data [C ] // Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology . Piscataway : IEEE Press , 2024 : 108 - 126 .
LI X L , LIANG P . Prefix-tuning: optimizing continuous prompts for generation [C ] // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing . Piscataway : IEEE Press , 2021 : 4582 - 4597 .
LESTER B , Al-RFOU R , CONSTANT N . The power of scale for parameter-efficient prompt tuning [C ] // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Piscataway : IEEE Press , 2021 : 3045 - 3059 .
HU E J , SHEN Y L , WALLIS P , et al . Lora: low-rank adaptation of large language models [J ] . arXiv Preprint , arXiv: 2106.09685 , 2021 .
LIU Y H , AGARWAL S , VENKATARAMAN S . AutoFreeze: automatically freezing model blocks to accelerate fine-tuning [J ] . arXiv Preprint , arXiv: 2102.01386 , 2021 .
CHRISTIANO P , LEIKE J , BROWN T B , et al . Deep reinforcement learning from human preferences [J ] . arXiv Preprint , arXiv: 1706.03741 , 2017 .
OUYANG L , WU J , JIANG X , et al . Training language models to follow instructions with human feedback [J ] . Advances in Neural Information Processing Systems , 2022 , 35 : 27730 - 27744 .
BAI Y T , KADAVATH S , KUNDU S , et al . Constitutional AI: harmlessness from AI feedback [J ] . arXiv Preprint , arXiv: 2212.08073 , 2022 .
TOUVRON H , MARTIN L , STONE K R , et al . Llama 2: open foundation and fine-tuned chat models [J ] . arXiv Preprint , arXiv: 2307.09288 , 2023 .
RAFAILOV R , SHARMA A , MITCHELL E , et al . Direct preference optimization: your language model is secretly a reward model [C ] // Proceedings of the 37th International Conference on Neural Information Processing Systems . Massachusetts : MIT Press , 2023 : 53728 - 53741 .
PAL A , KARKHANIS D , DOOLEY S , et al . Smaug: fixing failure modes of preference optimisation with DPO-positive [J ] . arXiv Preprint , arXiv: 2402.13228 , 2024 .
WU B , CHENG W H , ZHANG Y D , et al . Sequential prediction of social media popularity with deep temporal context networks [J ] . arXiv Preprint , arXiv: 1712.04443 , 2017 .
ACHIAM J , ADLER S , AGARWAL S , et al . Gpt-4 technical report [J ] . arXiv Preprint , arXiv: 2303.08774 , 2023 .
ZENG A H , LIU X , DU Z X , et al . GLM-130B: an open bilingual pre-trained model [J ] . arXiv Preprint , arXiv: 2210.02414 , 2022 .
DING M , YANG Z Y , HONG W Y , et al . CogView: mastering text-to-image generation via transformers [J ] . Advances in Neural Information Processing Systems , 2021 , 34 : 19822 - 19835 .
0
浏览量
603
下载量
0
CSCD
- 网络PDF下载
- WORD下载
- XML下载
- Meta-XML下载
- BibTeX下载
- Endnote下载
- RefMan 下载
- NoteExpress下载
- NoteFirst下载
关联资源
相关文章
相关作者
相关机构
AI问答- 技术支持由北京北大方正电子有限公司提供 京ICP备09064830号-19
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies
- 总访问量：584030 今日访问量：1846