基于多智能体元强化学习的车联网协同服务缓存和计算卸载

宁兆龙; 张凯源; 王小洁; 郭磊

doi:10.11959/j.issn.1000-436x.2021104

您当前的位置：

首页 >

文章列表页 >

基于多智能体元强化学习的车联网协同服务缓存和计算卸载

学术论文 | 更新时间：2024-06-05

- 基于多智能体元强化学习的车联网协同服务缓存和计算卸载
- Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning
- 通信学报 2021年42卷第6期页码：118-130
- 作者机构：
  
  1. 重庆邮电大学通信与信息工程学院，重庆 400065
  2. 大连理工大学软件学院，辽宁大连 116620
- 作者简介：
  
  [ "宁兆龙（1986− ），男，辽宁沈阳人，博士，重庆邮电大学教授，主要研究方向为边缘计算、车联网、网络优化等" ]
  [ "张凯源（1994− ），男，黑龙江哈尔滨人，大连理工大学硕士生，主要研究方向为人工智能、边缘计算" ]
  [ "王小洁（1988− ），女，河北张家口人，博士，重庆邮电大学特聘教授，主要研究方向为物联网、人工智能、边缘计算" ]
  [ "郭磊（1980− ），男，四川眉山人，博士，重庆邮电大学教授，主要研究方向为网络优化、网络通信、光网络等" ]
- 基金信息：
  
  国家重点研发计划基金资助项目(2018YFE0206800);国家自然科学基金资助项目(62025105);国家自然科学基金资助项目(61971084);国家自然科学基金资助项目(62001073);重庆英才计划基金资助项目(CQYC2020058659)
- DOI：10.11959/j.issn.1000-436x.2021104
  中图分类号： TN92
- 网络出版日期：2021-06，
  
  纸质出版日期：2021-06-25
- 稿件说明：
移动端阅览
宁兆龙, 张凯源, 王小洁, 等. 基于多智能体元强化学习的车联网协同服务缓存和计算卸载[J]. 通信学报, 2021,42(6):118-130.

Zhaolong NING, Kaiyuan ZHANG, Xiaojie WANG, et al. Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning[J]. Journal on communications, 2021, 42(6): 118-130.
宁兆龙, 张凯源, 王小洁, 等. 基于多智能体元强化学习的车联网协同服务缓存和计算卸载[J]. 通信学报, 2021,42(6):118-130. DOI： 10.11959/j.issn.1000-436x.2021104.

Zhaolong NING, Kaiyuan ZHANG, Xiaojie WANG, et al. Cooperative service caching and peer offloading in Internet of vehicles based on multi-agent meta-reinforcement learning[J]. Journal on communications, 2021, 42(6): 118-130. DOI： 10.11959/j.issn.1000-436x.2021104.

摘要

为了降低求解优化问题的难度，提出一种双层的多路侧单元（RSU）协同缓存框架将问题进行解耦。外层采用多智能体元强化学习方法，在每个本地智能体进行决策学习的同时，采用长短期记忆网络作为元智能体来平衡本地决策并加速学习过程，从而得到最优的 RSU 缓存策略；内层采用拉格朗日乘子法求解最佳协同卸载策略，实现 RSU 间的任务分配。基于杭州真实交通数据的实验表明，所提方法具有理想的能效性能，并且能够在大规模任务流下保持网络稳健性。

Abstract

In order to reduce computation complexity

a two-layer mutli-RSU (road side unit) service caching and peer offloading algorithm (MPO) was proposed to decouple the optimization problem.In the designed MPO

the outer layer utilized multi-agent meta-reinforcement learning

which established long short-term memory (LSTM) network as the meta-agent to balance decisions of local agents and accelerate learning progress.The inner layer utilized lagrange multiplier method to achieve optimal decision for RSU peer offloading.Experimental results based on real traffic data in Hangzhou demonstrate that the proposed method outperforms other methods and remains robust under large-scale workloads.

关键词

Keywords

references

NING Z L , DONG P R , WANG X J , et al . Mobile edge computing enabled 5G health monitoring for Internet of medical things:a decentralized game theoretic approach [J ] . IEEE Journal on Selected Areas in Communications , 2021 , 39 ( 2 ): 463 - 478 .

谢人超 , 廉晓飞 , 贾庆民 , 等 . 移动边缘计算卸载技术综述 [J ] . 通信学报 , 2018 , 39 ( 11 ): 138 - 155 .

XIE R C , LIAN X F , JIA Q M , et al . Survey on computation offloading in mobile edge computing [J ] . Journal on Communications , 2018 , 39 ( 11 ): 138 - 155 .

ZHUANG W H , YE Q , LYU F , et al . SDN/NFV-empowered future IoV with enhanced communication,computing,and caching [J ] . Proceedings of the IEEE , 2020 , 108 ( 2 ): 274 - 291 .

WANG X J , NING Z L , GUO S . Multi-agent imitation learning for pervasive edge computing:a decentralized computation offloading algorithm [J ] . IEEE Transactions on Parallel and Distributed Systems , 2021 , 32 ( 2 ): 411 - 425 .

刘雷 , 陈晨 , 冯杰 , 等 . 车载边缘计算中任务卸载和服务缓存的联合智能优化 [J ] . 通信学报 , 2021 , 42 ( 1 ): 18 - 26 .

LIU L , CHEN C , FENG J , et al . Joint intelligent optimization of task offloading and service caching for vehicular edge computing [J ] . Journal on Communications , 2021 , 42 ( 1 ): 18 - 26 .

CHEN Z Q , DUAN L Y , WANG S Q , et al . Toward knowledge as a service over networks:a deep learning model communication paradigm [J ] . IEEE Journal on Selected Areas in Communications , 2019 , 37 ( 6 ): 1349 - 1363 .

张彦 , 张科 , 曹佳钰 . 边缘智能驱动的车联网 [J ] . 物联网学报 , 2018 , 2 ( 4 ): 40 - 48 .

ZHANG Y , ZHANG K , CAO J Y . Internet of vehicles empowered by edge intelligence [J ] . Chinese Journal on Internet of Things , 2018 , 2 ( 4 ): 40 - 48 .

WANG X J , NING Z L , GUO S , et al . Imitation learning enabled task scheduling for online vehicular edge computing [J ] . IEEE Transactions on Mobile Computing , 2020 , PP ( 99 ): 1 .

NING Z L , DONG P R , WANG X J , et al . When deep reinforcement learning meets 5G-enabled vehicular networks:a distributed offloading framework for traffic big data [J ] . IEEE Transactions on Industrial Informatics , 2020 , 16 ( 2 ): 1352 - 1361 .

NING Z L , DONG P R , WANG X J , et al . Deep reinforcement learning for vehicular edge computing:an intelligent offloading system [J ] . ACM Transactions on Intelligent Systems and Technology , 2019 , 10 ( 6 ): 60 .

TRAN T X , HAJISAMI A , PANDEY P , et al . Collaborative mobile edge computing in 5G networks:new paradigms,scenarios,and challenges [J ] . IEEE Communications Magazine , 2017 , 55 ( 4 ): 54 - 61 .

亓晋 , 孙海蓉 , 巩锟 , 等 . 移动边缘计算中基于信誉值的智能计算卸载模型研究 [J ] . 通信学报 , 2020 , 41 ( 7 ): 141 - 151 .

QI J , SUN H R , GONG K , et al . Research on intelligent computing offloading model based on reputation value in mobile edge computing [J ] . Journal on Communications , 2020 , 41 ( 7 ): 141 - 151 .

XU J , CHEN L X , ZHOU P . Joint service caching and task offloading for mobile edge computing in dense networks [C ] // IEEE INFOCOM 2018 - IEEE Conference on Computer Communications . Piscataway:IEEE Press , 2018 : 207 - 215 .

WANG X J , NING Z L , WANG L . Offloading in Internet of vehicles:a fog-enabled real-time traffic management system [J ] . IEEE Transactions on Industrial Informatics , 2018 , 14 ( 10 ): 4568 - 4578 .

CHEN M , HAO Y X . Task offloading for mobile edge computing in software defined ultra-dense network [J ] . IEEE Journal on Selected Areas in Communications , 2018 , 36 ( 3 ): 587 - 597 .

TAN L T , HU R Q , HANZO L . Twin-timescale artificial intelligence aided mobility-aware edge caching and computing in vehicular networks [J ] . IEEE Transactions on Vehicular Technology , 2019 , 68 ( 4 ): 3086 - 3099 .

DAI Y Y , XU D , MAHARJAN S , et al . Joint computation offloading and user association in multi-task mobile edge computing [J ] . IEEE Transactions on Vehicular Technology , 2018 , 67 ( 12 ): 12313 - 12325 .

TAO O Y , ZHI Z , XU C . Follow me at the edge:mobility-aware dynamic service placement for mobile edge computing [C ] // 2018 IEEE/ACM 26th International Symposium on Quality of Service . Piscataway:IEEE Press , 2018 : 1 - 10 .

NING Z L , DONG P R , KONG X J , et al . A cooperative partial computation offloading scheme for mobile edge computing enabled Internet of Things [J ] . IEEE Internet of Things Journal , 2019 , 6 ( 3 ): 4804 - 4814 .

QIAO G H , LENG S P , ZHANG K , et al . Collaborative task offloading in vehicular edge multi-access networks [J ] . IEEE Communications Magazine , 2018 , 56 ( 8 ): 48 - 54 .

NEWELL G F . Simple queueing systems [M ] . Dordrecht : Springer Netherlands , 1982 .

HERINGS J J , PEETERS R . Stationary equilibria in stochastic games:structure,selection,and computation [J ] . Research Memorandum , 2000 , 118 ( 1 ): 32 - 60 .

浏览量

741

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

内容新鲜度保障的车联网多智能体缓存分发策略

面向新能源汽车能源交易系统的PoRT共识机制

基于新型可净化多重签名的车联网高效假名证书分发方案

面向双层区块链的人车分离信任管理方案

车联网联邦学习的数据异质性问题及基于个性化的解决方法综述