Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services

Li’na DU; Li ZHUO; Shuo YANG; Jiafeng LI; Jing ZHANG

doi:10.11959/j.issn.1000-436x.2021178

您当前的位置：

首页 >

文章列表页 >

Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services

Comprehensive Reviews | 更新时间：2024-06-05

- Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services
- Journal on Communications Vol. 42, Issue 9, Pages: 205-217(2021)
- 作者机构：
  
  1. 北京工业大学计算智能与智能系统北京市重点实验室，北京 100124
  2. 北京工业大学信息学部，北京 100124
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61531006);Beijing Municipal Education Commission Cooperation Beijing Natural Science Foundation(KZ201910005007)
- DOI：10.11959/j.issn.1000-436x.2021178
  CLC： TP391
- Online First：2021-09，
  
  Published：25 September 2021
- 稿件说明：
移动端阅览
Li’na DU, Li ZHUO, Shuo YANG, et al. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services[J]. Journal on Communications, 2021, 42(9): 205-217.
DOI：

Li’na DU, Li ZHUO, Shuo YANG, et al. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services[J]. Journal on Communications, 2021, 42(9): 205-217. DOI： 10.11959/j.issn.1000-436x.2021178.

摘要

近几年来，随着HTTP自适应流媒体（HAS）视频数据集和网络轨迹数据集的不断推出，强化学习、深度学习等机器学习方法被不断应用到码率自适应（ABR）算法中，通过交互学习来确定码率控制的最优策略，取得了远超过传统启发式方法的性能。在分析 ABR 算法研究难点的基础上，重点阐述了基于强化学习（包括深度强化学习）的ABR算法研究进展。此外，总结了代表性的HAS视频数据集和网络轨迹数据集，介绍了算法性能的评价准则，最后探讨了ABR研究目前存在的问题和未来的方向。

Abstract

In recent years

with the continuous release of HTTP adaptive streaming (HAS) video datasets and network trace datasets

the machine learning methods

such as deep learning and reinforcement learning

have been continuously applied to adaptive bit rate (ABR) algorithms

which obtain the optimal strategy of rate control through interactive learning

and achieve superior performance that surpasses the traditional heuristic methods.Based on the analysis of the research difficulties of ABR algorithms

the research advances of ABR algorithms based on reinforcement learning (including deep reinforcement learning) was investigated.Furthermore

several representative HAS video datasets and network trace datasets were summarized

the evaluation metrics of the performance were depicted.Finally

the existing problems and the future tendency of ABR research were discussed.

关键词

Keywords

references

Cisco . Cisco visual networking index:Forecast and methodology [R ] . 2019 .

XIAO A L , LIU J , LI Y Z , et al . Two-phase rate adaptation strategy for improving real-time video QoE in mobile networks [J ] . China Communications , 2018 , 15 ( 10 ): 12 - 24 .

ITU-T . Definition of quality of experience,international telecommunication union,liaison statement,Ref:TD109rev2 (PLEN/12) [S ] . 2007 .

STOCKHAMMER T , . Dynamic adaptive streaming over HTTP:standards and design principles [C ] // Proceedings of the Second Annual ACM Conference on Multimedia Systems . New York:ACM Press , 2011 : 133 - 144 .

LEVKOV M . Video encoding and transcoding recommendations for HTTP dynamic streaming on the Adobe®Flash®Platform [R ] . 2010 .

FECHEYR L A . A review of HTTP live streaming [R ] . 2010 .

ZAMBELLI A . IIS smooth streaming technical overview [R ] . 2009 .

KUA J , ARMITAGE G , BRANCH P . A survey of rate adaptation techniques for dynamic adaptive streaming over HTTP [J ] . IEEE Communications Surveys ＆ Tutorials , 2017 , 19 ( 3 ): 1842 - 1866 .

AYAD I , IM Y , KELLER E , et al . A practical evaluation of rate adaptation algorithms in HTTP-based adaptive streaming [J ] . Computer Networks , 2018 , 133 : 90 - 103 .

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [M ] . Cambridge : MIT Press , 1998 .

FRANÇOIS L , HENDERSON P , ISLAM R , et al . An introduction to deep reinforcement learning [J ] . Now Publishers , 2018 , 11 ( 3-4 ): 219 - 354 .

CLAEYS M , LATRÉ S ,, FAMAEY J , et al . Design of a Q-learning based client quality selection algorithm for HTTP adaptive video streaming [C ] // Adaptive and Learning Agents Workshop (ALA) .[S.n.:s.l. ] , 2013 : 30 - 37 .

CLAEYS M , LATRÉ S ,, FAMAEY J , et al . Design and evaluation of a self-learning HTTP adaptive video streaming client [J ] . IEEE Communications Letters , 2014 , 18 ( 4 ): 716 - 719 .

SUN Y , YIN X Q , JIANG J C , et al . CS 2 P:improving video bitrate selection and adaptation with data-driven throughput prediction [C ] // Proceedings of the 2016 ACM SIGCOMM Conference.Florianopolis Brazil . New York:ACM Press , 2016 : 272 - 285 .

CHIARIOTTI F , D’ARONCO S ,, TONI L , et al . Online learning adaptation strategy for DASH clients [C ] // Proceedings of the 7th International Conference on Multimedia Systems . New York:ACM Press , 2016 : 1 - 12 .

ANDELIN T , CHETTY V , HARBAUGH D , et al . Quality selection for dynamic adaptive streaming over HTTP with scalable video coding [C ] // Proceedings of the 3rd Multimedia Systems Conference on –MMSys’12 . New York:ACM Press , 2012 : 149 - 154 .

LI Z , BEGEN A C , GAHM J , et al . Streaming video over HTTP with consistent quality [C ] // Proceedings of the 5th ACM Multimedia Systems Conference on – MMSys’14 . New York:ACM Press , 2014 : 248 - 258 .

KELLY F P , MAULLOO A K , TAN D K H . Rate control for communication networks:shadow prices,proportional fairness and stability [J ] . Journal of the Operational Research Society , 1998 , 49 ( 3 ): 237 - 252 .

YIN X Q , JINDAL A , SEKAR V , et al . A control-theoretic approach for dynamic adaptive video streaming over HTTP [J ] . ACM SIGCOMM Computer Communication Review , 2015 , 45 ( 4 ): 325 - 338 .

GARCÍA S , CABRERA J , GARCÍA N , . Quality-optimization algorithm based on stochastic dynamic programming for MPEG DASH video streaming [C ] // IEEE International Conference on Consumer Electronics (ICCE),Piscataway:IEEE Press , 2014 : 574 - 575 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Playing atari with deep reinforcement learning [J ] . arXiv Preprint,arXiv:1312.5602 , 2013 .

MAO H Z , NETRAVALI R , ALIZADEH M . Neural adaptive video streaming with Pensieve [C ] // Proceedings of the Conference of the ACM Special Interest Group on Data Communication . New York:ACM Press , 2017 : 197 - 210 .

MNIH V , BADIA A P , MIRZA M , et al . Asynchronous methods for deep reinforcement learning [C ] // International Conference on Machine Learning (ICML) . New York:ACM Press , 2016 : 1928 - 1937 .

PAUL C , HUDSON A . CS 244’18:recreating and extending pensieve [R ] . 2018 .

SENGUPTA S , GANGULY N , CHAKRABORTY S , et al . HotDASH:hotspot aware adaptive video streaming using deep reinforcement learning [C ] // 2018 IEEE 26th International Conference on Network Protocols (ICNP) . Piscataway:IEEE Press , 2018 : 165 - 175 .

GAO G Y , DONG L S , ZHANG H Z , et al . Content-aware personalised rate adaptation for adaptive streaming via deep video analysis [C ] // 2019 IEEE International Conference on Communications (ICC) . Piscataway:IEEE Press , 2019 : 1 - 8 .

HU S H , XU M , ZHANG H M , et al . Affective content-aware adaptation scheme on QoE optimization of adaptive streaming over HTTP [J ] . ACM Transactions on Multimedia Computing,Communications,and Applications , 2020 , 15 ( 3s ): 100 .

ZHANG X , OU Y Y , SEN S , et al . SENSEI:aligning video streaming quality with dynamic user sensitivity [C ] // USENIX Symposium on Networked Systems Design and Implementation (NSDI) . Berkeley:USENIX Association , 2021 : 303 - 320 .

DUANMU Z F , LIU W T , CHEN D Q , et al . A knowledge-driven quality-of-experience model for adaptive streaming videos [J ] . arXiv Preprint,arXiv:1911.07944 , 2019 .

XIAO A L , HUANG X F , WU S , et al . Traffic-aware rate adaptation for improving time-varying QoE factors in mobile video streaming [J ] . IEEE Transactions on Network Science and Engineering , 2020 , 7 ( 4 ): 2392 - 2405 .

HUANG T C , YAO X , WU C L , et al . Tiyuntsong:a self-play reinforcement learning approach for ABR video streaming [C ] // IEEE International Conference on Multimedia and Expo (ICME) . Piscataway:IEEE Press , 2019 : 1678 - 1683 .

HUANG T C , ZHANG R X , SUN L F . Self-play reinforcement learning for video transmission [C ] // Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video . New York:ACM Press , 2020 : 1 - 10 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [J ] . arXiv Preprint,arXiv:1707.06347 , 2017 .

HUO L Y , WANG Z L , XU M , et al . A meta-learning framework for learning multi-user preferences in QoE optimization of DASH [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 9 ): 3210 - 3225 .

FRANS K , HO J , CHEN X , et al . Meta learning shared hierarchies [C ] // International Conference on Learning Representations (ICLR) .[S.n.:s.l. ] , 2018 : 1 - 11 .

LIU J , TAO X M , LU J H . QoE-oriented rate adaptation for DASH with enhanced deep Q-learning [J ] . IEEE Access , 2019 , 7 : 8454 - 8469 .

SALEEM M , SALEEM Y , ASIF H M S , et al . Quality enhanced multimedia content delivery for mobile cloud with deep reinforcement learning [J ] . Wireless Communications and Mobile Computing , 2019 , 2019 : 1 - 15 .

HUANG T C , ZHOU C , ZHANG R X , et al . Comyco:quality-aware adaptive video streaming via imitation learning [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM Press , 2019 : 429 - 437 .

LI W H , HUANG J W , WANG S Q , et al . DAVS:dynamic-chunk quality aware adaptive video streaming using apprenticeship learning [C ] // GLOBECOM 2020 - 2020 IEEE Global Communications Conference . Piscataway:IEEE Press , 2020 : 1 - 6 .

GADALETA M , CHIARIOTTI F , ROSSI M , et al . D-DASH:a deep Q-learning framework for DASH video streaming [J ] . IEEE Transactions on Cognitive Communications and Networking , 2017 , 3 ( 4 ): 703 - 718 .

LEKHARU A , MOULII K Y , SUR A , et al . Deep learning based prediction model for adaptive video streaming [C ] // 2020 International Conference on Communication Systems ＆ Networks (COMSNETS) . Piscataway:IEEE Press , 2020 : 152 - 159 .

ZHANG G H , LEE J Y B . Ensemble adaptive streaming – A new paradigm to generate streaming algorithms via specializations [J ] . IEEE Transactions on Mobile Computing , 2020 , 19 ( 6 ): 1346 - 1358 .

ZHAO Y , SHEN Q W , LI W , et al . Latency aware adaptive video streaming using ensemble deep reinforcement learning [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM Press , 2019 : 2647 - 2651 .

AKHTAR Z , NAM Y S , GOVINDAN R , et al . Oboe:auto-tuning video ABR algorithms to network conditions [C ] // Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication . New York:ACM Press , 2018 : 44 - 58 .

YEO H , JUNG Y , KIM J , et al . Neural adaptive content-aware internet video delivery [C ] // Symposium on Operating Systems Design and Implementation (OSDI) .[S.n.:s.l. ] , 2018 : 645 - 661 .

HUANG T C , ZHANG R X , ZHOU C , et al . QARC:video quality aware rate control for real-time video streaming based on deep reinforcement learning [C ] // Proceedings of the 26th ACM international conference on Multimedia . New York:ACM Press , 2018 : 1208 - 1216 .

HUANG T C , ZHANG R X , WU C L , et al . Generalizing rate control strategies for realtime video streaming via learning from deep learning [C ] // MMAsia '19:Proceedings of the ACM Multimedia Asia . New York:ACM Press , 2019 : 1 - 6 .

TIAN Z , ZHAO L P , NIE L H , et al . Deeplive:QoE optimization for live video streaming through deep reinforcement learning [C ] // 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS) . Piscataway:IEEE Press , 2019 : 827 - 831 .

NASRABADI A T , PRAKASH R . Layer-assisted adaptive video streaming [C ] // Proceedings of the 28th ACM SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video.Amsterdam Netherlands . New York:ACM Press , 2018 : 31 - 36 .

LIU Y Z , JIANG B , GUO T , et al . Grad:learning for overhead-aware adaptive video streaming with scalable video coding [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York:ACM Press , 2020 : 1 - 9 .

WANG Y N , WANG H L , SHANG J Y , et al . RESA:a real-time evaluation system for ABR [C ] // 2019 IEEE International Conference on Multimedia and Expo (ICME) . Piscataway:IEEE Press , 2019 : 1846 - 1851 .

HUANG T C , ZHOU C , YAO X , et al . Quality-aware neural adaptive video streaming with lifelong imitation learning [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 10 ): 2324 - 2342 .

MA N N , ZHANG X Y , ZHENG H T , et al . ShuffleNet V2:practical guidelines for efficient CNN architecture design [C ] // Computer Vision- ECCV 2018 . Berlin:Springer , 2018 : 116 - 131 .

HUANG T C , ZHOU C , ZHANG R X , et al . Stick:a harmonious fusion of buffer-based and learning-based approach for adaptive streaming [C ] // IEEE INFOCOM 2020 - IEEE Conference on Computer Communications . Piscataway:IEEE Press , 2020 : 1967 - 1976 .

HUANG T Y , JOHARI R , MCKEOWN N , et al . A buffer-based approach to rate adaptation:Evidence from a large video streaming service [C ] // Special Interest Group on Data Communication (SIGCOMM) . New York:ACM Press , 2014 : 187 - 198 .

RIISER H , VIGMOSTAD P , GRIWODZ C , et al . Commute path bandwidth traces from 3G networks:analysis and applications [C ] // Proceedings of the 4th ACM Multimedia Systems Conference on - MMSys '13 . New York:ACM Press , 2013 : 114 - 118 .

US Federal Communications Commission . Measuring Fixed Broadband Report [R ] . 2016 .

RACA D , QUINLAN J J , ZAHRAN A H , et al . Beyond throughput:a 4G LTE dataset with channel and context metrics [C ] // Proceedings of the 9th ACM Multimedia Systems Conference . New York:ACM Press , 2018 : 460 - 465 .

HOOFT J V D , PETRANGELI S , WAUTERS T , et al . HTTP/2-based adaptive streaming of HEVC video over 4G/LTE networks [J ] . IEEE Communications Letters , 2016 , 20 ( 11 ): 2177 - 2180 .

MENG Z L , CHEN J , GUO Y N , et al . PiTree:practical implementation of ABR algorithms using decision trees [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM Press , 2019 : 2431 - 2439 .

MOORTHY A K , CHOI L K , BOVIK A C , et al . Video quality assessment on mobile devices:subjective,behavioral and objective studies [J ] . IEEE Journal of Selected Topics in Signal Processing , 2012 , 6 ( 6 ): 652 - 671 .

CHEN C , CHOI L K , VECIANA G D , et al . Modeling the time—varying subjective quality of HTTP video streams with rate adaptations [J ] . IEEE Transactions on Image Processing , 2014 , 23 ( 5 ): 2206 - 2221 .

GHADIYARAM D , BOVIK A C , YEGANEH H , et al . Study of the effects of stalling events on the quality of experience of mobile streaming videos [C ] // 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP) . Piscataway:IEEE Press , 2014 : 989 - 993 .

GHADIYARAM D , PAN J , BOVIK A C . A subjective and objective study of stalling events in mobile streaming videos [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2019 , 29 ( 1 ): 183 - 197 .

BAMPIS C G , LI Z , KATSAVOUNIDIS I , et al . Towards perceptually optimized adaptive video streaming-a realistic quality of experience database [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 5182 - 5197 .

DUANMU Z F , REHMAN A , WANG Z . A quality-of-experience database for adaptive video streaming [J ] . IEEE Transactions on Broadcasting , 2018 , 64 ( 2 ): 474 - 487 .

DUANMU Z F , CHEN D , LI Z W , et al . Assessing the quality-of-experience of adaptive bitrate video streaming [J ] . arXiv Preprint,arXiv:2008.08804 , 2020 .

BOKANI A , HASSAN M , KANHERE S S , et al . Comprehensive mobile bandwidth traces from vehicular networks [C ] // Proceedings of the 7th International Conference on Multimedia Systems . New York:ACM Press , 2016 : 344 - 348 .

RACA D , LEAHY D , SREENAN C J , et al . Beyond throughput,the next generation:a 5G dataset with channel and context metrics [C ] // Proceedings of the 11th ACM Multimedia Systems Conference . New York:ACM Press , 2020 : 303 - 308 .

SPITERI K , URGAONKAR R , SITARAMAN R K . BOLA:near-optimal bitrate adaptation for online videos [J ] . IEEE/ACM Transactions on Networking , 2020 , 28 ( 4 ): 1698 - 1711 .

JIANG J C , SEKAR V , ZHANG H . Improving fairness,efficiency,and stability in HTTP-based adaptive video streaming with festive [J ] . IEEE/ACM Transactions on Networking , 2014 , 22 ( 1 ): 326 - 340 .

Views

1619

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Survey of node localization scheme in underwater wireless sensor network

Overview on intelligent wireless communication technology

Advantage estimator based on importance sampling

Review of unsupervised anomaly detection techniques for medical imaging

Related Author

YANG Qiuling

TANG Zhichao

ZHU Rongxin

LI Wenjie

CHENG Liangliang

HUANG Xiangdang

Yingchang LIANG

Junjie TAN

Related Institution

School of Computer Science and Technology, Hainan University

Faculty of Science and Engineering, University of Groningen, Groningen 9747AG

National Key Laboratory of Science and Technology on Communications,University of Electronic Science and Technology of China

School of Computer Science and Engineering,Nanyang Technological University

School of Computer Science and Technology,Soochow University

AI问答

⁰