Enhanced deep deterministic policy gradient algorithm

Jianping CHEN; Chao HE; Quan LIU; Hongjie WU; Fuyuan HU; Qiming FU

doi:10.11959/j.issn.1000-436x.2018238

您当前的位置：

首页 >

文章列表页 >

Enhanced deep deterministic policy gradient algorithm

Papers | 更新时间：2024-06-05

- Enhanced deep deterministic policy gradient algorithm
- Journal on Communications Vol. 39, Issue 11, Pages: 106-115(2018)
- 作者机构：
  
  1. 苏州科技大学电子与信息工程学院，江苏苏州 215009
  2. 苏州科技大学江苏省建筑智慧节能重点实验室，江苏苏州 215009
  3. 苏州科技大学苏州市移动网络技术与应用重点实验室，江苏苏州 215009
  4. 苏州科技大学苏州市虚拟现实智能交互及应用技术重点实验室，江苏苏州 215009
  5. 苏州大学计算机科学与技术学院，江苏苏州 215006
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61502329);The National Natural Science Foundation of China(61772357);The National Natural Science Foundation of China(61750110519);The National Natural Science Foundation of China(61772355);The National Natural Science Foundation of China(61702055);The National Natural Science Foundation of China(61672371);The National Natural Science Foundation of China(61602334);The National Natural Science Foundation of China(61502323);The Natural Science Foundation of Jiangsu Province(BK20140283);The Key Research and Development Program of Jiangsu Province(BE2017663);The High School Natural Foundation of Jiangsu Province(13KJB520020);Suzhou Industrial Application of Basic Research Program(SYG201422)
- DOI：10.11959/j.issn.1000-436x.2018238
  CLC： TP391
- Online First：2018-11，
  
  Published：25 November 2018
- 稿件说明：
移动端阅览
Jianping CHEN, Chao HE, Quan LIU, et al. Enhanced deep deterministic policy gradient algorithm[J]. Journal on Communications, 2018, 39(11): 106-115.
DOI：

Jianping CHEN, Chao HE, Quan LIU, et al. Enhanced deep deterministic policy gradient algorithm[J]. Journal on Communications, 2018, 39(11): 106-115. DOI： 10.11959/j.issn.1000-436x.2018238.

摘要

针对深度确定策略梯度算法收敛速率较慢的问题，提出了一种增强型深度确定策略梯度（E-DDPG）算法。该算法在深度确定策略梯度算法的基础上，重新构建两个新的样本池——多样性样本池和高误差样本池。在算法执行过程中，训练样本分别从多样性样本池和高误差样本池按比例选取，以兼顾样本多样性以及样本价值信息，提高样本的利用效率和算法的收敛性能。此外，进一步从理论上证明了利用自模拟度量方法对样本进行相似性度量的合理性，建立值函数与样本相似性之间的关系。将E-DDPG算法以及DDPG算法用于经典的Pendulum问题和MountainCar问题，实验结果表明，E-DDPG具有更好的收敛稳定性，同时具有更快的收敛速率。

Abstract

With the problem of slow convergence for deep deterministic policy gradient algorithm

an enhanced deep deterministic policy gradient algorithm was proposed.Based on the deep deterministic policy gradient algorithm

two sample pools were constructed

and the time difference error was introduced.The priority samples were added when the experience was played back.When the samples were trained

the samples were selected from two sample pools respectively.At the same time

the bisimulation metric was introduced to ensure the diversity of the selected samples and improve the convergence rate of the algorithm.The E-DDPG algorithm was used to pendulum problem.The experimental results show that the E-DDPG algorithm can effectively improve the convergence performance of the continuous action space problems and have better stability.

关键词

Keywords

references

SUTTON R S , BARTO G A . Reinforcement learning:an introduction [M ] . Cambridge : MIT pressPress , 1998 .

朱斐 , 刘全 , 傅启明 , 等 . 一种用于连续动作空间的最小二乘行动者-评论家方法 [J ] . 计算机研究与发展 , 2014 , 51 ( 3 ): 548 - 558 .

ZHU F , LIU Q , FU Q M . A least square actor-critic approach for continuous action space [J ] . Journal of Computer Research and Development , 2014 , 51 ( 3 ): 548 - 558 .

孙志军 , 薛磊 , 许阳明 , 等 . 深度学习研究综述 [J ] . 计算机应用研究 , 2012 , 29 ( 8 ): 2806 - 2810 .

SUN Z J , XUE L , XU Y M , et al . Overview of deep learning [J ] . Application Research of Computers , 2012 , 29 ( 8 ): 2806 - 2810 .

LECUN Y , BENGIO Y , HINTON G . Deep learning [J ] . Nature , 2015 , 521 ( 7553 ): 436 - 444 .

HINTON G E , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets [J ] . Neural Computation , 2006 , 18 ( 7 ): 1527 - 1554 .

SILVER D , HUANG A , MADDISON C J , et al . Mastering the game of Go with deep neural networks and tree search [J ] . Nature , 2016 , 529 ( 7587 ): 484 - 489 .

SILVER D , SCHRITTWIESER J , SIMONYAN K , et al . Mastering the game of go without human knowledge [J ] . Nature , 2017 , 550 ( 7676 ): 354 - 359 .

MNIH V , KAVUKCUOFLU K , SILVER D , et al . Playing atari with deep reinforcement learning [C ] // Workshops at the 26th Neural Information Processing Systems . 2013 .

MNIH V , KAVUKCUOFLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

WATKINS C J C H . Learning from delayed rewards [J ] . Robotics and Autonomous Systems , 1989 , 15 ( 4 ): 233 - 235 .

VAN H V , GUEZ A , SILVER D . Deep reinforcement learning with double q-learning [C ] // The AAAI Conference on Artificial Intelligence . 2016 .

HASSELT H V , . Double Q-learning [C ] // The Advances in Neural Information Processing Systems . 2010 .

SCHAUL T , QUAN J , ANTONOGLOU I , et al . Prioritized experience replay [C ] // The 4th International Conference on Learning Representations . 2016 : 322 - 355 .

SUTTON R S , MCALLESTER D , SINGH S , et al . Policy gradient methods for reinforcement learning with function approximation [J ] . Advances in Neural Information Processing Systems , 2000 , 12 : 1057 - 1063 .

LILLICRAP T P , HUNT J J , PRITZEL A , et al . Continuous control with deep reinforcement learning [C ] // The 4th International Conference on Learning Representations . 2015 .

SILVER D , LEVER G , HEESS N , et al . Deterministic policy gradient algorithms [C ] // The International Conference on Machine Learning . 2014 .

GIVAN R , DEAN T , GREIG M . Equivalence notions and model minimization in Markov decision processes [J ] . Artificial Intelligence , 2003 , 147 ( 1-2 ): 163 - 223 .

FERNS N , PANANGADEN P , PRECUP D . Metrics for finite markov decision processes [C ] // The 20th Conference on Uncertainty in Artificial Intelligence . 2004 .

Views

1986

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

GAT-based decision mechanism for decentralized joint routing and spectrum access

Multi-cluster computing power resource scheduling algorithm based on DDPG reinforcement learning

Task offloading and resource allocation strategy for vehicular edge computing assisted by intelligent reflecting surfaces

D3QN-based collaborative offloading algorithm for vehicular networks assisted by digital twins

Storage resource scheduling optimization method for separated data center based on deep reinforcement learning

Related Author

Zhou Zibo

Ren Baoquan

Zhong Xudong

Liu Qi

Qin Zhen

HU Yahui

WANG Yuelin

ZHANG Chenkang

Related Institution

Systems Engineering Institute, Academy of Military Science

Air Force Early Warning Academy

School of Artificial Intelligence, China University of Mining and Technology-Beijing

Computer Network information Center, Chinese Academy of Sciences

CHN Energy Zhi Shen Control Technology Co., Ltd.

AI问答

⁰