Actor-critic algorithm with incremental dual natural policy gradient

Peng ZHANG; Quan LIU; Shan ZHONG; Jian-wei ZHAI; Wei-sheng QIAN

doi:10.11959/j.issn.1000-436x.2017089

您当前的位置：

首页 >

文章列表页 >

Actor-critic algorithm with incremental dual natural policy gradient

Correspondences | 更新时间：2024-06-05

- Actor-critic algorithm with incremental dual natural policy gradient
- Journal on Communications Vol. 38, Issue 4, Pages: 166-177(2017)
- 作者机构：
  
  1. 苏州大学计算机科学与技术学院，江苏苏州 215006
  2. 软件新技术与产业化协同创新中心，江苏南京 210000
  3. 吉林大学符号计算与知识工程教育部重点实验室，吉林长春 130012
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61272005);The National Natural Science Foundation of China(61303108);The National Natural Science Foundation of China(61373094);The National Natural Science Foundation of China(61472262);The National Natural Science Foundation of China(61502323);The National Natural Science Foundation of China(61502329);The Natural Science Foundation of Jiangsu Province(BK2012616);The High School Natural Foundation of Jiangsu Province(13KJB520020);The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education of Jilin University(93K172014K04);Suzhou Industrial Application of Basic Research Program(SYG201422);Suzhou Industrial Application of Basic Research Program(SYG201308)
- DOI：10.11959/j.issn.1000-436x.2017089
  CLC： TP181
- Online First：2017-04，
  
  Published：25 April 2017
- 稿件说明：
移动端阅览
Peng ZHANG, Quan LIU, Shan ZHONG, et al. Actor-critic algorithm with incremental dual natural policy gradient[J]. Journal on Communications, 2017, 38(4): 166-177.
DOI：

Peng ZHANG, Quan LIU, Shan ZHONG, et al. Actor-critic algorithm with incremental dual natural policy gradient[J]. Journal on Communications, 2017, 38(4): 166-177. DOI： 10.11959/j.issn.1000-436x.2017089.

摘要

针对强化学习中已有连续动作空间算法未能充分考虑最优动作的选取方法和利用动作空间的知识，提出一种对自然梯度进行改进的行动者评论家算法。该算法采用最大化期望回报作为目标函数，对动作区间上界和下界进行加权来求最优动作，然后通过线性函数逼近器来近似动作区间上下界的权值，将最优动作求解转换为对双策略参数向量的求解。为了加快上下界的参数向量学习速率，设计了增量的Fisher信息矩阵和动作上下界权值的资格迹，并定义了双策略梯度的增量式自然行动者评论家算法。为了证明该算法的有效性，将该算法与其他连续动作空间的经典强化学习算法在3个强化学习的经典测试实验中进行比较。实验结果表明，所提算法具有收敛速度快和收敛稳定性好的优点。

Abstract

The existed algorithms for continuous action space failed to consider the way of selecting optimal action and utilizing the knowledge of the action space

so an efficient actor-critic algorithm was proposed by improving the natural gradient.The objective of the proposed algorithm was to maximize the expected return.Upper and the lower bounds of the action range were weighted to obtain the optimal action.The two bounds were approximated by linear function.Afterward

the problem of obtaining the optimal action was transferred to the learning of double policy parameter vectors.To speed the learning

the incremental Fisher information matrix and the eligibilities of both bounds were designed.At three reinforcement learning problems

compared with other representative methods with continuous action space

the simulation results show that the proposed algorithm has the advantages of rapid convergence rate and high convergence stability.

关键词

Keywords

references

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [M ] . Cambridge Massachusetts : MIT pressPress , 1998 .

BUSONIU L , BABUSKA R , SCHUTTER B D , et al . Reinforcement learning and dynamic programming using function approximators [M ] . Florida : CRC PressPress , 2010 .

LEE D , SEO H , JUNG M W . Neural basis of reinforcement learning and decision making [J ] . Annual Review of Neuroscience , 2012 , 35 ( 5 ): 287 - 308 .

WIERING M , VAN O M . Reinforcement learning:STATE-OFTHE-Art [M ] . Berlin Heidelberg:Springer . 2014 .

SUTTON R S , MCALLESTER D A , SINGH S P , et al . Policy gradient methods for reinforcement learning with function approximation [C ] // NIPS . 1999 , 99 : 1057 - 1063 .

PETERS J , SCHAAL S . NATURAL A C [J ] . Neurocomputing , 2008 , 71 ( 7-9 ): 1180 - 1190 .

PETERS J , VIJAYAKUMAR S , SCHAAL S . Reinforcement learning for humanoid robotics [J ] . Autonomous Robot , 2003 , 12 ( 1 ): 1 - 20 .

VAN H H . Reinforcement learning in continuous state and action spaces [M ] . Reinforcement Learning . Springer Berlin Heidelberg , 2012 : 207 - 251 .

WIERSTRA D , SCHAUL T , PETERS J , et al . Natural evolution strategies [C ] // 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence) . 2008 : 3381 - 3387 .

SUN Y , WIERSTRA D , SCHAUL T , et al . Efficient natural evolution strategies [C ] // The 11th Annual Conference on Genetic and Evolutionary Computation . 2009 : 539 - 546 .

RUBINSTEIN R Y , KROESE D P . The cross-entropy method [J ] . Information Science ＆ Statistics , 2008 , 50 ( 1 ): 92 - 92 .

BOTEV Z I , KROESE D P , RUBINSTEIN R Y , et al . The cross-entropy method for optimization [J ] . Machine Learning:Theory and Applications,Chennai:Elsevier BV , 2013 , 31 : 35 - 59 .

MARTIN H J A , DE LOPE J . x＜α＞:an effective algorithm for continuous actions reinforcement learning problems [C ] // The IEEE Industrial Electronics Society . 2009 : 2063 - 2068 .

LILLICRAP T P , HUNT J J , PRITZEL A , et al . Continuous control with deep reinforcement learning [J ] . Computer Science , 2015 , 8 ( 6 ): A187 .

GU S , LILLICRAP T , SUTSKEVER I , et al . Continuous deep Q-learning with model-based acceleration [J ] . arXiv Preprint arXiv:1603.00748 , 2016 .

KHAMASSI M , TZAFESTAS C . Active exploration in parameterized reinforcement learning [J ] . arXiv preprint arXiv:1610 , 2016 .

BHATNAGAR S , GHAVAMZADEH M , LEE M , et al . Incremental natural actor-critic algorithms [C ] // Advances in neural information processing systems . 2007 : 105 - 112 .

VIJAY R , KONDA , JOHN N. . Tsitsiklis.actor-critic algorithms [J ] . Siam Journal on Control ＆ Optimization , 2001 , 42 ( 4 ): 1008 - 1014 .

BERENJI H R , KHEDKAR P . Learning and tuning fuzzy logic controllers through reinforcements [J ] . IEEE Transactions on Neural Networks , 1992 , 3 ( 5 ): 724 - 740 .

SINGH S P , SUTTON R S . Reinforcement learning with replacing eligibility traces [J ] . Machine Learning , 1996 , 22 ( 1-3 ): 123 - 158 .

SUTTON R S . Generalization in reinforcement learning:successful examples using sparse coarse coding [J ] . Neural Information Processing Systems , 1996 : 1038 - 1044 .

Views

1852

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Mergeable adaptive tile coding method

RL-WGAN based method for 5G network anomalous data generation

Study on Co-EDCA mechanism for multi-AP collaboration in FTTR C-WAN architecture

Design and implementation of an IPv6+ based intelligent computing-network scheduling scheme for Internet of vehicles

Survey of node localization scheme in underwater wireless sensor network

Related Author

Meng-yu SHI

Qi-ming FU

Ning Zhaolong

Zou Daoyuan

Zhou Li

Ouyang Ruiqi

Xiong Xuanrui

WU Weimin

Related Institution

College of Electronic Science and Technology, National University of Defense Technology

School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications

School of Electronic Information and Communication, Huazhong University of Science and Technology

School of Information and Communication Engineering, Beijing University of Posts and Telecommunications

China Unicom Research Institute

AI问答

⁰