Advantage estimator based on importance sampling

Quan LIU; Yubin JIANG; Zhihui HU

doi:10.11959/j.issn.1000-436x.2019122

您当前的位置：

首页 >

文章列表页 >

Advantage estimator based on importance sampling

Papers | 更新时间：2024-06-05

- Advantage estimator based on importance sampling
- Journal on Communications Vol. 40, Issue 5, Pages: 108-116(2019)
- 作者机构：
  
  1. 苏州大学计算机科学与技术学院，江苏苏州 215006
  2. 苏州大学江苏省计算机信息处理技术重点实验室，江苏苏州 215006
  3. 吉林大学符号计算与知识工程教育部重点实验室，吉林长春 130012
  4. 软件新技术与产业化协同创新中心，江苏南京 210093
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61772355);The National Natural Science Foundation of China(61702055);The National Natural Science Foundation of China(61472262);The National Natural Science Foundation of China(61502323);The National Natural Science Foundation of China(61502329);Jiangsu Province Natural Science Research University Major Projects(18KJA520011);Jiangsu Province Natural Science Research University Major Projects(17KJA520004);The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education of Jilin University(93K172014K04);Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education,Jilin University(93K172017K18);Suzhou Industrial Application of Basic Research Program(SYG201422)
- DOI：10.11959/j.issn.1000-436x.2019122
  CLC： TP391
- Online First：2019-05，
  
  Published：25 May 2019
- 稿件说明：
移动端阅览
Quan LIU, Yubin JIANG, Zhihui HU. Advantage estimator based on importance sampling[J]. Journal on Communications, 2019, 40(5): 108-116.
DOI：

Quan LIU, Yubin JIANG, Zhihui HU. Advantage estimator based on importance sampling[J]. Journal on Communications, 2019, 40(5): 108-116. DOI： 10.11959/j.issn.1000-436x.2019122.

摘要

在连续动作任务中，深度强化学习通常采用高斯分布作为策略函数。针对高斯分布策略函数由于截断动作导致算法收敛速度变慢的问题，提出了一种重要性采样优势估计器（ISAE）。该估计器在通用优势估计器（GAE）的基础上，引入了重要性采样机制，通过计算边界动作的目标策略与行动策略比率修正截断动作带来的值函数偏差，提高了算法的收敛速度。此外，ISAE引入了L参数，通过限制重要性采样率的范围，提高了样本的可靠度，保证了网络参数的稳定。为了验证ISAE的有效性，将ISAE与近端策略优化结合并与其他算法在MuJoCo平台上进行比较。实验结果表明，ISAE具有更快的收敛速度。

Abstract

In continuous action tasks

deep reinforcement learning usually uses Gaussian distribution as a policy function.Aiming at the problem that the Gaussian distribution policy function slows down due to the clipped action

an importance sampling advantage estimator was proposed.Based on the general advantage estimator

an importance sampling mechanism was introduced by the estimator to improve the convergence speed of the algorithm and correct the deviation of the value function caused by calculating the target strategy and action strategy ratio of the boundary action.In addition

the L parameter was introduced by ISAE which improved the reliability of the sample and limited the stability of the network parameters by limiting the range of the importance sampling rate.In order to verify the effectiveness of the ISAE

applying it to proximal policy optimization and comparing it with other algorithms on the MuJoCo platform.Experimental results show that ISAE has a faster convergence rate.

关键词

Keywords

references

SUTTON R S , BARTO A G . Introduction to reinforcement learning [M ] . Cambridge : MIT pressPress , 1998 .

刘全 , 傅启明 , 龚声蓉 . 最小状态变元平均奖赏的强化学习方法 [J ] . 通信学报 , 2011 , 32 ( 1 ): 66 - 71 .

LIU Q , FU Q M , GONG S R . Reinforcement learning algorithm based on minimum state method and average reward [J ] . Journal on Communications , 2011 , 32 ( 1 ): 66 - 71 .

TANG J , DENG C , HUANG G B . Extreme learning machine for multilayer perceptron [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2016 , 27 ( 4 ): 809 - 821 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks [C ] // Advances in Neural Information Processing Systems . 2012 : 1097 - 1105 .

VEERIAH V , VAN S H , SUTTON R S . Forward actor-Critic for nonlinear function approximation in reinforcement learning [C ] // Conference on Autonomous Agents and Multiagent Systems . 2017 : 556 - 564 .

LECUN Y , BENGIO Y , HINTON G . Deep learning [J ] . Nature , 2015 , 521 ( 7553 ): 436 - 444 .

MNIH V , KAVUKCUOFLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

MNIH V , BADIA A P , MIRZA M , et al . Asynchronous methods for deep reinforcement learning [C ] // International Conference on Machine Learning . 2016 : 1928 - 1937 .

VAN H , GUEZ A , SILVER D . Deep reinforcement learning with double Q-learning [C ] // Thirtieth AAAI Conference on Artificial Intelligence . 2016 : 2094 - 2100 .

WANG Z , SCHAUL T , HESSEL M , et al . Dueling network architectures for deep reinforcement learning [C ] // International Conference on Machine Learning . 2016 : 1995 - 2003 .

SAMEJIMA K , DOYA K , KAWATO M . Inter-module credit assignment in modular reinforcement learning [J ] . Neural Networks , 2003 , 16 ( 7 ): 985 - 994 .

SINGH S P , SUTTON R S . Reinforcement learning with replacing eligibility traces [J ] . Machine Learning , 1996 , 22 ( 1-3 ): 123 - 158 .

WATKINS C J C H . Learning from delayed rewards [D ] . Cambridge:King’s College , 1989 .

SUTTON R S . Temporal credit assignment in reinforcement learning [D ] . Amherst:University of Massachusetts , 1984 .

VAN S H , MAHMOOD A R , PILARSKI P M , et al . True online temporal-difference learning [J ] . The Journal of Machine Learning Research , 2016 , 17 ( 1 ): 5057 - 5096 .

HO J , ERMON S . Generative adversarial imitation learning [C ] // Advances in Neural Information Processing Systems . 2016 : 4565 - 4573 .

MNIH V , BADIA A P , MIRZA M , et al . Asynchronous methods for deep reinforcement learning [C ] // International Conference on Machine Learning . 2016 : 1928 - 1937 .

SCHULMAN J , LEVINE S , ABBEEL P , et al . Trust region policy optimization [C ] // International Conference on Machine Learning . 2015 : 1889 - 1897 .

CHUA K , CALANDRA R , MCALLISTER R , et al . Deep reinforcement learning in a handful of trials using probabilistic dynamics models [C ] // Advances in Neural Information Processing Systems . 2018 .

FUJITA Y , MAEDA S . Clipped action policy gradient [C ] // International Conference on Machine Learning . 2018 : 1592 - 1601 .

THODOROFF P , DURAND A , PINEAU J , et al . Temporal regularization for Markov decision process [C ] // Advances in Neural Information Processing Systems . 2018 : 1779 - 1789 .

DOYA K . Reinforcement learning in continuous time and space [J ] . Neural Computation , 2000 , 12 ( 1 ): 219 - 245 .

HESSEL M , MODAYIL J , VAN H H , et al . Rainbow:combining improvements in deep reinforcement learning [C ] // Thirty-Second AAAI Conference on Artificial Intelligence . 2018 : 3215 - 3222 .

Views

1768

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services

GAT-based decision mechanism for decentralized joint routing and spectrum access

RL-WGAN based method for 5G network anomalous data generation

Multi-cluster computing power resource scheduling algorithm based on DDPG reinforcement learning

Study on Co-EDCA mechanism for multi-AP collaboration in FTTR C-WAN architecture

Related Author

Li’na DU

Li ZHUO

Shuo YANG

Jiafeng LI

Jing ZHANG

Zhou Zibo

Ren Baoquan

Zhong Xudong

Related Institution

Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology

Information Department, Beijing University of Technology

Systems Engineering Institute, Academy of Military Science

Air Force Early Warning Academy

School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications

AI问答

⁰