Heuristic Sarsa algorithm based on value function transfer

Jianping CHEN; Zhengxia YANG; Quan LIU; Hongjie WU; Yang XU; Qiming FU

doi:10.11959/j.issn.1000-436x.2018133

您当前的位置：

首页 >

文章列表页 >

Heuristic Sarsa algorithm based on value function transfer

Artificial Intelligence and Network Security | 更新时间：2024-06-05

- Heuristic Sarsa algorithm based on value function transfer
- Journal on Communications Vol. 39, Issue 8, Pages: 37-47(2018)
- 作者机构：
  
  1. 苏州科技大学电子与信息工程学院，江苏苏州 215009
  2. 苏州科技大学江苏省建筑智慧节能重点实验室，江苏苏州 215009
  3. 苏州科技大学苏州市移动网络技术与应用重点实验室，江苏苏州 215009
  4. 苏州大学计算机科学与技术学院，江苏苏州 215000
  5. 浙江纺织服装职业技术学院信息工程学院，浙江宁波 315000
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61502329);The National Natural Science Foundation of China(61772357);The National Natural Science Foundation of China(61750110519);The National Natural Science Foundation of China(61772355);The National Natural Science Foundation of China(61702055);The National Natural Science Foundation of China(61672371);The National Natural Science Foundation of China(61602334);The Natural Science Foundation of Jiangsu Province(BK20140283);The Key Research and Development Program of Jiangsu Province(BE2017663);The High School Natural Foundation of Jiangsu Province(13KJB520020);Suzhou Industrial Application of Basic Research Program(SYG201422)
- DOI：10.11959/j.issn.1000-436x.2018133
  CLC： TP391
- Online First：2018-08，
  
  Published：25 August 2018
- 稿件说明：
移动端阅览
Jianping CHEN, Zhengxia YANG, Quan LIU, et al. Heuristic Sarsa algorithm based on value function transfer[J]. Journal on Communications, 2018, 39(8): 37-47.
DOI：

Jianping CHEN, Zhengxia YANG, Quan LIU, et al. Heuristic Sarsa algorithm based on value function transfer[J]. Journal on Communications, 2018, 39(8): 37-47. DOI： 10.11959/j.issn.1000-436x.2018133.

摘要

针对 Sarsa 算法存在的收敛速度较慢的问题，提出一种改进的基于值函数迁移的启发式 Sarsa 算法（VFT-HSA）。该算法将Sarsa算法与值函数迁移方法相结合，引入自模拟度量方法，在相同的状态空间和动作空间下，对新任务与历史任务之间的不同状态进行相似性度量，对满足条件的历史状态进行值函数迁移，提高算法的收敛速度。此外，该算法结合启发式探索方法，引入贝叶斯推理，结合变分推理衡量信息增益，并运用获取的信息增益构建内在奖赏函数作为探索因子，进而加快算法的收敛速度。将所提算法用于经典的Grid World问题，并与Sarsa算法、Q-Learning算法以及收敛性能较好的VFT-Sarsa算法、IGP-Sarsa算法进行比较，实验表明，所提算法具有较快的收敛速度和较好的稳定性。

Abstract

With the problem of slow convergence for traditional Sarsa algorithm

an improved heuristic Sarsa algorithm based on value function transfer was proposed.The algorithm combined traditional Sarsa algorithm and value function transfer method

and the algorithm introduced bisimulation metric and used it to measure the similarity between new tasks and historical tasks in which those two tasks had the same state space and action space and speed up the algorithm convergence.In addition

combined with heuristic exploration method

the algorithm introduced Bayesian inference and used variational inference to measure information gain.Finally

using the obtained information gain to build intrinsic reward function model as exploring factors

to speed up the convergence of the algorithm.Applying the proposed algorithm to the traditional Grid World problem

and compared with the traditional Sarsa algorithm

the Q-Learning algorithm

and the VFT-Sarsa algorithm

the IGP-Sarsa algorithm with better convergence performance

the experiment results show that the proposed algorithm has faster convergence speed and better convergence stability.

关键词

Keywords

references

SUTTON R S , BARTO G A . Reinforcement learning:an introduction [M ] . Cambridge : MIT PressPress , 1998 .

SCHMIDHUBER J , INFORMATIK T T . On learning how to learn learning strategies [R ] . Germany:Technische University , 1995 .

AMMAR H B , EATON E , LUNA J M , et al . Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning [C ] // The 15th International Conference on Artificial Intelligence . 2015 : 3345 - 3351 .

GUPTA A , DEVIN C , LIU Y X , et al . Learning invariant feature spaces to transfer skills with reinforcement learning [C ] // The 5th International Conference on Learning Representations . 2017 : 2147 - 2153 .

LAROCHE R , BARLIER M . Transfer reinforcement learning with shared dynamics [C ] // The 31th International Conference on the Association for the Advance of Artificial Intelligence . 2017 : 2147 - 2153 .

BARRETO A , DABNEY W , MUNOS R , et al . Successor features for transfer in reinforcement learning [C ] // The 32th International Conference on Neural Information Processing Systems . 2017 : 4055 - 4065 .

DEARDEN R , NIR F , STUART R . Bayesian Q-learning [C ] // The 21th International Conference on the Association for the Advance of Artificial Intelligence . 1998 : 761 - 768 .

GUEZ A , SILVER D , DAYAN P . Scalable and efficient Bayes- adaptive reinforcement learning based on Monte-Carlo tree search [J ] . Journal of Artificial Intelligence Research , 2013 , 48 ( 1 ): 841 - 883 .

LITTLE D Y , SOMMER F T . Learning and exploration in action-perception loops [J ] . Frontiers in Neural Circuits , 2013 , 7 ( 7 ): 37 - 56 .

MANSOUR Y , SLIVKINS A , SYRGKANIS V . Bayesian incentive-compatible bandit exploration [C ] // The 16th International Conference on Economics and Computation . 2015 : 565 - 582 .

VIEN N A , LEE S G , CHUNG T C . Bayes-adaptive hierarchical MDPs [J ] . Applied Intelligence , 2016 , 45 ( 1 ): 112 - 126 .

WU B , FENG Y . Monte-Carlo Bayesian reinforcement learning using a compact factored representation [C ] // The 4th International Conference on Information Science and Control Engineering . 2017 : 466 - 469 .

傅启明 , 刘全 , 伏玉琛 , 等 . 一种高斯过程的带参近似策略迭代算法 [J ] . 软件学报 , 2013 , 24 ( 11 ): 2676 - 2687 .

FU Q M , LIU Q , FU Y C , et al . Parametric approximation policy strategy iteration algorithm based on Gaussian process [J ] . Journal of Software , 2013 , 24 ( 11 ): 2676 - 2687 .

GIVAN R , DEAN T , GREIG M . Equivalence notions and model minimization in Markov decision processes [J ] . Artificial Intelligence , 2003 , 147 ( 1 ): 163 - 223 .

FERNS N , PANANGADEN P , PRECUP D . Metrics for finite Markov decision processes [C ] // The 20th International Conference on Uncertainty in Artificial Intelligence . 2004 : 162 - 169 .

BEAL M J . Variational algorithms for approximate Bayesian inference [D ] . London:University of London , 2003 .

傅启明 , 刘全 , 尤树华 , 等 . 一种新的基于值函数迁移的快速Sarsa算法 [J ] . 电子学报 , 2014 , 42 ( 11 ): 2157 - 2161 .

FU Q M , LIU Q , YOU S H , et al . A novel fast sarsa algorithm based on value function transfer [J ] . Acta Electronica Sinica , 2014 , 42 ( 11 ): 2157 - 2161 .

MIERING M , HASSELT H V . The QV family compared to other reinforcement learning algorithms [C ] // The 17th International Conference on Approximate Dynamic Programming and Reinforcement Learning . 2008 : 101 - 108 .

CHUNG J J , LAWRANCE N R J , SUKKARIEH S . Gaussian processes for informative exploration in reinforcement learning [C ] // The 20th International Conference on Robotics and Automation . 2013 : 2633 - 2639 .

Views

1952

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

RL-WGAN based method for 5G network anomalous data generation

Study on Co-EDCA mechanism for multi-AP collaboration in FTTR C-WAN architecture

Design and implementation of an IPv6+ based intelligent computing-network scheduling scheme for Internet of vehicles

Survey of node localization scheme in underwater wireless sensor network

Adaptive defense model for critical assets against unknown network threats

Related Author

Ning Zhaolong

Zou Daoyuan

Zhou Li

Ouyang Ruiqi

Xiong Xuanrui

WU Weimin

ZENG Chen

YU Zhaoyang

Related Institution

College of Electronic Science and Technology, National University of Defense Technology

School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications

School of Electronic Information and Communication, Huazhong University of Science and Technology

Intelligent and Connected Vehicle Research Institute, China Unicom Smart Connection Technology Limited

School of Computer Science, Beijing University of Posts and Telecommunications

AI问答

⁰