scholarly journals An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

2021 ◽  
Vol 11 (21) ◽  
pp. 10337
Author(s):  
Junkai Ren ◽  
Yujun Zeng ◽  
Sihang Zhou ◽  
Yichuan Zhang
Keyword(s):  
Experimental Study ◽  
Reinforcement Learning ◽  
Network Architecture ◽  
Representation Learning ◽  
Evaluation Metrics ◽  
High Dimensional ◽  
Regularization Methods ◽  
Challenging Problem ◽  
State Representation ◽  
Sample Quality

Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

scholarly journals Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 4924-4931
Author(s):  
Yingru Liu ◽  
Xuewen Yang ◽  
Dongliang Xie ◽  
Xin Wang ◽  
Li Shen ◽  
...  
Keyword(s):  
Network Architecture ◽  
Regularization Methods ◽  
Main Principle ◽  
Challenging Problem ◽  
Activation Functions ◽  
Task Learning ◽  
Optimal Network ◽  
Multiple Tasks ◽  
Improved Performance ◽  
Deep Learning Model

Multi-task learning (MTL) is a common paradigm that seeks to improve the generalization performance of task learning by training related tasks simultaneously. However, it is still a challenging problem to search the flexible and accurate architecture that can be shared among multiple tasks. In this paper, we propose a novel deep learning model called Task Adaptive Activation Network (TAAN) that can automatically learn the optimal network architecture for MTL. The main principle of TAAN is to derive flexible activation functions for different tasks from the data with other parameters of the network fully shared. We further propose two functional regularization methods that improve the MTL performance of TAAN. The improved performance of both TAAN and the regularization methods is demonstrated by comprehensive experiments.

State Representation Learning with Adjacent State Consistency Loss for Deep Reinforcement Learning

2021 ◽  
pp. 1-1
Author(s):  
Tianyu Zhao ◽  
Jian Zhao ◽  
Wengang Zhou ◽  
Yun Zhou ◽  
Houqiang Li
Keyword(s):  
Reinforcement Learning ◽  
Representation Learning ◽  
State Representation

scholarly journals Grounding Language for Transfer in Deep Reinforcement Learning

2018 ◽  
Vol 63 ◽  
pp. 849-874 ◽  
Author(s):  
Karthik Narasimhan ◽  
Regina Barzilay ◽  
Tommi Jaakkola
Keyword(s):  
Reinforcement Learning ◽  
Policy Transfer ◽  
Policy Learning ◽  
Autonomous Agent ◽  
Prior Work ◽  
Challenging Problem ◽  
Wide Spread ◽  
State Representation ◽  
Model Free ◽  
Effective Policy

In this paper, we explore the utilization of natural language to drive transfer for reinforcement learning (RL). Despite the wide-spread application of deep RL techniques, learning generalized policy representations that work across domains remains a challenging problem. We demonstrate that textual descriptions of environments provide a compact intermediate channel to facilitate effective policy transfer. Specifically, by learning to ground the meaning of text to the dynamics of the environment such as transitions and rewards, an autonomous agent can effectively bootstrap policy learning on a new domain given its description. We employ a model-based RL approach consisting of a differentiable planning module, a model-free component and a factorized state representation to effectively use entity descriptions. Our model outperforms prior work on both transfer and multi-task scenarios in a variety of different environments. For instance, we achieve up to 14% and 11.5% absolute improvement over previously existing models in terms of average and initial rewards, respectively.

scholarly journals Integrating State Representation Learning Into Deep Reinforcement Learning

2018 ◽  
Vol 3 (3) ◽  
pp. 1394-1401 ◽  
Author(s):  
Tim de Bruin ◽  
Jens Kober ◽  
Karl Tuyls ◽  
Robert Babuska
Keyword(s):  
Reinforcement Learning ◽  
Representation Learning ◽  
State Representation

scholarly journals A State Representation for Reinforcement Learning and Decision-Making in the Orbitofrontal Cortex

2017 ◽  
Author(s):  
Nicolas W. Schuck ◽  
Robert Wilson ◽  
Yael Niv
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Orbitofrontal Cortex ◽  
High Dimensional ◽  
State Representation ◽  
Functional Studies ◽  
Related Information ◽  
Novel Theory ◽  
Partially Observable ◽  
Support Decision Making

AbstractDespite decades of research, the exact ways in which the orbitofrontal cortex (OFC) influences cognitive function have remained mysterious. Anatomically, the OFC is characterized by remarkably broad connectivity to sensory, limbic and subcortical areas, and functional studies have implicated the OFC in a plethora of functions ranging from facial processing to value-guided choice. Notwithstanding such diversity of findings, much research suggests that one important function of the OFC is to support decision making and reinforcement learning. Here, we describe a novel theory that posits that OFC’s specific role in decision-making is to provide an up-to-date representation of task-related information, called a state representation. This representation reflects a mapping between distinct task states and sensory as well as unobservable information. We summarize evidence supporting the existence of such state representations in rodent and human OFC and argue that forming these state representations provides a crucial scaffold that allows animals to efficiently perform decision making and reinforcement learning in high-dimensional and partially observable environments. Finally, we argue that our theory offers an integrating framework for linking the diversity of functions ascribed to OFC and is in line with its wide ranging connectivity.

scholarly journals Computation offloading through mobile vehicles in IoT-edge-cloud network

2020 ◽  
Vol 2020 (1) ◽  
Author(s):  
Jun Long ◽  
Yueyi Luo ◽  
Xiaoyu Zhu ◽  
Entao Luo ◽  
Mingfeng Huang
Keyword(s):  
Reinforcement Learning ◽  
Energy Consumption ◽  
Low Cost ◽  
Computation Offloading ◽  
Mobile Edge Computing ◽  
Challenging Problem ◽  
Learning Technique ◽  
Cloud Network ◽  
The City ◽  
Task Offloading

AbstractWith the developing of Internet of Things (IoT) and mobile edge computing (MEC), more and more sensing devices are widely deployed in the smart city. These sensing devices generate various kinds of tasks, which need to be sent to cloud to process. Usually, the sensing devices do not equip with wireless modules, because it is neither economical nor energy saving. Thus, it is a challenging problem to find a way to offload tasks for sensing devices. However, many vehicles are moving around the city, which can communicate with sensing devices in an effective and low-cost way. In this paper, we propose a computation offloading scheme through mobile vehicles in IoT-edge-cloud network. The sensing devices generate tasks and transmit the tasks to vehicles, then the vehicles decide to compute the tasks in the local vehicle, MEC server or cloud center. The computation offloading decision is made based on the utility function of the energy consumption and transmission delay, and the deep reinforcement learning technique is adopted to make decisions. Our proposed method can make full use of the existing infrastructures to implement the task offloading of sensing devices, the experimental results show that our proposed solution can achieve the maximum reward and decrease delay.

Sign in / Sign up

Export Citation Format

Share Document