scholarly journals Deep reinforcement learning based control for Autonomous Vehicles in CARLA

2022 ◽  
Author(s):  
Óscar Pérez-Gil ◽  
Rafael Barea ◽  
Elena López-Guillén ◽  
Luis M. Bergasa ◽  
Carlos Gómez-Huélamo ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Vehicle Control ◽  
Data Sources ◽  
Simulation Environment ◽  
Urban Simulation ◽  
Policy Gradient ◽  
Almost All ◽  
Control Layer

AbstractNowadays, Artificial Intelligence (AI) is growing by leaps and bounds in almost all fields of technology, and Autonomous Vehicles (AV) research is one more of them. This paper proposes the using of algorithms based on Deep Learning (DL) in the control layer of an autonomous vehicle. More specifically, Deep Reinforcement Learning (DRL) algorithms such as Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) are implemented in order to compare results between them. The aim of this work is to obtain a trained model, applying a DRL algorithm, able of sending control commands to the vehicle to navigate properly and efficiently following a determined route. In addition, for each of the algorithms, several agents are presented as a solution, so that each of these agents uses different data sources to achieve the vehicle control commands. For this purpose, an open-source simulator such as CARLA is used, providing to the system with the ability to perform a multitude of tests without any risk into an hyper-realistic urban simulation environment, something that is unthinkable in the real world. The results obtained show that both DQN and DDPG reach the goal, but DDPG obtains a better performance. DDPG perfoms trajectories very similar to classic controller as LQR. In both cases RMSE is lower than 0.1m following trajectories with a range 180-700m. To conclude, some conclusions and future works are commented.

scholarly journals Autonomous Driving in Roundabout Maneuvers Using Reinforcement Learning with Q-Learning

Electronics ◽  
2019 ◽  
Vol 8 (12) ◽  
pp. 1536 ◽  
Author(s):  
Laura García Cuenca ◽  
Enrique Puertas ◽  
Javier Fernandez Andrés ◽  
Nourdine Aliane
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Learning Algorithm ◽  
Autonomous Vehicle ◽  
Autonomous Driving ◽  
Simulation Environment ◽  
Q Learning ◽  
Driving Scenario

Navigating roundabouts is a complex driving scenario for both manual and autonomous vehicles. This paper proposes an approach based on the use of the Q-learning algorithm to train an autonomous vehicle agent to learn how to appropriately navigate roundabouts. The proposed learning algorithm is implemented using the CARLA simulation environment. Several simulations are performed to train the algorithm in two scenarios: navigating a roundabout with and without surrounding traffic. The results illustrate that the Q-learning-algorithm-based vehicle agent is able to learn smooth and efficient driving to perform maneuvers within roundabouts.

scholarly journals An Efficiency Enhancing Methodology for Multiple Autonomous Vehicles in an Urban Network Adopting Deep Reinforcement Learning

2021 ◽  
Vol 11 (4) ◽  
pp. 1514 ◽  
Author(s):  
Quang-Duy Tran ◽  
Sang-Hoon Bae
Keyword(s):  
Reinforcement Learning ◽  
Traffic Congestion ◽  
Autonomous Vehicles ◽  
Penetration Rate ◽  
Autonomous Vehicle ◽  
Effective Means ◽  
Urban Network ◽  
Learning Agents ◽  
Policy Optimization ◽  
The Impact

To reduce the impact of congestion, it is necessary to improve our overall understanding of the influence of the autonomous vehicle. Recently, deep reinforcement learning has become an effective means of solving complex control tasks. Accordingly, we show an advanced deep reinforcement learning that investigates how the leading autonomous vehicles affect the urban network under a mixed-traffic environment. We also suggest a set of hyperparameters for achieving better performance. Firstly, we feed a set of hyperparameters into our deep reinforcement learning agents. Secondly, we investigate the leading autonomous vehicle experiment in the urban network with different autonomous vehicle penetration rates. Thirdly, the advantage of leading autonomous vehicles is evaluated using entire manual vehicle and leading manual vehicle experiments. Finally, the proximal policy optimization with a clipped objective is compared to the proximal policy optimization with an adaptive Kullback–Leibler penalty to verify the superiority of the proposed hyperparameter. We demonstrate that full automation traffic increased the average speed 1.27 times greater compared with the entire manual vehicle experiment. Our proposed method becomes significantly more effective at a higher autonomous vehicle penetration rate. Furthermore, the leading autonomous vehicles could help to mitigate traffic congestion.

A novel path tracking approach considering safety of the intended functionality for autonomous vehicles

2021 ◽  
pp. 095440702110205
Author(s):  
Huiran Wang ◽  
Qidong Wang ◽  
Wuwei Chen ◽  
Linfeng Zhao ◽  
Dongkui Tan
Keyword(s):  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Coordinated Control ◽  
Steering System ◽  
Front Wheel ◽  
Path Tracking ◽  
Hierarchical Architecture ◽  
Automatic Steering ◽  
The Stability ◽  
Control Layer

To reduce the adverse effect of the functional insufficiency of the steering system on the accuracy of path tracking, a path tracking approach considering safety of the intended functionality is proposed by coordinating automatic steering and differential braking in this paper. The proposed method adopts a hierarchical architecture consisting of a coordinated control layer and an execution control layer. In coordinated control layer, an extension controller considering functional insufficiency of the steering system, tire force characteristics and vehicle driving stability is proposed to determine the weight coefficients of automatic steering and the differential braking, and a model predictive controller is designed to calculate the desired front wheel angle and additional yaw moment. In execution control layer, a H∞ steering angle controller considering external disturbances and parameter uncertainty is designed to track desired front wheel angle, and a braking force distribution module is used to determine the wheel cylinder pressure of the controlled wheels. Both simulation and experiment results show that the proposed method can overcome the functional insufficiency of the steering system and improve the accuracy of path tracking while maintaining the stability of the autonomous vehicle.

scholarly journals Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV’s Autonomous Motion Planning in Complex Unknown Environments

Sensors ◽  
2020 ◽  
Vol 20 (7) ◽  
pp. 1890 ◽  
Author(s):  
Zijian Hu ◽  
Kaifang Wan ◽  
Xiaoguang Gao ◽  
Yiwei Zhai ◽  
Qianglong Wang
Keyword(s):  
Reinforcement Learning ◽  
Motion Planning ◽  
Predictive Control ◽  
Experimental Testing ◽  
The Novel ◽  
Simulation Environment ◽  
Unknown Environments ◽  
Autonomous Motion ◽  
Policy Gradient ◽  
Speed Up

Autonomous motion planning (AMP) of unmanned aerial vehicles (UAVs) is aimed at enabling a UAV to safely fly to the target without human intervention. Recently, several emerging deep reinforcement learning (DRL) methods have been employed to address the AMP problem in some simplified environments, and these methods have yielded good results. This paper proposes a multiple experience pools (MEPs) framework leveraging human expert experiences for DRL to speed up the learning process. Based on the deep deterministic policy gradient (DDPG) algorithm, a MEP–DDPG algorithm was designed using model predictive control and simulated annealing to generate expert experiences. On applying this algorithm to a complex unknown simulation environment constructed based on the parameters of the real UAV, the training experiment results showed that the novel DRL algorithm resulted in a performance improvement exceeding 20% as compared with the state-of-the-art DDPG. The results of the experimental testing indicate that UAVs trained using MEP–DDPG can stably complete a variety of tasks in complex, unknown environments.

scholarly journals End-to-End Automated Lane-Change Maneuvering Considering Driving Style Using a Deep Deterministic Policy Gradient Algorithm

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5443
Author(s):  
Hongyu Hu ◽  
Ziyang Lu ◽  
Qi Wang ◽  
Chengyuan Zheng
Keyword(s):  
Autonomous Vehicles ◽  
The State ◽  
Steering Wheel ◽  
Lane Change ◽  
Simulation Environment ◽  
Longitudinal Control ◽  
Driving Style ◽  
Lane Changing ◽  
Policy Gradient ◽  
End To End

Changing lanes while driving requires coordinating the lateral and longitudinal controls of a vehicle, considering its running state and the surrounding environment. Although the existing rule-based automated lane-changing method is simple, it is unsuitable for unpredictable scenarios encountered in practice. Therefore, using a deep deterministic policy gradient (DDPG) algorithm, we propose an end-to-end method for automated lane changing based on lidar data. The distance state information of the lane boundary and the surrounding vehicles obtained by the agent in a simulation environment is denoted as the state space for an automated lane-change problem based on reinforcement learning. The steering wheel angle and longitudinal acceleration are used as the action space, and both the state and action spaces are continuous. In terms of the reward function, avoiding collision and setting different expected lane-changing distances that represent different driving styles are considered for security, and the angular velocity of the steering wheel and jerk are considered for comfort. The minimum speed limit for lane changing and the control of the agent for a quick lane change are considered for efficiency. For a one-way two-lane road, a visual simulation environment scene is constructed using Pyglet. By comparing the lane-changing process tracks of two driving styles in a simplified traffic flow scene, we study the influence of driving style on the lane-changing process and lane-changing time. Through the training and adjustment of the combined lateral and longitudinal control of autonomous vehicles with different driving styles in complex traffic scenes, the vehicles could complete a series of driving tasks while considering driving-style differences. The experimental results show that autonomous vehicles can reflect the differences in the driving styles at the time of lane change at the same speed. Under the combined lateral and longitudinal control, the autonomous vehicles exhibit good robustness to different speeds and traffic density in different road sections. Thus, autonomous vehicles trained using the proposed method can learn an automated lane-changing policy while considering safety, comfort, and efficiency.

scholarly journals Proximal Policy Optimization Through a Deep Reinforcement Learning Framework for Multiple Autonomous Vehicles at a Non-Signalized Intersection

2020 ◽  
Vol 10 (16) ◽  
pp. 5722 ◽  
Author(s):  
Duy Quang Tran ◽  
Sang-Hoon Bae
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Signalized Intersection ◽  
Continuous Control ◽  
Simulation Performance ◽  
Perceptron Algorithm ◽  
Learning Framework ◽  
Positive Effects ◽  
Policy Optimization

Advanced deep reinforcement learning shows promise as an approach to addressing continuous control tasks, especially in mixed-autonomy traffic. In this study, we present a deep reinforcement-learning-based model that considers the effectiveness of leading autonomous vehicles in mixed-autonomy traffic at a non-signalized intersection. This model integrates the Flow framework, the simulation of urban mobility simulator, and a reinforcement learning library. We also propose a set of proximal policy optimization hyperparameters to obtain reliable simulation performance. First, the leading autonomous vehicles at the non-signalized intersection are considered with varying autonomous vehicle penetration rates that range from 10% to 100% in 10% increments. Second, the proximal policy optimization hyperparameters are input into the multiple perceptron algorithm for the leading autonomous vehicle experiment. Finally, the superiority of the proposed model is evaluated using all human-driven vehicle and leading human-driven vehicle experiments. We demonstrate that full-autonomy traffic can improve the average speed and delay time by 1.38 times and 2.55 times, respectively, compared with all human-driven vehicle experiments. Our proposed method generates more positive effects when the autonomous vehicle penetration rate increases. Additionally, the leading autonomous vehicle experiment can be used to dissipate the stop-and-go waves at a non-signalized intersection.

Virtual Simulations Environment Development for Autonomous Vehicles Interaction

2020 ◽  
Author(s):  
Mohsen Malayjerdi ◽  
Vladimir Kuts ◽  
Raivo Sell ◽  
Tauno Otto ◽  
Barış Cem Baykara
Keyword(s):  
Virtual Environment ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Mapping Method ◽  
Human Robot Interaction ◽  
Industrial Robots ◽  
Simulation Environment ◽  
Road Users ◽  
Virtual Simulations ◽  
Road Testing

Abstract One of the primary verification criteria of the autonomous vehicle is safe interaction with other road users. Based on studies, real-road testing is not practical for safety validation due to its time and cost consuming. Therefore, simulating miles driven is the only feasible way to overcome this limitation. The primary goal of the related research project is to develop advanced techniques in the human-robot interaction (HRI) safety validation area by usage of immersive simulation technologies. Developing methods for the creation of the simulation environment will enable us to do experiments in a digital environment rather than real. The main aim of the paper is to develop an effective method of creating a virtual environment for performing simulations on industrial robots, mobile robots, and autonomous vehicles (AGV-s) from the safety perspective for humans. A mid-size drone was used for aerial imagery as the first step in creating a virtual environment. Then all the photos were processed in several steps to build the final 3D map. Next, this mapping method was used to create a high detail simulation environment for testing an autonomous shuttle. Developing efficient methods for mapping real environments and simulating their variables is crucial for the testing and development of control algorithms of autonomous vehicles.

scholarly journals Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2032
Author(s):  
Sampo Kuutti ◽  
Richard Bowden ◽  
Saber Fallah
Keyword(s):  
Reinforcement Learning ◽  
Autonomous Vehicles ◽  
Autonomous Vehicle ◽  
Model Performance ◽  
Model Parameters ◽  
Safe Driving ◽  
Weak Supervision ◽  
Rule Based ◽  
Learning Agent ◽  
Weakly Supervised

The use of neural networks and reinforcement learning has become increasingly popular in autonomous vehicle control. However, the opaqueness of the resulting control policies presents a significant barrier to deploying neural network-based control in autonomous vehicles. In this paper, we present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. By guiding the agent to meaningful states and actions, this weak supervision improves the convergence during training and enhances the safety of the final trained policy. This rule-based supervisory controller has the further advantage of being fully interpretable, thereby enabling traditional validation and verification approaches to ensure the safety of the vehicle. We compare models with and without safety cages, as well as models with optimal and constrained model parameters, and show that the weak supervision consistently improves the safety of exploration, speed of convergence, and model performance. Additionally, we show that when the model parameters are constrained or sub-optimal, the safety cages can enable a model to learn a safe driving policy even when the model could not be trained to drive through reinforcement learning alone.

Sign in / Sign up

Export Citation Format

Share Document