Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

<div>Recently, barrier function-based safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a model-based safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier-based control policy structure that can guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic learning algorithm is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify the offline deployment performance and the online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.</div>

Long-Haul Vehicle Routing and Scheduling with Biomathematical Fatigue Constraints

Hours of service (HOS) regulations are among the conventional safety constraints that are compiled by long-haul truck drivers. These regulations have been considered in models and algorithms of vehicle routing problems to assign safe schedules to drivers. However, the HOS regulations neglect a few crucial fatigue risk factors and, at times, fail to generate fatigue-reducing schedules. In this study, a set of biomathematical fatigue constraints (BFCs) derived from biomathematical models are considered for a long-haul vehicle routing and scheduling problem. A BFC scheduling algorithm and a BFC-HOS scheduling algorithm have been developed and then embedded within a tabu search heuristic to solve the combined vehicle routing and scheduling problem. All the solution methods have been tested on modified Solomon instances and a real-life instance, and the computational results confirm the advantages of employing a sophisticated and fatigue-reducing scheduling procedure when planning long-haul transportation.

Mode Switching Control Using Lane Keeping Assist and Waypoints Tracking for Autonomous Driving in a City Environment

This paper proposes a mode switching supervisory controller for autonomous vehicles. The supervisory controller selects the most appropriate controller based on safety constraints and on the vehicle location with respect to junctions. Autonomous steering, throttle and deceleration control inputs are used to perform variable speed lane keeping assist, standard or emergency braking and to manage junctions, including roundabouts. Adaptive model predictive control with lane keeping assist is performed on the main roads and a linear pure pursuit inspired controller is applied using waypoints at road junctions where lane keeping assist sensors present a safety risk. A multi-stage rule based autonomous braking algorithm performs stop, restart and emergency braking maneuvers. The controllers are implemented in MATLAB® and Simulink™ and are demonstrated using the Automatic Driving Toolbox™ environment. Numerical simulations of autonomous driving scenarios demonstrate the efficiency of the lane keeping assist mode on roads with curvature and the ability to accurately track waypoints at cross intersections and roundabouts using a simpler pure pursuit inspired mode. The ego vehicle also autonomously stops in time at signaled intersections or to avoid collision with other road users.

An Intelligent Scheduling Access Privacy Protection Model of Electric Vehicle Based on 5G-V2X

It is more and more important to optimize electric power system scheduling in the development of the Internet of Vehicles. How to improve the applicability and scientific nature of electric vehicle charging is an urgent problem to be solved. This paper proposes an intelligent scheduling access model for electric vehicles based on blockchain. Firstly, the layout simplification calculation is carried out for the layout of the traditional distributed power grid. Then, a data storage and consensus system is built using blockchain smart contracts to ensure that all historical data are not tampered with and are traceable. Finally, the model forms an electricity price guidance model in the intelligent scheduling and access of electric vehicles, optimizes the multivehicle line congestion in operation, and can dynamically schedule and correct the model. In terms of the experiment, 13 test electric vehicles were dispatched based on 12 real power station nodes and 36 test nodes of Yunnan Power Grid Co. Information Center for verification. The result analysis shows that the model can effectively and quickly solve the blocking and unblocking of the Internet of Vehicles and can develop a scheduling scheme conforming to the safety constraints of electric vehicles in a relatively short time.