Reinforcement Learning in Healthcare: A Survey

As a subfield of machine learning, reinforcement learning (RL) aims at optimizing decision making by using interaction samples of an agent with its environment and the potentially delayed feedbacks. In contrast to traditional supervised learning that typically relies on one-shot, exhaustive, and supervised reward signals, RL tackles sequential decision-making problems with sampled, evaluative, and delayed feedbacks simultaneously. Such a distinctive feature makes RL techniques a suitable candidate for developing powerful solutions in various healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged period with delayed feedbacks. By first briefly examining theoretical foundations and key methods in RL research, this survey provides an extensive overview of RL applications in a variety of healthcare domains, ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis, and many other control or scheduling problems that have infiltrated every aspect of the healthcare system. In addition, we discuss the challenges and open issues in the current research and highlight some potential solutions and directions for future research.

Integrated Parametric Graph Closure and Branch-and-Cut Algorithm for Open Pit Mine Scheduling under Uncertainty

Open pit mine production scheduling is a computationally expensive large-scale mixed-integer linear programming problem. This research develops a computationally efficient algorithm to solve open pit production scheduling problems under uncertain geological parameters. The proposed solution approach for production scheduling is a two-stage process. The stochastic production scheduling problem is iteratively solved in the first stage after relaxing resource constraints using a parametric graph closure algorithm. Finally, the branch-and-cut algorithm is applied to respect the resource constraints, which might be violated during the first stage of the algorithm. Six small-scale production scheduling problems from iron and copper mines were used to validate the proposed stochastic production scheduling model. The results demonstrated that the proposed method could significantly improve the computational time with a reasonable optimality gap (the maximum gap is 4%). In addition, the proposed stochastic method is tested using industrial-scale copper data and compared with its deterministic model. The results show that the net present value for the stochastic model improved by 6% compared to the deterministic model.

Rescheduling of Distributed Manufacturing System with Machine Breakdowns

This study attempts to explore the dynamic scheduling problem from the perspective of operational research optimization. The goal is to propose a rescheduling framework for solving distributed manufacturing systems that consider random machine breakdowns as the production disruption. We establish a mathematical model that can better describe the scheduling of the distributed blocking flowshop. To realize the dynamic scheduling, we adopt an “event-driven” policy and propose a two-stage “predictive-reactive” method consisting of two steps: initial solution pre-generation and rescheduling. In the first stage, a global initial schedule is generated and considers only the deterministic problem, i.e., optimizing the maximum completion time of static distributed blocking flowshop scheduling problems. In the second stage, that is, after the breakdown occurs, the rescheduling mechanism is triggered to seek a new schedule so that both maximum completion time and the stability measure of the system can be optimized. At the breakdown node, the operations of each job are classified and a hybrid rescheduling strategy consisting of “right-shift repair + local reorder” is performed. For local reorder, we designed a discrete memetic algorithm, which embeds the differential evolution concept in its search framework. To test the effectiveness of DMA, comparisons with mainstream algorithms are conducted on instances with different scales. The statistical results show that the ARPDs obtained from DMA are improved by 88%.

An Analysis of Research Trends in the Sustainability of Production Palnning

Sustainability has become of great interest in many fields, especially in production systems due to the continual increase in the scarcity of raw materials and environmental awareness. Recent literature has given significant attention to considering the three sustainability pillars (i.e., environmental, economic, and social sustainability) in solving production planning problems. Therefore, the present study conducts a review of the literature on sustainable production planning to analyze the relationships among different production planning problems (e.g., scheduling, lot sizing, aggregate planning, etc.) and the three sustainability pillars. In addition, we analyze the identified studies based on the indicators that define each pillar. The results show that the literature most frequently addresses production scheduling problems while it lacks studies on aggregate production planning problems that consider the sustainability pillars. In addition, there is a growing trend towards obtaining integrated solutions of different planning problems, e.g., combining production planning problems with maintenance planning or energy planning. Additionally, around 45% of the identified studies considered the integration of the economic and the environmental pillars in different production planning problems. In addition, energy consumption and greenhouse gas emissions are the most frequent sustainability indicators considered in the literature, while less attention has been given to social indicators. Another issue is the low number of studies that have considered all three sustainability pillars simultaneously. The finidings highlight the need for more future research towards holistic sustainable production planning approaches.

Scheduling by NSGA-II: Review and Bibliometric Analysis

NSGA-II is an evolutionary multi-objective optimization algorithm that has been applied to a wide variety of search and optimization problems since its publication in 2000. This study presents a review and bibliometric analysis of numerous NSGA-II adaptations in addressing scheduling problems. This paper is divided into two parts. The first part discusses the main ideas of scheduling and different evolutionary computation methods for scheduling and provides a review of different scheduling problems, such as production and personnel scheduling. Moreover, a brief comparison of different evolutionary multi-objective optimization algorithms is provided, followed by a summary of state-of-the-art works on the application of NSGA-II in scheduling. The next part presents a detailed bibliometric analysis focusing on NSGA-II for scheduling applications obtained from the Scopus and Web of Science (WoS) databases based on keyword and network analyses that were conducted to identify the most interesting subject fields. Additionally, several criteria are recognized which may advise scholars to find key gaps in the field and develop new approaches in future works. The final sections present a summary and aims for future studies, along with conclusions and a discussion.

Storage assignment optimization for fishbone robotic mobile fulfillment systems

Robotic Mobile Fulfillment System (RMFS) affects the traditional scheduling problems heavily while operating a warehouse. This paper focuses on storage assignment optimization for Fishbone Robotic Mobile Fulfilment Systems (FRMFS). Based on analyzing operation characteristics of FRMFS, a storage assignment optimization model is proposed with the objectives of maximizing operation efficiency and balancing aisle workload. Adaptive Genetic Algorithm (AGA) is designed to solve the proposed model. To validate the effectiveness of AGA in terms of iteration and optimization rate, this paper designs a variety of scenarios with different task sizes and storage cells. AGA outperforms other four algorithm in terms of fitness value and convergence and has better convergence rate and stability. The experimental results also show the advancement of AGA in large size FRMFS. In conclusion, this paper proposes a storage assignment model for FRMFS to reduce goods movement and travel distance and improve the order picking efficiency.

Solution of Multi-Crew Depots Railway Crew Scheduling Problems: The Chinese High-Speed Railway Case

This paper presents a novel mathematical formulation in crew scheduling, considering real challenges most railway companies face such as roundtrip policy for crew members joining from different crew depots and stricter working time standards under a sustainable development strategy. In China, the crew scheduling is manually compiled by railway companies respectively, and the plan quality varies from person to person. An improved genetic algorithm is proposed to solve this large-scale combinatorial optimization problem. It repairs the infeasible gene fragments to optimize the search scope of the solution space and enhance the efficiency of GA. To investigate the algorithm’s efficiency, a real case study was employed. Results show that the proposed model and algorithm lead to considerable improvement compared to the original planning: (i) Compared with the classical metaheuristic algorithms (GA, PSO, TS), the improved genetic algorithm can reduce the objective value by 4.47%; and (ii) the optimized crew scheduling plan reduces three crew units and increases the average utilization of crew unit working time by 6.20% compared with the original plan.