Linear Programming and Zero-Sum Two-Person Undiscounted Semi-Markov Games

2015 ◽  
Vol 32 (06) ◽  
pp. 1550043 ◽  
Author(s):  
Prasenjit Mondal
Keyword(s):  
Linear Programming ◽  
Transition Probabilities ◽  
Optimal Solution ◽  
Programming Algorithm ◽  
Stationary Strategy ◽  
Markov Games ◽  
Markov Decision ◽  
Transition Times ◽  
Optimal Stationary Strategies ◽  
Zero Sum

In this paper, zero-sum two-person finite undiscounted (limiting average) semi-Markov games (SMGs) are considered. We prove that the solutions of the game when both players are restricted to semi-Markov strategies are solutions for the original game. In addition, we show that if one player fixes a stationary strategy, then the other player can restrict himself in solving an undiscounted semi-Markov decision process associated with that stationary strategy. The undiscounted SMGs are also studied when the transition probabilities and the transition times are controlled by a fixed player in all states. If such games are unichain, we prove that the value and optimal stationary strategies of the players can be obtained from an optimal solution of a linear programming algorithm. We propose a realistic and generalized traveling inspection model that suitably fits into the class of one player control undiscounted unichain semi-Markov games.

Ordered Field Property for Semi-Markov Games when One Player Controls Transition Probabilities and Transition Times

2015 ◽  
Vol 17 (02) ◽  
pp. 1540022 ◽  
Author(s):  
Prasenjit Mondal ◽  
Sagnik Sinha
Keyword(s):  
Linear Programming ◽  
Transition Probabilities ◽  
Markov Games ◽  
Markov Strategy ◽  
Ordered Field ◽  
Player Game ◽  
Ordered Field Property ◽  
Transition Times ◽  
Zero Sum ◽  
Field Property

Two-person finite semi-Markov games (SMGs) are studied when the transition probabilities and the transition times are controlled by one player at all states. For the discounted games in this class, we prove that the ordered field property holds and there exist optimal/Nash equilibrium stationary strategies for the players. We illustrate that the zero-sum SMGs where only transition probabilities are controlled by one player, do not necessarily satisfy the ordered field property. An algorithm along with a numerical example for the discounted one player control zero-sum SMGs is given via linear programming. For the undiscounted version of such games, we exhibit with an example that if the game ceases to be unichain, an optimal stationary or Markov strategy need not exist, (though in this example of a one-player game we exhibit a semi-stationary optimal strategy/policy). Lastly, we prove that if such games are unichain, then they possess the ordered field property for the undiscounted case as well.

Optimal stationary strategies in leavable Markov decision processes

1990 ◽  
Vol 27 (01) ◽  
pp. 134-145
Author(s):  
Matthias Fassbender
Keyword(s):  
Transition Probabilities ◽  
Practical Applications ◽  
Stationary Strategies ◽  
Countable State ◽  
Markov Decision ◽  
Bias Optimality ◽  
The Mean ◽  
Optimal Stationary Strategies ◽  
The Cost ◽  
Reward Criterion

This paper establishes the existence of an optimal stationary strategy in a leavable Markov decision process with countable state space and undiscounted total reward criterion. Besides assumptions of boundedness and continuity, an assumption is imposed on the model which demands the continuity of the mean recurrence times on a subset of the stationary strategies, the so-called ‘good strategies'. For practical applications it is important that this assumption is implied by an assumption about the cost structure and the transition probabilities. In the last part we point out that our results in general cannot be deduced from related works on bias-optimality by Dekker and Hordijk, Wijngaard or Mann.

Optimal stationary strategies in leavable Markov decision processes

1990 ◽  
Vol 27 (1) ◽  
pp. 134-145
Author(s):  
Matthias Fassbender
Keyword(s):  
Transition Probabilities ◽  
Practical Applications ◽  
Stationary Strategies ◽  
Countable State ◽  
Markov Decision ◽  
Bias Optimality ◽  
The Mean ◽  
Optimal Stationary Strategies ◽  
The Cost ◽  
Reward Criterion

This paper establishes the existence of an optimal stationary strategy in a leavable Markov decision process with countable state space and undiscounted total reward criterion.Besides assumptions of boundedness and continuity, an assumption is imposed on the model which demands the continuity of the mean recurrence times on a subset of the stationary strategies, the so-called ‘good strategies'. For practical applications it is important that this assumption is implied by an assumption about the cost structure and the transition probabilities. In the last part we point out that our results in general cannot be deduced from related works on bias-optimality by Dekker and Hordijk, Wijngaard or Mann.

On zero-sum two-person undiscounted semi-Markov games with a multichain structure

2017 ◽  
Vol 49 (3) ◽  
pp. 826-849 ◽  
Author(s):  
Prasenjit Mondal
Keyword(s):  
Stochastic Game ◽  
Optimal Strategies ◽  
Markov Games ◽  
Stationary Strategies ◽  
Current State ◽  
Strategy Evaluation ◽  
Optimality Equations ◽  
Optimal Stationary Strategies ◽  
Zero Sum ◽  
Absorbing States

Abstract Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games (SMGs) are considered with a general multichain structure. We derive the strategy evaluation equations for stationary strategies of the players. A relation between the payoff in the multichain SMG and that in the associated stochastic game (SG) obtained by a data-transformation is established. We prove that the multichain optimality equations (OEs) for an SMG have a solution if and only if the associated SG has optimal stationary strategies. Though the solution of the OEs may not be optimal for an SMG, we establish the significance of studying the OEs for a multichain SMG. We provide a nice example of SMGs in which one player has no optimal strategy in the stationary class but has an optimal semistationary strategy (that depends only on the initial and current state of the game). For an SMG with absorbing states, we prove that solutions in the game where all players are restricted to semistationary strategies are solutions for the unrestricted game. Finally, we prove the existence of stationary optimal strategies for unichain SMGs and conclude that the unichain condition is equivalent to require that the game satisfies some recurrence/ergodicity/weakly communicating conditions.

A Policy Improvement Algorithm for Solving a Mixture Class of Perfect Information and AR-AT Semi-Markov Games

2020 ◽  
Vol 22 (02) ◽  
pp. 2040008
Author(s):  
P. Mondal ◽  
S. K. Neogy ◽  
A. Gupta ◽  
D. Ghorui
Keyword(s):  
Perfect Information ◽  
Policy Improvement ◽  
Markov Games ◽  
Markov Game ◽  
Improvement Algorithm ◽  
Finite State ◽  
Markov Decision ◽  
Mixture Class ◽  
Zero Sum ◽  
Action Spaces

Zero-sum two-person discounted semi-Markov games with finite state and action spaces are studied where a collection of states having Perfect Information (PI) property is mixed with another collection of states having Additive Reward–Additive Transition and Action Independent Transition Time (AR-AT-AITT) property. For such a PI/AR-AT-AITT mixture class of games, we prove the existence of an optimal pure stationary strategy for each player. We develop a policy improvement algorithm for solving discounted semi-Markov decision processes (one player version of semi-Markov games) and using it we obtain a policy-improvement type algorithm for computing an optimal strategy pair of a PI/AR-AT-AITT mixture semi-Markov game. Finally, we extend our results when the states having PI property are replaced by a subclass of Switching Control (SC) states.

MAINTENANCE OPTIMIZATION OF EQUIPMENT BY LINEAR PROGRAMMING

2005 ◽  
Vol 20 (1) ◽  
pp. 183-193 ◽  
Author(s):  
Archana Jayakumar ◽  
Sohrab Asgarpoor
Keyword(s):  
Linear Programming ◽  
Preventive Maintenance ◽  
Mathematical Optimization ◽  
Optimal Solution ◽  
Cost Effective ◽  
Maintenance Policy ◽  
Markov Decision ◽  
Random Failure ◽  
Optimal Maintenance ◽  
Optimal Maintenance Policy

Optimal levels of preventive maintenance performed on any system ensures cost-effective and reliable operation of the system. In this paper a component with deterioration and random failure is modeled using Markov processes while incorporating the concept of minor and major preventive maintenance. The optimal mean times to preventive maintenance (both minor and major) of the component is determined by maximizing its availability with respect to mean time to preventive maintenance. Mathematical optimization programs Maple 7 and Lingo 7 are used to find the optimal solution, which is illustrated using a numerical example. Further, an optimal maintenance policy is obtained using Markov Decision Processes (MDPs). Linear Programming (LP) is utilized to implement the MDP problem.

Zero-sum two-person semi-Markov games

1992 ◽  
Vol 29 (01) ◽  
pp. 56-72 ◽  
Author(s):  
Arbind K. Lal ◽  
Sagnik Sinha
Keyword(s):  
Stochastic Games ◽  
Optimal Strategies ◽  
Existence Of A Solution ◽  
Average Case ◽  
Optimality Equation ◽  
Markov Games ◽  
Special Cases ◽  
Ergodic Condition ◽  
Markov Decision ◽  
Zero Sum

Semi-Markov games are investigated under discounted and limiting average payoff criteria. The issue of the existence of the value and a pair of stationary optimal strategies are settled; the optimality equation is studied and under a natural ergodic condition the existence of a solution to the optimality equation is proved for the limiting average case. Semi-Markov games provide useful flexibility in constructing recursive game models. All the work on Markov/semi-Markov decision processes and Markov (stochastic) games can be viewed as special cases of the developments in this paper.

Sign in / Sign up

Export Citation Format

Share Document