Markov chains with binomial time change

The effect of a binomial time change on a given Markov chain is studied. Results are obtained for the hitting times, potential operator, and transience and recurrence properties of the time-changed chain. The limiting behavior is considered as the binomial parameter approaches 0 and the time variable approaches∞.

Download Full-text

On identifiability and order of continuous-time aggregated Markov chains, Markov-modulated Poisson processes, and phase-type distributions

Journal of Applied Probability ◽

10.2307/3215346 ◽

1996 ◽

Vol 33 (3) ◽

pp. 640-653 ◽

Cited By ~ 24

Author(s):

Tobias Rydén

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Continuous Time ◽

Poisson Processes ◽

Hitting Times ◽

Phase Type ◽

Phase Type Distributions ◽

Minimum Number ◽

Markov Modulated ◽

Two Parameters

An aggregated Markov chain is a Markov chain for which some states cannot be distinguished from each other by the observer. In this paper we consider the identifiability problem for such processes in continuous time, i.e. the problem of determining whether two parameters induce identical laws for the observable process or not. We also study the order of a continuous-time aggregated Markov chain, which is the minimum number of states needed to represent it. In particular, we give a lower bound on the order. As a by-product, we obtain results of this kind also for Markov-modulated Poisson processes, i.e. doubly stochastic Poisson processes whose intensities are directed by continuous-time Markov chains, and phase-type distributions, which are hitting times in finite-state Markov chains.

Download Full-text

Markov Chain–Based Stochastic Strategies for Robotic Surveillance

Annual Review of Control Robotics and Autonomous Systems ◽

10.1146/annurev-control-071520-120123 ◽

2020 ◽

Vol 4 (1) ◽

Author(s):

Xiaoming Duan ◽

Francesco Bullo

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Mobile Robots ◽

Performance Metrics ◽

Optimization Problems ◽

Autonomous Systems ◽

Hitting Times ◽

Annual Review ◽

Publication Date ◽

Motion Models

This article surveys recent advancements in strategy designs for persistent robotic surveillance tasks, with a focus on stochastic approaches. The problem describes how mobile robots stochastically patrol a graph in an efficient way, where the efficiency is defined with respect to relevant underlying performance metrics. We start by reviewing the basics of Markov chains, which are the primary motion models for stochastic robotic surveillance. We then discuss the two main criteria regarding the speed and unpredictability of surveillance strategies. The central objects that appear throughout the treatment are the hitting times of Markov chains, their distributions, and their expectations. We formulate various optimization problems based on the relevant metrics in different scenarios and establish their respective properties. Expected final online publication date for the Annual Review of Control, Robotics, and Autonomous Systems, Volume 4 is May 3, 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Download Full-text

Entropy of killed-resurrected stationary Markov chains

Journal of Applied Probability ◽

10.1017/jpr.2020.81 ◽

2021 ◽

Vol 58 (1) ◽

pp. 177-196

Author(s):

Servet Martínez

Keyword(s):

Markov Chain ◽

Random Walk ◽

Probability Measure ◽

Markov Chains ◽

Stochastic Matrix ◽

Hitting Times ◽

Stationary Distributions ◽

Natural Factors ◽

Absorbing States ◽

Stationary Representation

AbstractWe consider a strictly substochastic matrix or a stochastic matrix with absorbing states. By using quasi-stationary distributions we show that there is an associated canonical Markov chain that is built from the resurrected chain, the absorbing states, and the hitting times, together with a random walk on the absorbing states, which is necessary for achieving time stationarity. Based upon the 2-stringing representation of the resurrected chain, we supply a stationary representation of the killed and the absorbed chains. The entropies of these representations have a clear meaning when one identifies the probability measure of natural factors. The balance between the entropies of these representations and the entropy of the canonical chain serves to check the correctness of the whole construction.

Download Full-text

On identifiability and order of continuous-time aggregated Markov chains, Markov-modulated Poisson processes, and phase-type distributions

Journal of Applied Probability ◽

10.1017/s0021900200100087 ◽

1996 ◽

Vol 33 (03) ◽

pp. 640-653 ◽

Cited By ~ 3

Author(s):

Tobias Rydén

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Continuous Time ◽

Poisson Processes ◽

Hitting Times ◽

Phase Type ◽

Phase Type Distributions ◽

Minimum Number ◽

Markov Modulated ◽

Two Parameters

An aggregated Markov chain is a Markov chain for which some states cannot be distinguished from each other by the observer. In this paper we consider the identifiability problem for such processes in continuous time, i.e. the problem of determining whether two parameters induce identical laws for the observable process or not. We also study the order of a continuous-time aggregated Markov chain, which is the minimum number of states needed to represent it. In particular, we give a lower bound on the order. As a by-product, we obtain results of this kind also for Markov-modulated Poisson processes, i.e. doubly stochastic Poisson processes whose intensities are directed by continuous-time Markov chains, and phase-type distributions, which are hitting times in finite-state Markov chains.

Download Full-text

Asymptotic behaviour of sample weighted circuits representing recurrent Markov chains

Journal of Applied Probability ◽

10.1017/s0021900200039103 ◽

1990 ◽

Vol 27 (03) ◽

pp. 545-556 ◽

Cited By ~ 2

Author(s):

S. Kalpazidou

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Asymptotic Behaviour ◽

Probabilistic Interpretation ◽

Stationary Markov Chain

The asymptotic behaviour of the sequence (𝒞 n (ω), wc,n (ω)/n), is studied where 𝒞 n (ω) is the class of all cycles c occurring along the trajectory ωof a recurrent strictly stationary Markov chain (ξ n ) until time n and wc,n (ω) is the number of occurrences of the cycle c until time n. The previous sequence of sample weighted classes converges almost surely to a class of directed weighted cycles (𝒞∞, ω c ) which represents uniquely the chain (ξ n ) as a circuit chain, and ω c is given a probabilistic interpretation.

Download Full-text

On the absorption probabilities and mean time for absorption for discrete Markov chains

Monte Carlo Methods and Applications ◽

10.1515/mcma-2021-2084 ◽

2021 ◽

Vol 0 (0) ◽

Author(s):

Nikolaos Halidias

Keyword(s):

Markov Chain ◽

Random Walk ◽

Markov Chains ◽

Generating Function ◽

Discrete Time ◽

Probability Generating Function ◽

The Mean ◽

Mean Time ◽

Absorption Probabilities

Abstract In this note we study the probability and the mean time for absorption for discrete time Markov chains. In particular, we are interested in estimating the mean time for absorption when absorption is not certain and connect it with some other known results. Computing a suitable probability generating function, we are able to estimate the mean time for absorption when absorption is not certain giving some applications concerning the random walk. Furthermore, we investigate the probability for a Markov chain to reach a set A before reach B generalizing this result for a sequence of sets A 1 , A 2 , … , A k {A_{1},A_{2},\dots,A_{k}} .

Download Full-text

Proportional lumpability and proportional bisimilarity

Acta Informatica ◽

10.1007/s00236-021-00404-y ◽

2021 ◽

Author(s):

Andrea Marin ◽

Carla Piazza ◽

Sabina Rossi

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Upper And Lower Bounds ◽

General Definition ◽

Performance Indices ◽

State Aggregation ◽

Structural Regularity ◽

Original Definition ◽

Aggregation Technique ◽

Definition Of

AbstractIn this paper, we deal with the lumpability approach to cope with the state space explosion problem inherent to the computation of the stationary performance indices of large stochastic models. The lumpability method is based on a state aggregation technique and applies to Markov chains exhibiting some structural regularity. Moreover, it allows one to efficiently compute the exact values of the stationary performance indices when the model is actually lumpable. The notion of quasi-lumpability is based on the idea that a Markov chain can be altered by relatively small perturbations of the transition rates in such a way that the new resulting Markov chain is lumpable. In this case, only upper and lower bounds on the performance indices can be derived. Here, we introduce a novel notion of quasi-lumpability, named proportional lumpability, which extends the original definition of lumpability but, differently from the general definition of quasi-lumpability, it allows one to derive exact stationary performance indices for the original process. We then introduce the notion of proportional bisimilarity for the terms of the performance process algebra PEPA. Proportional bisimilarity induces a proportional lumpability on the underlying continuous-time Markov chains. Finally, we prove some compositionality results and show the applicability of our theory through examples.

Download Full-text