4 The Long Run Behavior of Markov Chains 4.1 Regular Transition Probability Matrices Suppose that a transition probability matrix P = P ij on a finite number of states labeled 0, 1, . . . , N has the property that when raised to some power k, the matrix P k has all of its elements strictly positive. Such a transition probability matrix, or the corresponding Markov chain, is called regular. The most important fact con- cerning a regular Markov chain is the existence of a limiting probability distribution = ( 0 , 1 , . . . , N ), where j > 0 for j = 0, 1, . . . , N and j j = 1, and this distribu- tion is independent of the initial state. Formally, for a regular transition probability matrix P = P ij , we have the convergence (n) lim P n ij = j > 0 for j = 0, 1, . . . , N, or, in terms of the Markov chain {X n },