7.2 Discrete-Time Markov Chains

The first study on MCs dates back to 1907, when the Russian mathematician Andrei Andreyevich Markov defined and investigated the properties of this specific type of random process [1]. Since then, the literature on the subject has kept growing in size and depth, producing a number of interesting results that find application in several different fields. In particular, a branch of MC theory has originated the so-called queueing theory, which will be presented in Chapter 8.

7.2.1 Definition of Discrete-Time MC

Let , with , be a discrete-time rp that takes values in a countable state space . For ease of notation, in the following we will use in place of to denote the random process at the th time instant. Furthermore, unless otherwise stated, the state space will be labeled here by the non-negative integers, that is to say, . Hence, is also discrete valued.