---

Markov chains

---

• Markov processes
  • Definition:

    A Markov process (MP) { X(t), t ∈ T} is a process such that: Ҏ[ X(t+1) = xt+1 | X(1) = x1, X(2) = x2, ..., X(t) = xt ] = Ҏ[ X(t+1) = xt+1 | X(t) = xt ] In other words: to determine the next state X(t+1), we only have to look at the present state X(t). I.e., the continuation of a Markov process depends only on the current state and does not depend on any historical states

  ---

  ---

• Categories of Markov processes
  • Markov processes are categorized by the nature of the time and space domains:

    { X(t), t ∈ T } ^ ^ | | | +--- Time domain +------ (State) Space domain

    ---

  • Categories of Markov processes:

    State space domain | Discrete | Continuous -----+------------------+----------------- Time dom: | | Discrete | Discrete time | Discrete time, | Markov chain | continuous space | | Markov process -----+------------------+------------------ | | Continuous | Continuous time | Diffusion | Markov chain | processes | | -----+------------------+------------------

  ---

  ---

• Example Markov Chains
  • X(t) = number of customers in the system

    t ∈ [0, ∞) (all positive time values) X(t) ∈ [0, 1, 2, .... ) (# of customers) Ҏ[ X(t+1) = k-1 | X(t) = k ] = probab that a customer departs in the next time unit Ҏ[ X(t+1) = k+1 | X(t) = k ] = probab that a customer arrives in the next time unit