Course Outline
Markov Chains
MARKOV CHAINS
Discrete-time chains
- Definition and basic properties, the transition matrix. Calculation of n-step transition probabilities. Communicating classes, closed classes, absorption, irreducibility. Calculation of hitting probabilities and mean hitting times; survival probability for birth and death chains. Stopping times and statement of the strong Markov property. [5]
- Recurrence and transience; equivalence of transience and summability of n-step transition probabilities; equivalence of recurrence and certainty of return. Recurrence as a class property, relation with closed classes. Simple random walks in dimensions one, two and three. [3]
- Invariant distributions, statement of existence and uniqueness. Mean return time, positive recurrence; equivalence of positive recurrence and the existence of an invariant distribution. Convergence to equilibrium for irreducible, positive recurrent, aperiodic chains and proof by coupling. *Long-run proportion of time spent in given state*. [3]
- Time reversal, detailed balance, reversibility; random walk on a graph. [1]
Appropriate books
- G.R. Grimmett and D.R. Stirzaker Probability and Random Processes. OUP 2001
- G.R. Grimmett and D. Welsh Probability, An Introduction. OUP, 2nd edition, 2014
- J.R. Norris Markov Chains. CUP 1997