Continuous-time Markov chains
A continuous-time Markov chain (CTMC) describes a random system that can change state at any time. Unlike a discrete-time Markov chain, we do not wait for fixed daily or weekly update points. Instead, biological events occur at random continuous times.
Why continuous time?
In a real epidemic, an infection might occur at 2.37 days and a recovery at 2.91 days. There is no biological reason that events must occur exactly at the end of each day.
A CTMC therefore treats time as continuous:
\[t\ge0.\]but the state can still be discrete. If \(I(t)\) counts infectious people, then
\[I(t)\in\{0,1,2,\ldots,N\}.\]What does one CTMC trajectory look like?
The process remains in one state while waiting for an event. When an event occurs, the state jumps. It then waits again for the next event.
The horizontal sections are waiting periods. The vertical jumps are biological events.
Start with a simple SIS epidemic
Suppose the population size is \(N\) and the current number of infectious individuals is
\[I(t)=i.\]Then the number susceptible is
\[S(t)=N-i.\]Two events can change the state:
| Event | State change |
|---|---|
| infection | \(i\to i+1\) |
| recovery | \(i\to i-1\) |
Transition rates
Instead of directly specifying a probability for the next fixed time step, a CTMC specifies rates.
For the SIS model, take
\[b(i)=\beta\frac{(N-i)i}{N}\]as the total infection-event rate and
\[d(i)=\gamma i\]as the total recovery-event rate when the current state is \(i\).
From a rate to probability over a short interval
For a sufficiently small interval \(\Delta t\),
\[\boxed{P(\text{infection in }\Delta t)\approx b(i)\Delta t},\]\[\boxed{P(\text{recovery in }\Delta t)\approx d(i)\Delta t}.\]The probability of no event is approximately
\[P(\text{no event in }\Delta t)\approx1-[b(i)+d(i)]\Delta t.\]The probability of two or more events in such a very short interval is of smaller order and becomes negligible relative to \(\Delta t\) as \(\Delta t\to0\).
The CTMC transition diagram
From an interior SIS state \(i\), there are two possible jumps:
Notice that the arrows are labelled by rates, not one-step probabilities. This is a fundamental difference from the DTMC transition diagram.
Total rate of leaving the current state
When \(I(t)=i\), either an infection or a recovery can be the next event. The total event rate is therefore
\[\boxed{a(i)=b(i)+d(i)}.\]This total rate controls how long the process waits before some event occurs.
How long until the next event?
If the total rate is \(a(i)\), the waiting time \(T\) to the next event has an exponential distribution:
\[\boxed{T\sim\operatorname{Exp}(a(i))}.\]Its mean is
\[\boxed{E[T]=\frac{1}{a(i)}}.\]For \(a(i)=3.7\) per day,
\[E[T]=\frac{1}{3.7}\approx0.270\text{ days}.\]This is the mean waiting time, not a fixed waiting time. The actual next event may occur earlier or later.
Why the waiting time is random
The probability that no event has occurred by time \(t\) is
\[P(T>t)=e^{-a(i)t}.\]Therefore the probability that at least one event has occurred by time \(t\) is
\[P(T\le t)=1-e^{-a(i)t}.\]As the total rate increases, the process tends to wait less time for the next event.
Which event occurs?
Once we know that an event occurs, the relative rates determine its type:
\[\boxed{P(\text{infection next})=\frac{b(i)}{b(i)+d(i)}},\]\[\boxed{P(\text{recovery next})=\frac{d(i)}{b(i)+d(i)}}.\]With \(b(i)=2.7\) and \(d(i)=1.0\),
\[P(\text{infection next})=\frac{2.7}{3.7}\approx0.730,\]\[P(\text{recovery next})=\frac{1.0}{3.7}\approx0.270.\]So the model separates the next transition into two questions:
A complete numerical SIS step
Let
\[N=100,\qquad i=10,\qquad\beta=0.30,\qquad\gamma=0.10.\]Then
\[b(10)=0.30\frac{90\times10}{100}=2.7,\qquad d(10)=0.10(10)=1.0.\]Hence
\[a(10)=3.7.\]The waiting time is sampled from
\[T\sim\operatorname{Exp}(3.7),\]and, when the event occurs, infection is selected with probability approximately 0.730 and recovery with probability approximately 0.270.
If infection is selected, the new state is 11. The rates are then recalculated using \(i=11\). If recovery is selected, the new state is 9 and the rates are recalculated using \(i=9\).
How a CTMC simulation proceeds
This is the basic logic behind the Gillespie stochastic simulation algorithm developed later.
Generating the waiting time from a random number
If \(U\) is uniformly distributed on \((0,1)\), an exponential waiting time can be generated using
\[\boxed{T=-\frac{\ln U}{a(i)}}.\]For example, if \(a(i)=3.7\) and a simulation produces \(U=0.40\), then
\[T=-\frac{\ln(0.40)}{3.7}\approx0.248\text{ days}.\]The simulation clock therefore advances by approximately 0.248 days before the next event.
The Markov property in continuous time
A CTMC satisfies the Markov property. Once the present state is known, the probability law of future states does not require additional knowledge of the earlier path.
For the simple SIS model, knowing \(I(t)=i\) determines \(S(t)=N-i\), the infection rate \(b(i)\), the recovery rate \(d(i)\), the waiting-time distribution and the probabilities of the possible next events.
Why exponential waiting times fit the Markov property
The exponential distribution is memoryless:
\[P(T>s+t\mid T>s)=P(T>t).\]If the process has already waited \(s\) units without an event, the remaining waiting-time distribution does not depend on how long it has already waited.
This is exactly compatible with a CTMC whose current state contains all the information needed for future evolution.
General transition rates
For a general CTMC, let \(q_{ij}\) be the transition rate from state \(i\) to a different state \(j\). For sufficiently small \(\Delta t\),
\[P(X(t+\Delta t)=j\mid X(t)=i)=q_{ij}\Delta t+o(\Delta t),\qquad i\ne j.\]The notation \(o(\Delta t)\) represents terms that become negligible compared with \(\Delta t\) as \(\Delta t\to0\).
The total rate of leaving state \(i\) is
\[a_i=\sum_{j\ne i}q_{ij}.\]Conditional on leaving state \(i\), the probability that the next state is \(j\) is
\[\boxed{\frac{q_{ij}}{a_i}}.\]From transition rates to the generator matrix
All transition rates can be organised into a generator matrix \(Q\). For \(i\ne j\), the entry \(q_{ij}\) is the rate from state \(i\) to state \(j\).
The diagonal entries are defined by
\[q_{ii}=-\sum_{j\ne i}q_{ij}.\]Therefore each row of \(Q\) sums to zero.
Boundary and absorbing states
In an SIS epidemic without imported infections, \(i=0\) is absorbing. There are no infectious people to transmit infection and no infectious people to recover, so
\[b(0)=d(0)=0.\]Once the process reaches 0, it remains there.
At \(i=N\), no additional infection is possible because there are no susceptible individuals, so \(b(N)=0\), although recoveries may still occur.
DTMC versus CTMC
| DTMC | CTMC |
|---|---|
| changes considered at fixed time steps | events can occur at any continuous time |
| specifies one-step transition probabilities | specifies transition rates |
| next update time is fixed | next event time is random |
| transition matrix \(P\) | generator matrix \(Q\) |
| rows of \(P\) sum to 1 | rows of \(Q\) sum to 0 |
Why CTMCs are useful in biology
Many biological systems are naturally event-driven. Infections, recoveries, births, deaths, mutations and chemical reactions do not usually occur at predetermined clock times. CTMCs allow these events to occur individually at random times while retaining explicit biological event rates.
They are particularly useful when population numbers are small, extinction matters, event timing matters, or we need probabilities of different possible outcomes rather than only an average trajectory.
What comes next?
The SIS CTMC above is an example of a birth–death process: the state can increase by one or decrease by one. The next section develops this important class of CTMCs more systematically.