Stochastics Processes — Part 1 : Getting started with a mathematical introduction and simple… (2025)

A few months back, one of our clients from a big pharma company came to us and asked us to predict the enrolment rate for patients getting onboarded onto clinical trials for different drugs in development. The problem was simple: our client wanted to understand which patient enrolment sites were performing well/which needed to be shut down as different sites(or clinics) had different enrolment rates of patients getting enrolled for different drugs. Whatever was the profile of the drug, or the geography of the site, one thing was quite clear while looking at the data: due to severe sparsity in the data, there was a heavy element of randomness involved with the enrolment. Some sites which might have enrolled patients at a fast pace during the initial phase of the site’s activation, might become dead but then regain the pace of enrolment. Other sites which started slow for the same drug in the same geography, might become dead, and then enroll at a fast pace again for a few weeks, before becoming dead again. Given this extreme randomness observed in the data, we decided that modelling this process could be done best using stochastic random walk simulations. Why? Because deterministic models don’t work well when trying to capture inherently unpredictable, random phenomena. To delve deeper into the problem statement and how we solved it using a combination of random walk simulations from poisson and binomial distributions, please refer to the following links:

Part 1 of the blog :

Part 2 of the blog :

While solving the above problem, I personally got quite interested in stochastics once again after college and have been reading quite a lot on it recently. In light of this study and the beauty that is stochastic processes, this is the first in a series of introductory stochastics blogs. This blog will give a gentle introduction of stochastic processes and MCMC modelling, with the underlying mathematical foundations and applications in R language. Although there are other types of models to handle randomness and uncertainty in different ways, models such as chaotic process (population growth models, fluid turbulence), fuzzy logic models (control systems, decision support systems) or agent based systems, covering them makes more sense once stochastic modelling basics have been understood. It is anyways quite useful to study stochastics for any engineering, data science or finance major, given the various applications stochastic processes find in modelling uncertainty/randomness. The exact applications of stochastic processes will be dealt with in another blog post.

How stochastic processes are defined ?

A stochastic process

is a collection of random variables, where the random variables are indexed by time, and the entire set of time stamps is called set T.

Whatever possible values of X(t) can be achieved, are stored in the State Space or set S of the stochastic process, for any t in T.

There can be two types of stochastic processes based on countability of set T, i.e., discrete time processes for when T is countable, or continuous time processes for when T is infinite.

What are discrete time Markov chains

When a discrete time stochastic process behaves in such a way that in a sequence of events indexed by time, any event being in a given state (let’s say state j) depends only on the it’s previous event (let’s say state i) and not on any other event before or after it, then such a process is called a discrete time Markov chain. Stochastic processes behaving in such a way are said to be Markovian in nature, such that they follow the following system of conditional probabilities

where Pᵢⱼ is called the one-step transition probability of the markov chain

If you think about it, this gives a unique way to simplify an extremely complicated random process.

Let’s say there is an event with 4 different states A,B,C and D at time = 0,1,2,3 respectively, as shown in figure below

Let’s suppose that each successive state, with the passage of time, depends on all the previous states. In such a case,

• state D will depend simultaneously on state C,B and A
• state C will depend simultaneously on state B and A
• state B will depend on state A
• state A is the beginning state at time=0, so it won’t have any dependance on other states.

Can you imagine, for the above process, how a mathematical equation will be used to represent the relationships, and how difficult it can be to track the changes in a given state with a change in any of the previous states!

In essence, if we were to solve the process in Fig 2, we have to keep track of the history of the process, meaning the algorithm needed to solve the process needs to have a memory of the past states.

For a Markovian process, we neglect the history of the process (Fig 3), as in, we assume that any given state at time t is only dependent on the previous state, and not any other state, as shown in Fig 4. It is called a chain since just like a chain, each state is linked to the state coming before it.

One Step transition probability in Markov Chains

The probability Pᵢⱼ discussed earlier, for different values of i and j in the state space, can be put together in a matrix form as given below:

As an example, consider a student who has 3 states as follows:

• He can be at Home (H)
• He can be at School (S)
• He can be in the Playground (P)

For each state, whenever he steps out, he can be either return back to the same state or transition to another state, as given in the one step transition probability matrix below.

To understand the above One step transition probability matrix, let’s take the case of when the student is initially at Home, i.e., row 1 of the matrix above. If the student steps out of Home, he will return back to Home with probability 0.3, or he will go to School with probability 0.4 or to the Playground with probability 0.3. Similarly for each row, you can evaluate the one step transition probabilities for each combination.

It is important to note that all the probabilities in each row must sum up to 1, because if the student is in a state i, the next state must be one the possible states.

Vice versa, let’s say if the sum is not 1 but 0.9 for one of the rows, it would mean that there is a leakage in state the student can be in by a probability of 0.1. That is, for 10% of the times (or probability of 0.1), we do not know where the student will be. This absolutely cannot be the case as the student needs to be somewhere.

Calculating Conditional and Joint probabilities in Markov chains using one step transition probability matrix

For the above example on the states a student can be in during a Markovian process, the conditional and joint probabilities are quite easy to calculate if we keep the basic conditional probability rule given in Equation 2 earlier.

As a refresher, joint probability is the probability of two (or more) events happening together, whereas conditional probability is the probability of an event happening, given the occurrence of another event.

So, lets try to calculate the conditional probability that the student is at the Playground (at time t=2), given he first was at his Home (at time t=0) and then at his School (at time t=1), that is :

is calculated using the Markovian rule (given in Equation 2) as follows:

Note that Pₛₚ denotes the one step transition probability from School to Playground, as seen from Fig 6.

Now, lets also calculate the joint probability that the student starts from being at School at time t=0, then goes to the Playground at time t = 1 and then ends at home at time t=2, that is:

To solve Equation 5 above, we first apply the chain rule of probability as follows:

Now, we can apply the Markovian conditional property rule as follows:

Combining Equation 7 with Equation 6 and referring for appropriate one step transition probabilities from Fig 6, we get

Application in R : Diagrammatic representation of Markov Chain using R

As a simple introduction, we will introduce how to plot the Markovian process for the different states of the student using the diagram package in R.

We first make a matrix which mimics the one-step transition probability matrix discussed in Step-6

> trn.prob.mat <- matrix(c(0.3, 0.4, 0.3, 0.0, 0.3, 0.7, 0.6, 0.1, 0.3), nrow=3, ncol=3,byrow=TRUE)
> print(trn.prob.mat)
 [,1] [,2] [,3]
 [1,] 0.3 0.4 0.3
 [2,] 0.0 0.3 0.7
 [3,] 0.6 0.1 0.3

We then take the transpose of the one-step transition probability matrix as the diagram::plotmat function in R rearranges the matrix for plotting, so only transpose (and not the original matrix) works for correct arrow directions. This is because, the original matrix in Fig 6 has columns representing ‘to’ relation, while the diagram::plotmat function has columns representing ‘from’ relation. Please refer the documentation of diagram::plotmat function in CRAN for more info on this, given in References section of this blog at the end.

> transpose.trn.prob.mat <- t(trn.prob.mat) # t() is the transpose function in R
> print(transpose.trn.prob.mat)
 [,1] [,2] [,3]
 [1,] 0.3 0.0 0.6
 [2,] 0.4 0.3 0.1
 [3,] 0.3 0.7 0.3

install.packages("diagram") # Functions for Visualising Simple Graphs (Networks), Plotting Flow Diagrams
library(diagram)
plotmat(transpose.trn.prob.mat, # square coefficient matrix, specifying the links (rows=to, cols=from).
 pos = c(1,2), # vector, specifying the number of elements in each graph row
 name = c("H", "S", "P"), # string vector, specifying the names of elements
 # for all the below arguments, refer the official documentation of plotmat function
 arr.length = 0.5, arr.width = 0.5, arr.type = "curved",
 shadow.size = 0,
 box.col = "light blue", box.lwd = 0.9, box.prop = 0.6, box.size = 0.15, box.type = "circle", 
 cex.txt = 1, lwd = 1, self.cex = 0.5, self.shiftx = 0.17, self.shifty = -0.01)

With this, we conclude the Part 1 of our blog series on stochastic processes. Please watch the below video for a better understanding of the concepts discussed in this blog.

In the next part (Part 2 : link given below) of this blog series, we will discuss about getting and using the n-step transition probability using Chapman-Kolmogorov Equation, as well as the defining the different states of a Markovian process, along with implementation of these concepts in R.

References:

1. ‘diagram’ package in R : https://cran.r-project.org/package=diagram/vignettes/diagram.pdf