Stochastic Processes for Data Science and Engineering

An introduction to stochastic processes with emphasis on modern applications and computational implementation, this course builds from foundational concepts through discrete-time Markov chains, branching processes, and continuous-time processes including Poisson processes, birth-death processes, and Brownian motion. Each unit pairs mathematical rigour with applications across artificial intelligence, robotics, economics, business, finance, and biological systems.

This course provides a comprehensive introduction to stochastic processes with a strong emphasis on modern applications and computational implementation. The course is structured to build knowledge progressively, starting with basic stochastic process concepts and advancing through discrete-time Markov chains, Branching process, continuous-time processes such as Poisson processes, birth-death processes, and Brownian motion. Each unit combines rigorous mathematical treatment with contemporary applications across multiple domains including artificial intelligence, robotics, economics, business, finance, and biological systems.

By the end of this course, students should be able to:

• Demonstrate comprehensive understanding of stochastic process definitions, classifications, and fundamental properties
• Analyze discrete-time Markov chains including state classification, limiting behaviour, and ergodic properties
• Understand and apply Poisson processes, birth-death processes, and their applications
• Analyze Brownian motion, its variations, and applications in finance and physics
• Implement simulation algorithms for various stochastic processes using programming tools
• Apply stochastic models to analyze real-world data and extract meaningful insights
• Model complex systems using appropriate stochastic frameworks across multiple domains
• Design and execute research projects involving stochastic modelling

Upon completion of this course, students will be able to:

• Understand fundamental stochastic models and their  properties - theoretical
• Model complex systems across multiple domains using appropriate stochastic frameworks - applied
• Implement and analyze stochastic algorithms using modern tools - computational
• Design, execute, and present original research in stochastic modelling - research
• Evaluate model assumptions, limitations - critical analysis

1. Grimmett, G and Stirzaker, D (2020). Probability and Random Processes, 4th Edition. Oxford University Press.
2. Karlin, S.and Taylor, HM (1994). An Introduction to Stochastic Modelling. Revised Edition, Academic Press.
3. Morel, PD and Penev, S (2014). Stochastic Processes: From Applications to Theory, CRC Press
4. Ross, SM (2024). Introduction to Probability Models, 13th Edition. Academic Press.
5. Sivaprasad, M and Deshmukh, S (2023). Introduction to Stochastic Processes Using R, Springer

Some advanced topics related to the course contents: (Students can do projects based on these topics collaboratively with expert faculties from different majors)

• Kalman filters and particle Filtering – Dynamic models
• Hidden Markov Models and state estimation, Viterbi’s algorithm, Baum-Welch Algorithm, Applications in speech recognition and bioinformatics
• Variance reduction techniques, MCMC Methods,  Metropolis-Hastings and Gibbs sampling, Sequential Monte Carlo methods – Applications in Bayesian estimation
• Stochastic optimization; Gradient methods and variants
• Anomaly detection in IoT networks using stochastic processes
• Uncertainty quantification in deep learning  -
• Rapidly exploring random trees (RRT) – Applications in Cyber physical systems and robotics
• Reinforcement learning and Markov decision processes (MDP) - Applications in autonomous systems
• Probabilistic SLAM (Simultaneous localization and mapping) implementation for autonomous navigation – Applications in Robotics
• Stochastic control and optimal filtering – Applications in Finance and Economics, Robotics and autonomous systems, Signal processing etc.
• Stochastic integrals and stochastic differential equations (SDEs)  - Applications in finance
• Lévy processes, Jump-diffusion models, financial modelling, Applications in high-frequency trading
• Network analysis and graph processes, Social media influence propagation using stochastic models
• SIR/SEIR (Susceptible, Exposed, Infected, Recovered) models with stochastic components and policy interventions – Applications in Epidemiology
• Protein folding and molecular dynamics, Enzyme kinetics - Applications in systems biology and bioengineering
• Any other applications of stochastic models other than those stated above.

Books for advanced and related topics:

1. Gallager, R (2013). Stochastic Processes: Theory and Applications, Cambridge University Press
2. Murphy, KP(2023). Probabilistic Machine Learning: Advanced Topics. MIT Press.
3. Norris, JR (1998). Markov Chains, Cambridge University Press.
4. Øksendal, B (2023). Stochastic Differential Equations, 6th Edition. Springer. 4.
5. Privault, N (2018). Understanding Markov Chains: Examples and Applications, 3rd Edition. Springer.
6. Privault, N (2024). Discrete Stochastic Processes: Tools for Machine Learning and Data Science, Springer
7. Thrun, S, Burgard, W and Fox, D (2005). Probabilistic Robotics, 2nd Edition. MIT Press.
8. Wilkinson, DJ (2019). Stochastic Modelling for Systems Biology, 3rd Edition. CRC Press.

Unit 1: Stochastic process fundamentals (1 Week)

• Probability, joint and conditional distributions, Conditional probability and conditional expectations
• Definition and classification of stochastic processes – Examples (Bernoulli Process, Markov chain, Branching process, Poisson process, Birth-death process, Random walk process, Weiner process/Brownian motion, ARMA process – with use cases)
• Process with independent increments, Stationary processes
• Simulation of some of the important random processes

Unit 2: Markov chains and their applications (6 Weeks)

• Discrete-time Markov chains: Transition probability matrices and Chapman-Kolmogorov equations
• State classification: recurrence, transience, periodicity
• Stationarity, ergodicity
• First passage, Stationary and limiting distributions
• Applications of Markov chains in all the four major domains of specialization
• Random Walks and Applications
• Branching Process and its applications in population dynamics
• Time Reversible Markov chains (Applications in thermodynamics)
• Application of Markov Chain, PageRank algorithm implementation and analysis; Credit scoring and customer behaviour, Weather prediction, Application of branching processes for epidemiological modelling, Labour market dynamics etc.

Unit 3: Continuous-time Markov processes (4 Weeks)

• Continuous time Markov chains
• Poisson processes and exponential distributions, Applications
• Birth-death processes and Kolmogorov equations, Applications
• Applications in queuing theory (Network packet buffering, call centers, Hospital/clinic patient flow), reliability engineering,  population dynamics, job scheduling, epidemic modelling, and business applications such as supply chain etc.

Unit 4: Brownian motion and stationary processes (3 Weeks)

• Brownian motion and properties
• Variations of Brownian motion (Geometric Brownian motion)
• Stationary and weakly stationary processes (Time series, signal processing, control systems)
• Gaussian processes for regression and classification
• Applications in Finance and risk management, Evolutionary biology, Physical diffusion processes, Control theory and signal processing, Probabilistic motion planning, Artificial intelligence and other areas

Unit 5: Other relevant topics (1 Week)

• Preliminaries on statistical inference related stochastic models
• Other relevant topics (if time permits)