Computational models can be hard to test. They have large input spaces (often hundreds of parameters). A single execution can take hours days. They can be inherently non-deterministic. Thus, running large numbers of carefully controlled tests is not feasible. Furthermore, given that they are often used to explore behaviours of unfamiliar phenomena, there may be a lack of certainty around what constitutes a “correct” output.

Causal Inference offers promising solutions to these problems. It provides a framework to enable some lightweight domain-knowledge into the analysis process. This can enable the tester to answer questions about the (causal) relationships between inputs (or parameters) and outputs. These can be answered without the need for large numbers of controlled tests.

This work has been funded by the EPSRC (CITCOM grant - 2020-2025)

Research Contributions

The CITCOM project runs from 2020-2025. The core conceptual contributions are listed and linked to their relevant papers here. A more complete bibliography is provided below.

• To show how causal tests can be characterised as a specific form of Metamorphic relationship. (ICST’23)
• To enable metamorphic relationships to be established from test executions without the need to control inputs. (TOSEM’23)
• To provide a statistical test adequacy metric for causal test cases (ICST’24)
• Illustrating how Causal Inference can test systems with hidden and interacting variables (EASE’25 – awarded the best research paper award)
• Illustrating how Causal Inference can be used to minimise sequential test cases of cyber-physical systems (TOSEM’26) - to appear

Software: The CITCOM Causal Testing Framework

A tool implementing the CITCOM Causal Testing approach has been made available on GitHub under an MIT License.

• CITCOM Causal Testing Framework GitHub Repository
• Accompanying link to Journal of Open Source Software paper (JOSS’25)

Target Systems

We have primarily applied Causal Inference to test scientific computational models and cyberphysical systems. Specific examples include:

• Scientific software models (covered in our TOSEM’23 paper)
  • The CovaSim COVID pandemic simulator. We showed how the effect of different Covid variants could be accurately established from uncontrolled data, despite the variation of a large number of parameters.
  • The Luo-Rudy Cardiac Action Potential model. We replicated a sensitivity analysis of the parameters, but without requiring large numbers of controlled inputs.
  • A Poisson Line Tessellation model. Highly stochastic behaviour makes this hard to test (using traditional techniques). We showed how this could be managed with causal techniques.
  • Recently we have extended this work into a practical setting, working on testing national water resource models with the UK Environment Agency.

• Cyber-physical systems

  • We have tested models of Artificial Pancreas Systems. In these systems a diabetic user is fitted with a glucose-monitor, and an insulin pump. The dosage of insulin is moderated by a controller that monitors the glucose-levels. This presents an interesting testing challenge, because a lot of the behaviour depends on factors in the human body which cannot be controlled. Details on the testing problem itself and on a Digital Twin test harness to support testing this are available in Richard Somers’ STVR paper on the topic (STVR’25).
  • Automated driving simulators (ADSs) present a particular testing problem, because there are lots of factors that sit outside of control of the tester, and there are lots of hidden variables. We show how causal testing can be used to test ADSs in our EASE’25 paper.
  • We are currently testing Smart Manufacturing Systems. This work is being carried out by Joel Hogg, who is working on his Ph.D. as part of the EPSRC Made4Manufacturing CDT in Machining, Assembly and Digital Engineering for Manufacturing.