Computational models can be hard to test. They have large input spaces (often hundreds of parameters). A single execution can take hours days. They can be inherently non-deterministic. Thus, running large numbers of carefully controlled tests is not feasible. Furthermore, given that they are often used to explore behaviours of unfamiliar phenomena, there may be a lack of certainty around what constitutes a "correct" output.
Causal Inference offers promising solutions to these problems. It provides a framework to enable some lightweight domain-knowledge into the analysis process. This can enable the tester to answer questions about the (causal) relationships between inputs (or parameters) and outputs. These can be answered without the need for large numbers of controlled tests.
The CITCOM project runs from 2020-2025. The core conceptual contributions are listed and linked to their relevant papers here. A more complete bibliography is provided below.
A tool implementing the CITCOM Causal Testing approach has been made available on GitHub under an MIT License.
Causal testing is quite a flexible approach and can be applied to a range of classes of systems. We are currently in a phase of the project where we are exploring its application to a range of types of systems. These include:
We have particularly focussed on Artificial Pancreas Systems. In these systems a diabetic user is fitted with a glucose-monitor, and an insulin pump. The dosage of insulin is moderated by a controller that monitors the glucose-levels. This presents an interesting testing challenge, because a lot of the behaviour depends on factors in the human body which cannot be controlled.
Details on the testing problem itself and on a Digital Twin test harness to support testing this are available in Richard Somers' recent STVR paper on the topic (STVR'25).
Automated driving systems (ADSs) present a particular testing problem, because there are lots of factors that sit outside of control of the tester, and there are lots of hidden variables. We show how causal testing can be used to test ADSs in a forthcoming EASE'25 paper.
This work has been funded by the EPSRC CITCOM grant.