Assessing machine learning-based pandemic crisis prediction and management tools in STADEM trials

The problem of evaluating pandemic crisis prediction and management tools (PCPMT) involves multiple categories of stakeholders as well as tools that’s (computational) nature greatly differs. As the recent events showed, the relevance of robust evaluation methodologies cannot be overstated. Misapplication of existing tools to unsupported health crises situations leads to mismanagement of limited resources, ultimately causing unnecessary loss of human life and downstream deterioration of population-wide health characteristics.
The situation is further complicated by the fact that PCPMT should facilitate foresight and thus inform effectual health, economic, and other policies at the national as well as European level. During health crises that reach the pandemic level it is difficult to estimate the confidence level of inferences based on existing evidence, as it is often the case that the future will be foundationally divorced from the past and present. Therefore, the existing evidence does not always represent a reliable guide to the optimal decision-making.
The epistemic limits of existing evidence apply not only to the decision- and policymakers but also to the PCPMT that are being used. This is a result of evidence-based assumptions informing the software development processes that produce PCPMT. This document defines the lifecycle of machine learning-based PCPMT and its stages that need to be observed by the stakeholders involved in the PCPMT development and use. In case that the trial guidance methodology is applied to PCPMT, it ensures that the risks following from PCPMT uses are qualified in full, thus increasing the accountability of stakeholders.
The trial guidance methodology (TGM), CWA 17514:2020, was developed as part of the DRIVER+ project as a methodology for assessing innovations of crisis management processes (CM). The methodology allows practitioners to assess new solutions without heavy investments required for the full rollouts of the intended solutions. The STAMINA project adapted the taxonomy of crisis management functions developed by the DRIVER+ project to reflect processes occurring during the pandemic crisis prediction and management within and across European borders. The result of this process is an TGM-based STADEM methodology developed by STAMINA, defining a pandemic management functions taxonomy that connects general descriptions of computational tools with the problems that need to be solved in pandemic management. This CWA XXX uses TGM/STADEM to show how the machine learning lifecycle can be used to define, develop, verify, deploy, monitor, and update PCPMPT. STADEM focuses on pandemic crisis management functions and uses an adapted TGM-based functions taxonomy.