Measuring Transformative Outcomes: A Multi-Level Approach for Evaluating Transformative Processes

Up to now, evaluations in the research, technology and innovation context primarily evolved around the question
of programme effectiveness and efficiency. At the core lies the question to what extent (pre-defined) programme
objectives have been achieved and if value for money has been delivered. In our research, we are
widening the perspective. We are quantitatively assessing the perception of transition processes addressing
wider societal challenges among experts in the field. By better understanding the progress of transition processes
we lay the empirical foundations how to assess the contribution of transformative research and innovation
(R&I) programmes to such processes. This research draws on survey data that has been collected in the
context of the accompanying evaluation for Germany‘s “7th Energy Research Programme - Innovations for the
Energy Transition” (EFP) (BMWK 2018). In the survey, which forms the basis for the present analysis, project
managers were asked to share their perspective on general developments in the energy transition and in energy
research in Germany, in their role of being experts in these fields. The research question of how to effectively
measure transition processes is addressed using three models rooted in distinct theoretical frameworks and
tested comparing construct validity across different models. To our knowledge it is the first to quantitatively
operationalize the concept of transformative outcomes and to test how the concept can be used to quantitatively
track transition processes. The results of the study confirm our hypothesis that a more complex model, which
incorporates transformative outcomes as latent factor variables, displays higher construct validity compared
to two alternative, less complex models: one that only reflects the multi-level perspective, and an even less
complex single factor model. Thus, our research highlights the importance of distinguishing between transition
processes on a detailed level of transformative outcomes when trying to track their progress in quantitative
terms. We expect that our approach can be replicated and may form a viable basis for future research on
transformation-oriented R&I programs and policies, while contributing to enhanced learning for policy makers,
program owners, and actors within the program.