GenAI experiments: Extracting knowledge from educational materials (data set)

This collection contains the results of four ClimEmpower / MAIA GenAI experiments. These experiments aim to asess how and to what extent the Generative AI models can help knowledge curators extract knowledge from documents they need to analyse.

Concrete high-level research questions these experiments aim to resolve are:

RQ1: To what extent can the AI answers be used to formulate the final answers, without reading the whole document?

RQ2: Which types of questions are easier or more difficult for GenAI models to answer?

RQ3: How, and to what extent, can the answers be improved through prompt engineering?

RQ4: To what extent do the GenAI models follow instructions to base the answers (only) on the content provided in the document?

RQ5: How does the choice of GenAI model reflect in experiment results?

In addition, we were also interested in finding out the ways to further improve the SumQA, a Generative AI service that was developed in the MAIA project and supports batch-processing of documents.