Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Kristina Weishäupl; Mario Schuster; Christina Mayrl; Diana Humberger; Antonia Hoffstädter; Theresa Fischer; Bianca Punz-Reidlinger; Fabian Wiesmüller; Aaron Lauschensky; Dieter Hayn; Karl Kreiner; Bettina Fetz; Luca Brunelli; Gerhard Pölzl; Bernhard Erich Pfeifer; Gerald Slamanig; Sabrina Neururer; Günter Schreier

doi:10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Kristina Weishäupl (Autor:in, Hauptvortrag), Mario Schuster, Christina Mayrl, Diana Humberger, Antonia Hoffstädter, Theresa Fischer, Bianca Punz-Reidlinger, Fabian Wiesmüller, Aaron Lauschensky, Dieter Hayn, Karl Kreiner, Bettina Fetz, Luca Brunelli, Gerhard Pölzl, Bernhard Erich Pfeifer, Gerald Slamanig, Sabrina Neururer, Günter Schreier

Digital Health Information Systems

Publikation: Beitrag in Buch oder Tagungsband › Beitrag in Tagungsband ohne Präsentation › Begutachtung

Abstract

Background:
The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patients
and reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past.

Objective:
This work aims to analyze the inter-annotator variability among two professional groups (healthcare and
engineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,
annotators, and their professional backgrounds.

Results:
The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84).

Conclusion:
Professional background had no significant impact on the annotation of free-text HerzMobil notes.

Originalsprache	Englisch
Titel	dHealth 2023
Untertitel	Proceedings of the 17th Health Informatics Meets Digital Health Conference
Redakteure/-innen	Bernhard Pfeifer, Günter Schreier, Martin Baumgartner, Dieter Hayn
Herausgeber (Verlag)	IOS Press BV
Seiten	248 - 253
Seitenumfang	6
Band	301
ISBN (elektronisch)	978-1-64368-387-4
ISBN (Print)	978-1-64368-386-7
DOIs	https://doi.org/10.3233/SHTI230048
Publikationsstatus	Veröffentlicht - 2 Mai 2023
Veranstaltung	dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health - Apothekertrakt / Schönbrunn Palace, Vienna, Austria, Vienna, Österreich Dauer: 16 Mai 2023 → 17 Mai 2023 https://dhealth.at/

Publikationsreihe

Name	Studies in Health Technology and Informatics
Herausgeber (Verlag)	IOS Press
Band	301
ISSN (Print)	0926-9630
ISSN (elektronisch)	1879-8365

Konferenz

Konferenz	dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health
Kurztitel	dHealth 2023
Land/Gebiet	Österreich
Stadt	Vienna
Zeitraum	16/05/23 → 17/05/23
Internetadresse	https://dhealth.at/

Research Field

Exploration of Digital Health

Zugriff auf Dokument

10.3233/SHTI230048Lizenz: CC BY-NC

Andere Dateien und Links

https://www.mendeley.com/catalogue/d0a55359-949d-3345-9320-b84eee37e916/

Diese Publikation zitieren

Weishäupl, K., Schuster, M., Mayrl, C., Humberger, D., Hoffstädter, A., Fischer, T., Punz-Reidlinger, B., Wiesmüller, F., Lauschensky, A., Hayn, D., Kreiner, K., Fetz, B., Brunelli, L., Pölzl, G., Pfeifer, B. E., Slamanig, G., Neururer, S., & Schreier, G. (2023). Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. in B. Pfeifer, G. Schreier, M. Baumgartner, & D. Hayn (Hrsg.), dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference (Band 301, S. 248 - 253). ( Studies in Health Technology and Informatics; Band 301). IOS Press BV. https://doi.org/10.3233/SHTI230048

Weishäupl, Kristina ; Schuster, Mario ; Mayrl, Christina et al. / Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. Hrsg. / Bernhard Pfeifer ; Günter Schreier ; Martin Baumgartner ; Dieter Hayn. Band 301 IOS Press BV, 2023. S. 248 - 253 ( Studies in Health Technology and Informatics).

@inbook{0cab5b1c707d46e6ae044453ef40db4d,

title = "Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes",

abstract = "Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.",

keywords = "telemedicine, heart failure, Natural Language Processing, Electronic Health Records, Austria",

author = "Kristina Weish{\"a}upl and Mario Schuster and Christina Mayrl and Diana Humberger and Antonia Hoffst{\"a}dter and Theresa Fischer and Bianca Punz-Reidlinger and Fabian Wiesm{\"u}ller and Aaron Lauschensky and Dieter Hayn and Karl Kreiner and Bettina Fetz and Luca Brunelli and Gerhard P{\"o}lzl and Pfeifer, {Bernhard Erich} and Gerald Slamanig and Sabrina Neururer and G{\"u}nter Schreier",

note = "{\textcopyright} 2023 The authors, AIT Austrian Institute of Technology and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0); dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health ; Conference date: 16-05-2023 Through 17-05-2023",

year = "2023",

month = may,

day = "2",

doi = "10.3233/SHTI230048",

language = "English",

isbn = "978-1-64368-386-7",

volume = "301",

series = " Studies in Health Technology and Informatics",

publisher = "IOS Press BV",

pages = "248 -- 253",

editor = "Pfeifer, {Bernhard } and G{\"u}nter Schreier and Martin Baumgartner and Dieter Hayn",

booktitle = "dHealth 2023",

address = "Netherlands",

url = "https://dhealth.at/",

}

Weishäupl, K, Schuster, M, Mayrl, C, Humberger, D, Hoffstädter, A, Fischer, T, Punz-Reidlinger, B, Wiesmüller, F , Lauschensky, A , Hayn, D , Kreiner, K, Fetz, B, Brunelli, L, Pölzl, G, Pfeifer, BE, Slamanig, G, Neururer, S & Schreier, G 2023, Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. in B Pfeifer, G Schreier, M Baumgartner & D Hayn (Hrsg.), dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. Bd. 301, Studies in Health Technology and Informatics, Bd. 301, IOS Press BV, S. 248 - 253, dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health, Vienna, Österreich, 16/05/23. https://doi.org/10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. / Weishäupl, Kristina (Autor:in, Hauptvortrag); Schuster, Mario; Mayrl, Christina et al.
dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. Hrsg. / Bernhard Pfeifer; Günter Schreier; Martin Baumgartner; Dieter Hayn. Band 301 IOS Press BV, 2023. S. 248 - 253 ( Studies in Health Technology and Informatics; Band 301).

Publikation: Beitrag in Buch oder Tagungsband › Beitrag in Tagungsband ohne Präsentation › Begutachtung

TY - CHAP

T1 - Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

AU - Schuster, Mario

AU - Mayrl, Christina

AU - Humberger, Diana

AU - Hoffstädter, Antonia

AU - Fischer, Theresa

AU - Punz-Reidlinger, Bianca

AU - Wiesmüller, Fabian

AU - Lauschensky, Aaron

AU - Hayn, Dieter

AU - Kreiner, Karl

AU - Fetz, Bettina

AU - Brunelli, Luca

AU - Pölzl, Gerhard

AU - Pfeifer, Bernhard Erich

AU - Slamanig, Gerald

AU - Neururer, Sabrina

AU - Schreier, Günter

A2 - Weishäupl, Kristina

A2 - Pfeifer, Bernhard

A2 - Schreier, Günter

A2 - Baumgartner, Martin

A2 - Hayn, Dieter

N1 - © 2023 The authors, AIT Austrian Institute of Technology and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0)

PY - 2023/5/2

Y1 - 2023/5/2

N2 - Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.

AB - Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.

KW - telemedicine

KW - heart failure

KW - Natural Language Processing

KW - Electronic Health Records

KW - Austria

UR - https://www.mendeley.com/catalogue/d0a55359-949d-3345-9320-b84eee37e916/

U2 - 10.3233/SHTI230048

DO - 10.3233/SHTI230048

M3 - Conference Proceedings without Presentation

C2 - 37172189

SN - 978-1-64368-386-7

VL - 301

T3 - Studies in Health Technology and Informatics

SP - 248

EP - 253

BT - dHealth 2023

PB - IOS Press BV

T2 - dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health

Y2 - 16 May 2023 through 17 May 2023

ER -

Weishäupl K, Schuster M, Mayrl C, Humberger D, Hoffstädter A, Fischer T et al. Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. in Pfeifer B, Schreier G, Baumgartner M, Hayn D, Hrsg., dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. Band 301. IOS Press BV. 2023. S. 248 - 253. ( Studies in Health Technology and Informatics). doi: 10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Abstract

Publikationsreihe

Konferenz

Research Field

Zugriff auf Dokument

Andere Dateien und Links

Fingerprint

Diese Publikation zitieren