Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Kristina Weishäupl; Mario Schuster; Christina Mayrl; Diana Humberger; Antonia Hoffstädter; Theresa Fischer; Bianca Punz-Reidlinger; Fabian Wiesmüller; Aaron Lauschensky; Dieter Hayn; Karl Kreiner; Bettina Fetz; Luca Brunelli; Gerhard Pölzl; Bernhard Erich Pfeifer; Gerald Slamanig; Sabrina Neururer; Günter Schreier

doi:10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Kristina Weishäupl (Autor, Keynote), Mario Schuster, Christina Mayrl, Diana Humberger, Antonia Hoffstädter, Theresa Fischer, Bianca Punz-Reidlinger, Fabian Wiesmüller, Aaron Lauschensky, Dieter Hayn, Karl Kreiner, Bettina Fetz, Luca Brunelli, Gerhard Pölzl, Bernhard Erich Pfeifer, Gerald Slamanig, Sabrina Neururer, Günter Schreier

Digital Health Information Systems

Research output: Chapter in Book or Conference Proceedings › Conference Proceedings without Presentation › peer-review

Abstract

Background:
The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patients
and reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past.

Objective:
This work aims to analyze the inter-annotator variability among two professional groups (healthcare and
engineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,
annotators, and their professional backgrounds.

Results:
The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84).

Conclusion:
Professional background had no significant impact on the annotation of free-text HerzMobil notes.

Original language	English
Title of host publication	dHealth 2023
Subtitle of host publication	Proceedings of the 17th Health Informatics Meets Digital Health Conference
Editors	Bernhard Pfeifer, Günter Schreier, Martin Baumgartner, Dieter Hayn
Publisher	IOS Press BV
Pages	248 - 253
Number of pages	6
Volume	301
ISBN (Electronic)	978-1-64368-387-4
ISBN (Print)	978-1-64368-386-7
DOIs	https://doi.org/10.3233/SHTI230048
Publication status	Published - 2 May 2023
Event	dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health - Apothekertrakt / Schönbrunn Palace, Vienna, Austria, Vienna, Austria Duration: 16 May 2023 → 17 May 2023 https://dhealth.at/

Publication series

Name	Studies in Health Technology and Informatics
Publisher	IOS Press
Volume	301
ISSN (Print)	0926-9630
ISSN (Electronic)	1879-8365

Conference

Conference	dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health
Abbreviated title	dHealth 2023
Country/Territory	Austria
City	Vienna
Period	16/05/23 → 17/05/23
Internet address	https://dhealth.at/

Research Field

Exploration of Digital Health

Keywords

telemedicine
heart failure
Natural Language Processing
Electronic Health Records
Austria

Access to Document

10.3233/SHTI230048Licence: CC BY-NC

Cite this

Weishäupl, K., Schuster, M., Mayrl, C., Humberger, D., Hoffstädter, A., Fischer, T., Punz-Reidlinger, B., Wiesmüller, F., Lauschensky, A., Hayn, D., Kreiner, K., Fetz, B., Brunelli, L., Pölzl, G., Pfeifer, B. E., Slamanig, G., Neururer, S., & Schreier, G. (2023). Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. In B. Pfeifer, G. Schreier, M. Baumgartner, & D. Hayn (Eds.), dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference (Vol. 301, pp. 248 - 253). ( Studies in Health Technology and Informatics; Vol. 301). IOS Press BV. https://doi.org/10.3233/SHTI230048

Weishäupl, Kristina ; Schuster, Mario ; Mayrl, Christina et al. / Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. editor / Bernhard Pfeifer ; Günter Schreier ; Martin Baumgartner ; Dieter Hayn. Vol. 301 IOS Press BV, 2023. pp. 248 - 253 ( Studies in Health Technology and Informatics).

@inbook{0cab5b1c707d46e6ae044453ef40db4d,

title = "Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes",

abstract = "Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.",

keywords = "telemedicine, heart failure, Natural Language Processing, Electronic Health Records, Austria",

author = "Kristina Weish{\"a}upl and Mario Schuster and Christina Mayrl and Diana Humberger and Antonia Hoffst{\"a}dter and Theresa Fischer and Bianca Punz-Reidlinger and Fabian Wiesm{\"u}ller and Aaron Lauschensky and Dieter Hayn and Karl Kreiner and Bettina Fetz and Luca Brunelli and Gerhard P{\"o}lzl and Pfeifer, {Bernhard Erich} and Gerald Slamanig and Sabrina Neururer and G{\"u}nter Schreier",

note = "{\textcopyright} 2023 The authors, AIT Austrian Institute of Technology and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0); dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health ; Conference date: 16-05-2023 Through 17-05-2023",

year = "2023",

month = may,

day = "2",

doi = "10.3233/SHTI230048",

language = "English",

isbn = "978-1-64368-386-7",

volume = "301",

series = " Studies in Health Technology and Informatics",

publisher = "IOS Press BV",

pages = "248 -- 253",

editor = "Pfeifer, {Bernhard } and G{\"u}nter Schreier and Martin Baumgartner and Dieter Hayn",

booktitle = "dHealth 2023",

address = "Netherlands",

url = "https://dhealth.at/",

}

Weishäupl, K, Schuster, M, Mayrl, C, Humberger, D, Hoffstädter, A, Fischer, T, Punz-Reidlinger, B, Wiesmüller, F , Lauschensky, A , Hayn, D , Kreiner, K, Fetz, B, Brunelli, L, Pölzl, G, Pfeifer, BE, Slamanig, G, Neururer, S & Schreier, G 2023, Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. in B Pfeifer, G Schreier, M Baumgartner & D Hayn (eds), dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. vol. 301, Studies in Health Technology and Informatics, vol. 301, IOS Press BV, pp. 248 - 253, dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health, Vienna, Austria, 16/05/23. https://doi.org/10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. / Weishäupl, Kristina (Autor, Keynote); Schuster, Mario; Mayrl, Christina et al.
dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. ed. / Bernhard Pfeifer; Günter Schreier; Martin Baumgartner; Dieter Hayn. Vol. 301 IOS Press BV, 2023. p. 248 - 253 ( Studies in Health Technology and Informatics; Vol. 301).

Research output: Chapter in Book or Conference Proceedings › Conference Proceedings without Presentation › peer-review

TY - CHAP

T1 - Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

AU - Schuster, Mario

AU - Mayrl, Christina

AU - Humberger, Diana

AU - Hoffstädter, Antonia

AU - Fischer, Theresa

AU - Punz-Reidlinger, Bianca

AU - Wiesmüller, Fabian

AU - Lauschensky, Aaron

AU - Hayn, Dieter

AU - Kreiner, Karl

AU - Fetz, Bettina

AU - Brunelli, Luca

AU - Pölzl, Gerhard

AU - Pfeifer, Bernhard Erich

AU - Slamanig, Gerald

AU - Neururer, Sabrina

AU - Schreier, Günter

A2 - Weishäupl, Kristina

A2 - Pfeifer, Bernhard

A2 - Schreier, Günter

A2 - Baumgartner, Martin

A2 - Hayn, Dieter

N1 - © 2023 The authors, AIT Austrian Institute of Technology and IOS Press. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0)

PY - 2023/5/2

Y1 - 2023/5/2

N2 - Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.

AB - Background: The aging population's need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patientsand reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past. Objective: This work aims to analyze the inter-annotator variability among two professional groups (healthcare andengineering) involved in this program's annotation process of free-text clinical notes using categories. Methods: A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories,annotators, and their professional backgrounds. Results: The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84). Conclusion: Professional background had no significant impact on the annotation of free-text HerzMobil notes.

KW - telemedicine

KW - heart failure

KW - Natural Language Processing

KW - Electronic Health Records

KW - Austria

UR - https://www.mendeley.com/catalogue/d0a55359-949d-3345-9320-b84eee37e916/

U2 - 10.3233/SHTI230048

DO - 10.3233/SHTI230048

M3 - Conference Proceedings without Presentation

C2 - 37172189

SN - 978-1-64368-386-7

VL - 301

T3 - Studies in Health Technology and Informatics

SP - 248

EP - 253

BT - dHealth 2023

PB - IOS Press BV

T2 - dHealth 2023 - 17th Annual Conference on Health Informatics meets Digital Health

Y2 - 16 May 2023 through 17 May 2023

ER -

Weishäupl K, Schuster M, Mayrl C, Humberger D, Hoffstädter A, Fischer T et al. Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes. In Pfeifer B, Schreier G, Baumgartner M, Hayn D, editors, dHealth 2023: Proceedings of the 17th Health Informatics Meets Digital Health Conference. Vol. 301. IOS Press BV. 2023. p. 248 - 253. ( Studies in Health Technology and Informatics). doi: 10.3233/SHTI230048

Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes

Abstract

Publication series

Conference

Research Field

Keywords

Access to Document

Other files and links

Fingerprint

Cite this