Lessons Learned for Identifying and Annotating Permissions in Clinical Consent Forms

Lessons Learned for Identifying and Annotating Permissions in Clinical Consent Forms
Elizabeth E. Umberfield, Yun Jiang, Susan H. Fenton, Cooper Stansbury, Kathleen Ford, Kaycee Crist, Sharon L. R. Kardia, Andrea K. Thomer, Marcelline R. Harris
Applied Clinical Information, 23 June 2021; 12(3) pp 429-435
The lack of machine-interpretable representations of consent permissions precludes development of tools that act upon permissions across information ecosystems, at scale.
To report the process, results, and lessons learned while annotating permissions in clinical consent forms.
We conducted a retrospective analysis of clinical consent forms. We developed an annotation scheme following the MAMA (Model-Annotate-Model-Annotate) cycle and evaluated interannotator agreement (IAA) using observed agreement (A o), weighted kappa (κw ), and Krippendorff’s α.
The final dataset included 6,399 sentences from 134 clinical consent forms. Complete agreement was achieved for 5,871 sentences, including 211 positively identified and 5,660 negatively identified as permission-sentences across all three annotators (A o = 0.944, Krippendorff’s α = 0.599). These values reflect moderate to substantial IAA. Although permission-sentences contain a set of common words and structure, disagreements between annotators are largely explained by lexical variability and ambiguity in sentence meaning.
Our findings point to the complexity of identifying permission-sentences within the clinical consent forms. We present our results in light of lessons learned, which may serve as a launching point for developing tools for automated permission extraction.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s