Subjects: Library Science，Information Science >> Information Science submitted time 2024-08-06
Abstract： [Purpose/Significance] This study aims to extract structured item information from free-text clinical scales using ChatGPT without annotations, which efficiently advances the structuring and intellectualization of medical scale resources. [Method/Process] A framework for item information extraction that includes eight types of attributes and considers the structural differences in clinical scale measurement concepts was defined. A dataset was constructed by collecting 59 commonly used clinical psychometric assessment scale documents; zero-shot prompt templates were designed based on measurement concept levels, and experiments were conducted using he official ChatGPT-3.5 and ChatGPT-4 interfaces; the extraction performance and possible influencing factors of different versions of ChatGPT in processing different clinical scale texts were analyzed from multiple perspectives. [Result/Conclusion] The extraction performance of scale item sources was the best, with Micro-F1 and Macro-F1 scores of at least 98.90% and 97.83%, respectively. This was followed by response options, instructional guidance, and scoring rules, with item numbers and instructions showing moderate performance. Clinical explanations had the lowest performance, with Micro-F1 and Macro-F1 scores of 47.73% and 45.51%, respectively. ChatGPT-4 performed better overall, but the recall rate of some attributes was weaker than that of ChatGPT-3.5. The increase in measurement concept levels, dimensionality, number of items, and text length was found to reduce model performance. In summary, ChatGPT can efficiently assist in the structuring of medical scale resources, especially when dealing with simple scales.
Hits
YES
Hits
Downloads
Comment