{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,10]],"date-time":"2026-05-10T09:27:27Z","timestamp":1778405247534,"version":"3.51.4"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T00:00:00Z","timestamp":1724889600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPTs) and Llama present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to explore the performance of LLMs in extracting acupoint-related location relations and assess the impact of fine-tuning on GPT\u2019s performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations (\u201cdirection_of\u201d, \u201cdistance_of\u201d, \u201cpart_of\u201d, \u201cnear_acupoint\u201d, and \u201clocated_near\u201d) (n\u2009=\u20093174) between acupoints were annotated. Four models were compared: pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, as well as pretrained Llama 3. Performance metrics included micro-average exact match precision, recall, and F1 scores.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The superior performance of the fine-tuned GPT-3.5 model, as shown by its F1 scores, underscores the importance of domain-specific fine-tuning in enhancing relation extraction capabilities for acupuncture-related tasks. In light of the findings from this study, it offers valuable insights into leveraging LLMs for developing clinical decision support and creating educational modules in acupuncture.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study underscores the effectiveness of LLMs like GPT and Llama in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae233","type":"journal-article","created":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T22:11:00Z","timestamp":1724969460000},"page":"2622-2631","source":"Crossref","is-referenced-by-count":34,"title":["Relation extraction using large language models: a case study on acupuncture point locations"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8784-1745","authenticated-orcid":false,"given":"Yiming","family":"Li","sequence":"first","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Xueqing","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Jianfu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"given":"Xu","family":"Zuo","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston , Houston, TX 77030,","place":["United States"]}]},{"given":"Suyuan","family":"Peng","sequence":"additional","affiliation":[{"name":"Institute of Information on Traditional Chinese Medicine, China Academy of Chinese Medical Sciences , Beijing 100010,","place":["China"]}]},{"given":"Donghong","family":"Pei","sequence":"additional","affiliation":[{"name":"The University of Texas MD Anderson Cancer Center , Houston, TX 77030,","place":["United States"]}]},{"given":"Cui","family":"Tao","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Informatics, Mayo Clinic , Jacksonville, FL 32224,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5274-4672","authenticated-orcid":false,"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]},{"given":"Na","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, School of Medicine, Yale University , New Haven, CT 06510,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,8,29]]},"reference":[{"issue":"5","key":"2024102107540519100_ocae233-B1","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1016\/S2095-4964(16)60269-8","article-title":"Puncturing the myths of acupuncture","volume":"14","author":"Mallory","year":"2016","journal-title":"J Integr Med"},{"key":"2024102107540519100_ocae233-B2","doi-asserted-by":"publisher","first-page":"868079","DOI":"10.3389\/fmed.2022.868079","article-title":"Efficacy of acupuncture and moxibustion in alopecia: a narrative review","volume":"9","author":"Li","year":"2022","journal-title":"Front Med (Lausanne)"},{"issue":"2","key":"2024102107540519100_ocae233-B3","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1016\/s0095-4543(01)00007-0","article-title":"Acupuncture","volume":"29","author":"Nasir","year":"2002","journal-title":"Prim Care"},{"issue":"4","key":"2024102107540519100_ocae233-B4","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1007\/s11655-014-1776-0","article-title":"The status of acupuncture and oriental medicine in the United States","volume":"20","author":"Stone","year":"2014","journal-title":"Chin J Integr Med"},{"key":"2024102107540519100_ocae233-B5","volume-title":"WHO Global Report on Traditional and Complementary Medicine 2019","author":"Organization WH","year":"2019"},{"issue":"12","key":"2024102107540519100_ocae233-B6","doi-asserted-by":"publisher","first-page":"1333","DOI":"10.13703\/j.0255-2930.2017.12.021","article-title":"Comments on the definition of \u201cacupuncture science\u201d","volume":"37","author":"Wang","year":"2017","journal-title":"Zhongguo Zhen Jiu"},{"issue":"4","key":"2024102107540519100_ocae233-B7","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1007\/s11655-017-2791-3","article-title":"Understanding Qi running in the Meridians as interstitial fluid flowing via interstitial space of low hydraulic resistance","volume":"24","author":"Zhang","year":"2018","journal-title":"Chin J Integr Med"},{"issue":"2","key":"2024102107540519100_ocae233-B8","first-page":"89","article-title":"Acupuncture for pain","volume":"100","author":"Kelly","year":"2019","journal-title":"Am Fam Physician"},{"issue":"9480","key":"2024102107540519100_ocae233-B9","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1016\/S0140-6736(05)66871-7","article-title":"Acupuncture in patients with osteoarthritis of the knee: a randomised trial","volume":"366","author":"Witt","year":"2005","journal-title":"Lancet"},{"issue":"4","key":"2024102107540519100_ocae233-B10","first-page":"587","article-title":"Effect of hand-ear acupuncture on chronic low-back pain: a randomized controlled trial","volume":"39","author":"Luo","year":"2019","journal-title":"J Tradit Chin Med"},{"issue":"13","key":"2024102107540519100_ocae233-B11","doi-asserted-by":"publisher","first-page":"2084","DOI":"10.1002\/cncr.34766","article-title":"Effect of acupuncture versus usual care on sleep quality in cancer survivors with chronic pain: secondary analysis of a randomized clinical trial","volume":"129","author":"Yang","year":"2023","journal-title":"Cancer"},{"issue":"1","key":"2024102107540519100_ocae233-B12","first-page":"103","article-title":"Effect of acupuncture therapies combined with usual medical care on knee osteoarthritis","volume":"39","author":"Zhang","year":"2019","journal-title":"J Tradit Chin Med"},{"issue":"4","key":"2024102107540519100_ocae233-B13","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1080\/00207144.2018.1494444","article-title":"A comparison of the efficacy of acupuncture and hypnotherapy in patients with migraine","volume":"66","author":"Tastan","year":"2018","journal-title":"Int J Clin Exp Hypn"},{"issue":"4","key":"2024102107540519100_ocae233-B14","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1016\/j.cnur.2020.07.001","article-title":"Efficacy of acupuncture\/acupressure in the prevention and treatment of nausea and vomiting across multiple patient populations: implications for practice","volume":"55","author":"Morehead","year":"2020","journal-title":"Nurs Clin North Am"},{"issue":"1","key":"2024102107540519100_ocae233-B15","first-page":"136","article-title":"Chronic respiratory conditions and acupuncture therapy","volume":"4","author":"Schwartz","year":"1992","journal-title":"Probl Vet Med"},{"issue":"8","key":"2024102107540519100_ocae233-B16","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1111\/ner.12724","article-title":"Applications of acupuncture therapy in modulating plasticity of central nervous system","volume":"21","author":"Xiao","year":"2018","journal-title":"Neuromodulation"},{"issue":"1","key":"2024102107540519100_ocae233-B17","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1089\/acm.1999.5.27","article-title":"Acupuncture for gastrointestinal and hepatobiliary disorders","volume":"5","author":"Diehl","year":"1999","journal-title":"J Altern Complement Med"},{"key":"2024102107540519100_ocae233-B18","doi-asserted-by":"publisher","first-page":"1147718","DOI":"10.3389\/fimmu.2023.1147718","article-title":"The immunomodulatory mechanisms for acupuncture practice","volume":"14","author":"Wang","year":"2023","journal-title":"Front Immunol"},{"key":"2024102107540519100_ocae233-B19","doi-asserted-by":"publisher","first-page":"102602","DOI":"10.1016\/j.conb.2022.102602","article-title":"Somatotopic organization of autonomic reflexes by acupuncture","volume":"76","author":"Ma","year":"2022","journal-title":"Curr Opin Neurobiol"},{"issue":"2","key":"2024102107540519100_ocae233-B20","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/S2005-2901(10)60014-3","article-title":"Defining Meridians: a modern basis of understanding","volume":"3","author":"Longhurst","year":"2010","journal-title":"J Acupunct Meridian Stud"},{"issue":"10","key":"2024102107540519100_ocae233-B21","first-page":"979","article-title":"The two-step location method of acupoint in Internal Canon of Medicine and its clinical application","volume":"34","author":"Xie","year":"2014","journal-title":"Zhongguo Zhen Jiu"},{"issue":"6","key":"2024102107540519100_ocae233-B22","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1016\/j.jams.2020.11.003","article-title":"Locating specific acupoints large intestine 4 (LI4) and large intestine 6 (LI6) in cadavers using anthropometric and cun measurement systems","volume":"13","author":"Casey","year":"2020","journal-title":"J Acupunct Meridian Stud"},{"issue":"2","key":"2024102107540519100_ocae233-B23","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1016\/j.jams.2018.10.009","article-title":"Accuracy and precision in acupuncture point location: a critical systematic review","volume":"12","author":"Godson","year":"2019","journal-title":"J Acupunct Meridian Stud"},{"issue":"2","key":"2024102107540519100_ocae233-B24","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1007\/s41666-024-00162-9","article-title":"Prompt tuning in biomedical relation extraction","volume":"8","author":"He","year":"2024","journal-title":"J Healthc Inform Res"},{"key":"2024102107540519100_ocae233-B25","doi-asserted-by":"publisher","first-page":"103968","DOI":"10.1016\/j.jbi.2021.103968","article-title":"An attentive joint model with transformer-based weighted graph convolutional network for extracting adverse drug event relation","volume":"125","author":"El-Allaly","year":"2022","journal-title":"J Biomed Inform"},{"key":"2024102107540519100_ocae233-B26","doi-asserted-by":"publisher","first-page":"104621","DOI":"10.1016\/j.jbi.2024.104621","article-title":"Artificial intelligence-powered pharmacovigilance: a review of machine and deep learning in clinical text-based adverse drug event detection for benchmark datasets","volume":"152","author":"Li","year":"2024","journal-title":"J Biomed Inform"},{"key":"2024102107540519100_ocae233-B27","first-page":"745","author":"Han","year":"2020"},{"key":"2024102107540519100_ocae233-B28","doi-asserted-by":"crossref","first-page":"151","DOI":"10.18653\/v1\/2020.bea-1.15","volume-title":"Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Mayfield","year":"2020"},{"key":"2024102107540519100_ocae233-B29","doi-asserted-by":"publisher","first-page":"1199350","DOI":"10.3389\/frai.2023.1199350","article-title":"Human-like problem-solving abilities in large language models using ChatGPT","volume":"6","author":"Orr\u00f9","year":"2023","journal-title":"Front Artif Intell"},{"issue":"1","key":"2024102107540519100_ocae233-B30","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1186\/s13326-024-00318-x","article-title":"Mapping vaccine names in clinical trials to vaccine ontology using cascaded fine-tuned domain-specific language models","volume":"15","author":"Li","year":"2024","journal-title":"J Biomed Semantics"},{"key":"2024102107540519100_ocae233-B31","first-page":"1","author":"Pokale","year":"2023"},{"key":"2024102107540519100_ocae233-B32","doi-asserted-by":"publisher","first-page":"ocae129","DOI":"10.1093\/jamia\/ocae129","article-title":"RefAI: a GPT-powered retrieval-augmented generative tool for biomedical literature recommendation and summarization","author":"Li","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2024102107540519100_ocae233-B33","first-page":"179","author":"Gillioz","year":"2020"},{"issue":"1","key":"2024102107540519100_ocae233-B34","doi-asserted-by":"publisher","first-page":"18","DOI":"10.18231\/j.jmra.2023.004","article-title":"Exploring the capabilities and limitations of GPT and Chat GPT in natural language processing","volume":"10","author":"Thakkar","year":"2023","journal-title":"JMRA"},{"key":"2024102107540519100_ocae233-B35","doi-asserted-by":"publisher","author":"Hu","DOI":"10.48550\/arXiv.2303.16416"},{"issue":"3","key":"2024102107540519100_ocae233-B36","doi-asserted-by":"publisher","first-page":"e0300919","DOI":"10.1371\/journal.pone.0300919","article-title":"AE-GPT: using large language models to extract adverse events from surveillance reports-a use case with influenza vaccine adverse events","volume":"19","author":"Li","year":"2024","journal-title":"PLoS One"},{"key":"2024102107540519100_ocae233-B37","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1109\/ICHI57859.2023.00053","author":"Li","year":"2023"},{"issue":"2","key":"2024102107540519100_ocae233-B38","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1093\/ecam\/nep006","article-title":"WHO Standard acupuncture point locations","volume":"7","author":"Lim","year":"2010","journal-title":"Evid Based Complement Alternat Med"},{"issue":"3","key":"2024102107540519100_ocae233-B39","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2024102107540519100_ocae233-B40","doi-asserted-by":"publisher","first-page":"100048","DOI":"10.1016\/j.nlp.2023.100048","article-title":"A survey of GPT-3 family large language models including ChatGPT and GPT-4","volume":"6","author":"Kalyan","year":"2024","journal-title":"Nat Lang Process J"},{"key":"2024102107540519100_ocae233-B41","author":"Davier"},{"key":"2024102107540519100_ocae233-B42","author":"Wang"},{"issue":"10","key":"2024102107540519100_ocae233-B43","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.1038\/s41433-024-02972-y","article-title":"A side-by-side evaluation of Llama 2 by meta with ChatGPT and its application in ophthalmology","volume":"38","author":"Masalkhi","year":"2024","journal-title":"Eye (Lond)"},{"key":"2024102107540519100_ocae233-B44","author":"Meta Llama 3","year":"2024"},{"key":"2024102107540519100_ocae233-B45","author":"Aird","year":"2005"},{"issue":"2","key":"2024102107540519100_ocae233-B46","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1111\/j.1469-7580.2009.1093_1.x","article-title":"Terminologia anatomica: international anatomical terminology and terminologia histologica: international terms for human cytology and histology","volume":"215","author":"Allen","year":"2009","journal-title":"J Anat"},{"issue":"1","key":"2024102107540519100_ocae233-B47","doi-asserted-by":"publisher","first-page":"R5","DOI":"10.1186\/gb-2012-13-1-r5","article-title":"Uberon, an integrative multi-species anatomy ontology","volume":"13","author":"Mungall","year":"2012","journal-title":"Genome Biol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/11\/2622\/59813632\/ocae233.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/11\/2622\/59813632\/ocae233.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T07:54:20Z","timestamp":1729497260000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/11\/2622\/7745292"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,29]]},"references-count":47,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,8,29]]},"published-print":{"date-parts":[[2024,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae233","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,11]]},"published":{"date-parts":[[2024,8,29]]}}}