[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"trial:NCT07199231":3,"trial-entities:NCT07199231":138,"trial-summary:NCT07199231":142},{"id":4,"nct_id":4,"org_study_id":5,"brief_title":6,"official_title":7,"overall_status":8,"completion_date":9,"status_verified_date":10,"last_update_date":11,"start_date":12,"sponsor_name":13,"lead_sponsor_class":14,"has_dmc":15,"brief_summary":16,"detailed_description":17,"conditions":18,"keywords":22,"study_type":23,"primary_purpose":24,"phases":25,"enrollment_info":26,"interventions":29,"primary_outcomes":36,"secondary_outcomes":47,"sex":84,"minimum_age":24,"maximum_age":24,"healthy_volunteers":85,"eligibility_criteria":86,"std_ages":93,"locations":97,"central_contacts":106,"overall_officials":107,"references":111,"see_also_links":131},"NCT07199231","CHA-IRB-25-26-444","OpenEvidence Safety and Comparative Efficacy of Four LLM's in Clinical Practice","A Comparative Performance Evaluation of Four Publicly Available Large Language Models Against Gold Standard Medical References","ENROLLING_BY_INVITATION","2026-09-30","2026-05","2026-05-22","2025-10-01","Cambridge Health Alliance","OTHER",false,"OpenEvidence is an online tool that aggregates and synthesizes data from peer-reviewed medical studies, then producing a response to a user's questions using generative AI. While it is in use by a number of clinicians (including residents) today, there is little to no published data on whether the tool's outputs are accurate and whether this information appropriately informs clinical decision making. Similarly, a number of clinicians are turning to other large language models (LLM's) to assist in decision making when providing clinical care. While there have been a number of studies published on the accuracy of these LLM's responses to medical boards questions or clinical vignettes, there have been few studies to date examining their performance in a real world clinical setting, and even fewer comparing this performance.\n\nIn this study, investigators have two goals:\n\n1. To determine whether the use of the AI tool \"OpenEvidence\" leads to clinically appropriate decisions when utilized by family medicine, internal medicine, and psychiatry residents in the course of clinical practice.\n2. To determine how the output of the OpenEvidence tool compares with three other commonly-used, publicly-available large language models (OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini) in answering common questions that residents have in the course of clinical practice.\n\nTo accomplish study goal #1, investigators have enlisted residents in the above specialties to use the OpenEvidence tool in the course of clinical practice. In order to mitigate any safety risks, the residents will also use a typical reference tool for their question, which is referred to as the \"Gold Standard\" tool. These tools include PubMed and UpToDate. The residents will:\n\n1. State their clinical question.\n2. Query OpenEvidence, capturing their prompt and the OpenEvidence output for data analysis. All residents will undergo training in prompt engineering at the start of the study.\n3. State their clinical conclusion based on the OpenEvidence data.\n4. Query the Gold Standard Resource.\n5. State their final clinical conclusion.\n6. Answer a question on whether their clinical conclusion was modified by the Gold Standard reference.\n7. Answer a question on whether they had any clinical safety concerns on the output from OpenEvidence.\n\nAttending physician Subject Matter Experts (SMEs) matched by specialty with at least 5 years of post-training clinical experience will then evaluate the residents' responses. 5 years was chosen based the book \"Outliers\" by Malcolm Gladwell, in which he asserts that 10,000 hours of focused practice is needed to achieve expertise in a field.\n\nSMEs will be asked to evaluate the residents' initial clinical questions and their conclusions based only on OpenEvidence. They will be asked to rate the clinical appropriateness of those conclusions on a scale of 1-10. For questions where the SME's rate the clinical appropriateness of the residents' conclusions poorly (\\\u003C 5\u002F10), they will be asked to review the OpenEvidence output and answer an additional question as to whether the output was incorrect or the resident misinterpreted the output from the tool.\n\nTo accomplish goal #2, the initial prompt entered by the residents into OpenEvidence will be copied by the research team into ChatGPT, Gemini, and Claude. The outputs from each tool (including OpenEvidence) will be surfaced to SMEs, who will be asked to rate each output based on accuracy, completeness, and bias. Likert scales will be used for these ratings. SMEs will also be asked an open-ended question to identify any patient safety issues from any of the outputs.","OpenEvidence is an online tool built out of the May Clinic Platform Accelerate \\[OpenEvidence\\] that aggregates and synthesizes data from peer-reviewed medical studies, then producing a response to a user's questions using generative AI. While it is in use by a number of clinicians (including residents) today, there is little to no published data on whether the tool's outputs are accurate and whether this information appropriately informs clinical decision making. Similarly, a number of clinicians are turning to other large language models (LLM's) to assist in decision making when providing clinical care.\n\nOpenEvidence is an online tool designed to aggregate and synthesize data from peer-reviewed clinical studies, subsequently generating responses to user inquiries through the application of generative AI. Although increasingly utilized by both seasoned clinicians and trainees, there is a notable absence of published data regarding the accuracy of the tool's outputs, their safety and efficacy in appropriately informing clinical decision-making. Concurrently, a growing number of clinicians are leveraging other publicly-available large language models (LLMs) to support decision-making in clinical care. While a number of studies have examined the accuracy of LLM responses to medical board questions or clinical vignettes, there is limited research on their performance in real-world clinical settings, and even fewer studies offer comparative analyses of this performance.\n\nIn a review of the literature, one article shows LLM's may be better at detecting anxiety than practitioners, but this was based on clinical vignettes. \\[Levkovich et al.\\] Another looked at diagnostic sensitivity of LLM's using patient-reported outcome measures in a structured questionnaire. \\[Pagano et al.\\] An additional study comparing LLM's for oncology also uses fictional vignettes. \\[Benary et al.\\] A randomized control trial using clinical vignettes did not show any clinical improvement for providers who had access to LLM's. \\[Goh et al.\\] One case study explored integration of ChatGPT 3.5 into daily rounds and evaluated its use qualitatively, but did not compare it with other LLM's or gold standard reference tools. \\[Skryd et al.\\] Another compared ChatGPT's responses to American College of Radiology appropriateness criteria for breast pain and breast cancer screening, but again did not compare it with other LLM's. \\[Rao et al.\\] In our review, only one study evaluated LLM's in a real world clinical setting. This was a series of papers that looked at their use for complex decision making in breast-cancer care, using a small number of actual cases and a standardized prompt template. \\[Griewing, Knitza et al.; Griewing, Gremke et al.\\] That study found issues with consistency and deterioration of accuracy (particularly with GPT 3.5), leading the authors to conclude that the clinical use of LLM's for that purpose was not yet feasible at the time of publication. Still, health systems leaders see the use of these tools rapidly accelerating in clinical practice. For this reason, investigators believe it is imperative to study their safety and the clinical appropriateness of the decisions clinicians are making as a result of their use.\n\nCambridge Health Alliance (CHA) is a public, academic safety-net health system in the Boston area, serving a diverse population of patients. CHA has a robust primary care and outpatient psychiatry footprint, and supports a large graduate medical education program through both Harvard Medical School and Tufts University School of Medicine. Investigators chose residents as our primary study participants as many trainees are already using OpenEvidence, and found them more incentivized to participate in the study if given access to the tool at CHA (where it is otherwise blacklisted from network services and prohibited by policy until results of this study can be determined).\n\nStudy outcomes are as follows:\n\nDetermine whether the use of OpenEvidence leads to clinically appropriate decisions by residents in the course of clinical practice in a community health setting.\n\nDetermine how the output of OpenEvidence compares with three other commonly-used, publicly-available large language models (OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini) in accuracy, completeness, and bias when addressing clinical questions residents have in the course of clinical practice in a community health setting.\n\nMethods:\n\nData collection is planned to take place over a 6-month period in order to minimize vendor version upgrades during the study period. Residents are grouped by specialty into \"medicine\" (internal medicine\u002Ffamily medicine) and \"psychiatry\" (adult\u002Fchild psychiatry). In order to simplify matching to appropriate specialty subject matter experts, medicine residents are asked to use OpenEvidence only for adult primary care cases (excluding OB\u002FGYN-related issues). Psychiatry residents are asked to use OpenEvidence only for adult psychiatry cases.\n\nBefore being accepted as participants, trainees were all asked to agree to the following:\n\n* Cross check any OpenEvidence query against a Gold Standard tool, defined in the study protocol to include PubMed, UpToDate, Dynamed, a clinical specialty society guideline, or other similar clinical reference source that must be documented in the study form.\n* Not enter any personal health information (PHI) into OpenEvidence (as defined below).\n* Attempt to use OpenEvidence at least 3 times per week, if appropriate to the clinical care of the patient.\n* Document 100% of their OpenEvidence queries into the study research form to avoid selection bias.\n\nAll residents will be given brief training in prompt engineering for healthcare before data collection begins. Standardized prompts will not be used, as one of the subgoals of the study is to understand what types of queries residents submit to OpenEvidence in a real world setting.\n\nAll residents will be educated on the definition of PHI, as follows:\n\nQueries should not include any of PHI, as defined by the Safe Harbor identifiers \\[HHS\\]; queries can include patient age in years (days\u002Fweeks\u002Fmonths for pediatrics), and legal sex; for patients age 89 or older, the user must instead use the term \"over age 89\" to comply with Safe Harbor standards.\n\nQueries should not include patients suspected of having extremely rare conditions as defined by the National Organization for Rare Disorders, as these are also prone to reidentification \\[NORD\\]. If a rare condition is not initially suspected but becomes suspected through the research process of using the AI tool, the user will be asked to stop their query at that point.\n\nData collection will involve the use of a HIPAA-compliant Google Form within CHA's enterprise Google Workspace for Health cloud infrastructure. The data collection form will ask trainees do the following:\n\n1. Enter their initial clinical question.\n2. Paste their OpenEvidence prompt (numbered sequentially for iterative prompts) and the full OpenEvidence output(s) generated.\n3. Enter their clinical conclusion based on the OpenEvidence output.\n4. Enter the Gold Standard reference tool used.\n5. State their final clinical conclusion based on information from both OpenEvidence and the Gold Standard Reference tool.\n6. Answer a question on the extent to which their initial clinical conclusion was modified by the Gold Standard reference.\n7. Answer an open-ended question on whether they noticed any clinical safety issues, inaccuracies, or bias in the output from OpenEvidence.\n\nQueries will be sorted by specialty (medicine vs. psychiatry), and each query will receive a sequential study number.\n\nAttending physician Subject Matter Experts (SMEs) Board Certified in Internal Medicine, Family Medicine, or Psychiatry with at least 5 years of post-training clinical experience were recruited. Five years of post-training clinical experience was chosen based on the fact that Malcolm Gladwell, in his book, Outliers, asserts that 10,000 hours of focused practice is needed to achieve expertise in a field.\n\nSMEs will be asked to evaluate the residents' initial clinical questions and their conclusions based only on OpenEvidence. They will be asked to rate the clinical appropriateness of those conclusions on a 10-point Likert scale. SME's will also be provided with the OpenEvidence output for each query, and where the SME rates the clinical appropriateness of the residents' conclusions poorly (\\\u003C 5\u002F10), the SME will additionally be asked a follow-up question to assess whether the tool's output itself provided a clinically inappropriate response, in order to ascertain whether the trainee may have misinterpreted the tool's output. SME review will include a 2.5-5% overlap between reviewers to calculate a kappa score for interrater reliability.\n\nIn part two of the study, the research team will sort the OpenEvidence queries into themes, and choose a random sampling of queries from each specialty and theme for comparison between LLM's. The research team confirm that prompts do not include any PHI according to study protocol. They will then copy the OpenEvidence prompts entered by residents for the selected queries and paste them exactly into ChatGPT, Gemini, CoPilot and Claude.\n\nThe outputs of each of the five tools (OpenEvidence, ChatGPT, Gemini, Copilot, and Claude) will be surfaced in a Google webform. SMEs will be asked to rate each output on a Likert scale for accuracy, completeness, and bias, as well as to answer a qualitative question identifying any patient safety issues in the output.\n\nResults:\n\nPrimary outcome results will be reported as follows:\n\nClinical appropriateness of decision made by residents using OpenEvidence (mean with SD, median), by specialty\n\n* If, in cases of low clinical appropriateness, SME's identified that this was due not to the tool's output but instead due to the resident's interpretation of the tool, metrics will also be provided with these cases excluded\n* Interrater reliability (kappa value)\n\nSecondary outcome results will be reported as follows:\n\nFor each specialty and each variable (accuracy, completeness, and bias), investigators will report:\n\n* The \"win\" rate for each LLM and average margin from the second place score.\n* The effect size (using each LLM as a separate reference) using Cohen's d test.\n* Interrater reliability (kappa value)",[19,20,21],"AI (Artificial Intelligence)","Large Language Model","Generative Artificial Intelligence",[],"OBSERVATIONAL",null,[],{"count":27,"type":28},20,"ESTIMATED",[30],{"type":14,"name":31,"description":32,"armGroupLabels":33},"AI clinical reference tool","Residents will use OpenEvidence clinical reference tool in the course of routine clinical care. They must also use a Gold Standard clinical reference tool (e.g. PubMed, UpToDate) to mitigate risk.",[34,35],"Medicine Residents","Psychiatry residents",[37,41,44],{"measure":38,"description":39,"timeFrame":40},"Clinical Appropriateness: Mean with SD","Clinical appropriateness score of resident decisions based on OpenEvidence output.\n\nThis is numeric score on a 10-point Likert scale. (For Likert scales described in this and all of our outcome metrics, higher scores are better outcomes.)\n\nMean score with standard deviation will be used for primary outcome.","6 months",{"measure":42,"description":43,"timeFrame":40},"Clinical Appropriateness: Median","Clinical appropriateness score of resident decisions based on OpenEvidence output.\n\nThis is numeric score on a 10-point Likert scale.\n\nMedian clinical appropriateness scores will also be reported.",{"measure":45,"description":46,"timeFrame":40},"Clinical Appropriateness: Interrater Reliability","SME's will evaluate Clinical Appropriateness scores of resident decisions based on OpenEvidence output on a 10-point Likert scale.\n\nInterrater reliability of SME Clinical Appropriateness scores will be calculated using kappa value.",[48,51,54,57,60,63,66,69,72,75,78,81],{"measure":49,"description":50,"timeFrame":40},"Comparative Accuracy of LLM's: Win Rate","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nWhichever model wins will be given a score of \"1\". If there is a tie, each gets 0.5 or 0.33 points, depending on the division of the tie.\n\nFor each specialty (medicine and psychiatry), and each LLM, we will report the \"win rate\" with average margin from the second place LLM.\n\nFor example, for all Medicine queries, we will report the percentage of times OpenEvidence \"won\" over the other LLM's on the accuracy Likert scale (where there is SME overlap to determine the kappa value), we will average the SME's scores).\n\nAs it pertains to ACCURACY, we will report the WIN RATE as a percentage.",{"measure":52,"description":53,"timeFrame":40},"Comparative Accuracy: Margin of Win","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will calculate the win rate for each LLM on accuracy and then also report:\n\nAverage margin from 2nd place LLM for ACCURACY.\n\n* If 1st and 2nd place tie, average margin will reported as 0.",{"measure":55,"description":56,"timeFrame":40},"Comparative Accuracy: Effect Size","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will use Cohen's d test to calculate the effect size of each LLM's average ACCURACY score in comparison to each of the other three LLM's.\n\nAn effect size of 0.2 is small, 0.5 is medium, and 0.8 is considered large.",{"measure":58,"description":59,"timeFrame":40},"Comparative Accuracy of LLM's: Interrater Reliability","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nInterrater reliability of Accuracy scores will be calculated using kappa value.",{"measure":61,"description":62,"timeFrame":40},"Comparative Completeness of LLM's: Win Rate","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for COMPLETNESS on a 10-point Likert scale.\n\nWhichever model wins will be given a score of \"1\". If there is a tie, each gets 0.5 or 0.33 points, depending on the division of the tie.\n\nFor each specialty (medicine and psychiatry), and each LLM, we will report the \"win rate\" with average margin from the second place LLM.\n\nFor example, for all Medicine queries, we will report the percentage of times OpenEvidence \"won\" over the other LLM's on the accuracy Likert scale (where there is SME overlap to determine the kappa value), we will average the SME's scores).\n\nAs it pertains to COMPLETENESS, we will report the WIN RATE as a percentage.",{"measure":64,"description":65,"timeFrame":40},"Comparative Completeness: Margin of Win","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will calculate the win rate for each LLM on completeness and then also report:\n\nAverage margin from 2nd place LLM for COMPLETENESS.\n\n* If 1st and 2nd place tie, average margin will reported as 0.",{"measure":67,"description":68,"timeFrame":40},"Comparative Completeness: Effect Size","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will use Cohen's d test to calculate the effect size of each LLM's average COMPLETENESS score in comparison to each of the other three LLM's.\n\nAn effect size of 0.2 is small, 0.5 is medium, and 0.8 is considered large.",{"measure":70,"description":71,"timeFrame":40},"Comparative Completeness of LLM's: Interrater Reliability","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nInterrater reliability of Completeness scores will be calculated using kappa value.",{"measure":73,"description":74,"timeFrame":40},"Comparative Bias of LLM's: Win Rate","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for signs of bias on a 10-point Likert scale.\n\nWhichever model wins will be given a score of \"1\". If there is a tie, each gets 0.5 or 0.33 points, depending on the division of the tie.\n\nFor each specialty (medicine and psychiatry), and each LLM, we will report the \"win rate\" with average margin from the second place LLM.\n\nFor example, for all Medicine queries, we will report the percentage of times OpenEvidence \"won\" over the other LLM's on the accuracy Likert scale (where there is SME overlap to determine the kappa value), we will average the SME's scores).\n\nAs it pertains to BIAS, we will report the WIN RATE as a percentage.",{"measure":76,"description":77,"timeFrame":40},"Comparative Bias: Margin of Win","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for bias on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will calculate the win rate for each LLM on bias and then also report:\n\nAverage margin from 2nd place LLM for BIAS.\n\n* If 1st and 2nd place tie, average margin will reported as 0.",{"measure":79,"description":80,"timeFrame":40},"Comparative Bias: Effect Size","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nFor each specialty (medicine and psychiatry), we will use Cohen's d test to calculate the effect size of each LLM's average BIAS score in comparison to each of the other three LLM's.\n\nAn effect size of 0.2 is small, 0.5 is medium, and 0.8 is considered large.",{"measure":82,"description":83,"timeFrame":40},"Comparative Bias of LLM's: Interrater Reliability","Comparing outputs of OpenEvidence, ChatGPT, Gemini, and Claude, SME's will rate each tool for accuracy on a 10-point Likert scale.\n\nInterrater reliability of Bias scores will be calculated using kappa value.","ALL",true,{"inclusion":87,"exclusion":91,"raw_text":92},[88,89,90],"Active trainees PGY-1 through PGY-6 in Internal Medicine, Family Medicine, Adult Psychiatry, or Child Psychiatry at Cambridge Health Alliance","Must agree to the study protocol requirements outlined in the study description.","Residents who plan to leave CHA prior to the end of the study collection period.",[],"Inclusion Criteria:\n\n* Active trainees PGY-1 through PGY-6 in Internal Medicine, Family Medicine, Adult Psychiatry, or Child Psychiatry at Cambridge Health Alliance\n* Must agree to the study protocol requirements outlined in the study description.\n\nExclusion Criteria:\n\n* Anyone who does not meet inclusion criteria\n* Residents who plan to leave CHA prior to the end of the study collection period.",[94,95,96],"CHILD","ADULT","OLDER_ADULT",[98],{"facility":13,"city":99,"state":100,"zip":101,"country":102,"geoPoint":103},"Cambridge","Massachusetts","02193","United States",{"lat":104,"lon":105},42.3751,-71.10561,[],[108],{"name":109,"affiliation":13,"role":110},"Hannah K Galvin, MD","PRINCIPAL_INVESTIGATOR",[112,116,119,122,125,128],{"pmid":113,"type":114,"citation":115},"38717811","BACKGROUND","Skryd A, Lawrence K. ChatGPT as a Tool for Medical Education and Clinical Decision-Making on the Wards: Case Study. JMIR Form Res. 2024 May 8;8:e51346. doi: 10.2196\u002F51346.",{"pmid":117,"type":114,"citation":118},"37356806","Rao A, Kim J, Kamineni M, Pang M, Lie W, Dreyer KJ, Succi MD. Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot. J Am Coll Radiol. 2023 Oct;20(10):990-997. doi: 10.1016\u002Fj.jacr.2023.05.003. Epub 2023 Jun 21.",{"pmid":120,"type":114,"citation":121},"39850460","Pagano S, Strumolo L, Michalk K, Schiegl J, Pulido LC, Reinhard J, Maderbacher G, Renkawitz T, Schuster M. Evaluating ChatGPT, Gemini and other Large Language Models (LLMs) in orthopaedic diagnostics: A prospective clinical study. Comput Struct Biotechnol J. 2024 Dec 26;28:9-15. doi: 10.1016\u002Fj.csbj.2024.12.013. eCollection 2025.",{"pmid":123,"type":114,"citation":124},"39687523","Levkovich I, Rabin E, Brann M, Elyoseph Z. Large language models outperform general practitioners in identifying complex cases of childhood anxiety. Digit Health. 2024 Dec 15;10:20552076241294182. doi: 10.1177\u002F20552076241294182. eCollection 2024 Jan-Dec.",{"pmid":126,"type":114,"citation":127},"38806945","Griewing S, Knitza J, Boekhoff J, Hillen C, Lechner F, Wagner U, Wallwiener M, Kuhn S. Evolution of publicly available large language models for complex decision-making in breast cancer care. Arch Gynecol Obstet. 2024 Jul;310(1):537-550. doi: 10.1007\u002Fs00404-024-07565-4. Epub 2024 May 29.",{"pmid":129,"type":114,"citation":130},"37976064","Benary M, Wang XD, Schmidt M, Soll D, Hilfenhaus G, Nassir M, Sigler C, Knodler M, Keller U, Beule D, Keilholz U, Leser U, Rieke DT. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Netw Open. 2023 Nov 1;6(11):e2343689. doi: 10.1001\u002Fjamanetworkopen.2023.43689.",[132,135],{"label":133,"url":134},"National Organization of Rare Disorders (NORD)","https:\u002F\u002Frarediseases.org\u002Frare-diseases\u002F",{"label":136,"url":137},"OpenEvidence","https:\u002F\u002Fwww.openevidence.com\u002Fabout",{"nct_id":4,"conditions":139,"biomarkers":141},[140,21,20],"Artificial Intelligence",[],{"nct_id":4,"found":15,"summary":24,"prompt_version":24}]