A new study published on ArXiv (cs.AI) investigates how the composition of training data influences the capabilities of medical large language models (LLMs). Researchers conducted token-matched experiments, varying the ratio between didactic data (such as textbooks) and clinical data (such as patient records).
Asymmetric Transfer of Knowledge
The research uncovered a significant asymmetry in how different data types affect model performance. While didactic data primarily improves knowledge-intensive tasks, clinical data proved to be more versatile. Specifically, clinical data enhances clinic-oriented tasks while remaining competitive on tasks requiring pure knowledge recall.
The 'Knowing-Doing Gap'
One of the most compelling findings from the error analysis is the so-called "knowing-doing gap." The researchers observed that improvements in knowledge recall do not reliably generalize to clinical reasoning. This suggests that possessing medical information does not guarantee the ability to apply it within real-world clinical scenarios.
Data Curation Strategies
According to the study, data curation for medical LLMs should be application-driven:
- Modest amounts of clinical data yield most of the gains on EHR-grounded tasks.
- Higher proportions of clinical data are preferred for reasoning-intensive use cases.
- The optimal mixture ratio varies depending on the specific demands of the downstream task.
