Fine-Tuning Open-Weight Large Language Models to Extract Cannabis Use Frequency and Mode from Clinical Notes

Masoud Rouhizadeh, PhD
University of Florida

Co-authors: Surya Yadavilli¹, Amie Goodin^1
1University of Florida

Background: Detecting cannabis users in clinical notes is no longer the bottleneck. Our foundational work at UF Health paired high-sensitivity retrieval with high-specificity fine-tuned large language models (LLMs) to extract cannabis use status, identifying more than three times as many users as diagnosis codes. Use status alone, however, does not capture use frequency or mode of administration, both of which are clinically meaningful. Electronic health records (EHRs) have no structured field for either, and toxicology confirms recent use without indicating frequency or route, leaving free text as the only source.

Objective: To measure the prevalence of documentation for cannabis use frequency and mode in clinical notes, and to evaluate extraction accuracy across open-weight LLMs.

Methods: From a UF Health cohort (1,000 randomly selected patients; comprising 132,500 notes; 2017–2025), 2,965 snippets with adjudicated-positive cannabis use were annotated for frequency and mode (inter-annotator agreement 96.7%). Frequency and mode were modeled as separate two-step cascades, mirroring the clinical reading of a note. The frequency cascade first detected whether frequency was documented, then classified it as daily or non-daily; the mode cascade first detected whether mode was documented, then classified it as inhaled or oral. Topical mentions were excluded (n=7). Six open-weight models from two families (Gemma 3, 270M to 27B; GPT-OSS, 20B and 120B) were fine-tuned with low-rank adaptation (LoRA) and evaluated by 10-fold cross-validation, with macro F1 reported for each step.

Results: Frequency was documented in 778 of 2,965 snippets (26.2%) and mode in 602 (20.4%); among documented snippets, 41.4% were daily versus 58.6% non-daily, and 57.5% inhaled versus 42.5% oral. Detection was high and largely unaffected by model scale, with macro F1 of 0.958 to 0.975 for frequency and 0.961 to 0.973 for mode across all six models. Mode classification was similarly stable (0.945 to 0.971), whereas frequency classification was the only step to gain from scale, ranging from 0.857 at 270M to 0.952 at 120B parameters. The smallest model (270M) reached 0.958, 0.961, and 0.950 on frequency detection, mode detection, and mode classification, within 2 points of a model 440 times its size.

Conclusion: One in four cannabis-positive snippets documents use frequency and one in five documents mode, and both can be extracted with near-perfect accuracy using open-weight LLMs. Detection and mode classification are within reach of models small enough to run on modest local hardware; only use frequency classification benefits from scale.