Jayashree Kalpathy-Cramer

Summary

Machine learning approaches in healthcare have traditionally focused on diagnosis or prediction using a single type of data, often an image. However, clinical decision making relies on multiple sources of information collected over time. In this course, we will discuss how multimodal AI can integrate imaging with clinical, demographic, and other data to build more complete patient-specific models. We will begin with examples from ophthalmology, radiology, and cancer screening to illustrate both the opportunities and challenges of working with multimodal and longitudinal clinical data. We will then review approaches for patient-specific modeling and prediction from images, text, and structured data, including recent foundation models, and discuss what these approaches add beyond more traditional task-specific models. We will also discuss some of the challenges that arise when models are moved outside the setting in which they were developed, including brittleness under domain shift, stochastic behavior, and limited explainability. We will consider the importance of external validation, quantifying uncertainty, preserving privacy, and federated learning. We will consider how these different components can contribute toward the development of digital twins, using geographic atrophy in age-related macular degeneration as a worked example of modeling individual disease trajectories and exploring how patient-specific forecasts could be used in precision medicine and clinical trials. Throughout, we will distinguish prediction, digital shadows, virtual patients, and digital twins, and ask what evidence is required before such models can support precision medicine.

Syllabus

• From prediction to digital twins: snapshots, trajectories, patient-specific state, feedback, and updating.
• Multimodal longitudinal clinical data: imaging, text, structured records.
• Learning patient representations: quantitative imaging biomarkers, task-specific models, and multimodal foundation models.
• Robustness beyond the development set: scanner and site shift, calibration, uncertainty, and external validation.
• When the data cannot move: privacy, federated learning, and the tradeoffs of distributed model development.
• Worked example in geographic atrophy: segmentation, disease trajectory modeling, individualized forecasting, and clinical trial design.

References

Pre-requisites

Basic familiarity with machine learning concepts will be helpful, but no clinical background is required. Clinical terminology and examples will be introduced in the session. No coding is required.

Short bio

Jayashree Kalpathy-Cramer, PhD, is the Endowed Chair in Ophthalmic Data Sciences and founding Chief of the Division of Artificial Medical Intelligence in the Department of Ophthalmology at the University of Colorado School of Medicine. She also serves as Director of Health Informatics at the Colorado Clinical and Translational Sciences Institute. Her research focuses on developing, validating, and translating AI methods for medical imaging and multimodal clinical data, with applications in ophthalmology, oncology, and radiology. Her work spans quantitative imaging biomarkers, robust and federated learning, longitudinal disease modeling, and AI-enabled precision medicine. Before entering academic medicine, she spent nearly a decade in the semiconductor industry. She has authored more than 300 peer-reviewed publications and is a co-inventor on 15 patents.