Jelmer M. Wolterink

Summary

Health digital twinning requires representations that connect patient observations to models of anatomy, physiology, and disease. These representations must preserve clinically relevant information while supporting learning from limited, heterogeneous data. This course examines how data representations, neural network architectures, and learning objectives jointly shape patient-specific analysis. We will discuss representations ranging from handcrafted features, voxel grids, meshes, and point clouds to neural fields and learned embeddings provided by foundation models. We will explore how symmetry can encode relevant structure, and how cross-patient representations can support downstream tasks with limited labeled data.

The second part of the course connects these foundations to multimodal integration, examining how anatomical representations, learned embeddings, and clinical variables can provide complementary patient information. Recent approaches, including MedFuncta, Merlin, and nnFoundation, will illustrate different strategies for learning reusable representations. Cardiovascular examples will connect the methodological concepts to personalized diagnosis, prognosis, and treatment planning.

Syllabus

Session 1: Data representations and operators in medical imaging

This session examines how representation choices influence patient-specific analysis. Using cardiovascular examples, we will discuss:

• Representation choices: comparing features, voxel grids, meshes, and point clouds, focusing on the information they preserve and their suitability for different tasks.
• Neural fields: a worked example introducing continuous representations of medical images or anatomy and their relationship to conventional discrete representations.
• Architectures and symmetry: how representations guide architectural choices, and how invariance and equivariance can encode relevant structure.

Session 2: Learned representations and multimodal integration

This session moves from representing individual observations to learning reusable representations across patients and connecting complementary patient information. We will discuss:

• Learning representations across patients: how learning objectives shape the information retained, using MedFuncta to illustrate the transition from individual neural fields to shared models.
• Foundation-model representations: how pretrained imaging models can support downstream tasks with limited labeled data, illustrated through a selected example.
• Multimodal integration: a case study combining imaging representations with clinical variables, discussing their complementarity and how to evaluate their added value for patient-specific prediction.

References

Pre-requisites

Participants should have a basic understanding of deep learning, including multilayer perceptrons, convolutional neural networks, and training with gradient descent. Familiarity with medical imaging is helpful but not required. Neural fields, geometric deep learning, and multimodal representation learning will be introduced within the course.

Short bio

Dr. Jelmer Wolterink is an associate professor in the Department of Applied Mathematics and the Technical Medical Center at the University of Twente, the Netherlands. His research focuses on deep learning for diagnosis, prognosis, and treatment planning in cardiovascular imaging, including shape analysis, surrogate modeling for biomechanics, and integration of multimodal clinical data. He has published over 100 peer-reviewed articles and collaborates extensively with clinical partners in radiology, vascular surgery, and cardiology.