Image Capture and Multimodal AI for Interactive Radiology Assistance

This project aims to investigate the development of advanced "smart" software systems capable of processing and reacting to clinician inputs in near real-time, in order to interact with radiological images such as CT and X-Ray scans. This will involve technologies such as voice- and gesture-activated assistants, interactive image analysis tools, and multimodal AI models.

Such technology will facilitate expert medical tasks such as image navigation, anatomy/pathology detection, visual question answering, and medical reporting. This has the potential to revolutionise radiology workflows, particularly in the interventional radiology domain. The project will require research into integration of diverse multimodal data sources, likely including the application of large vision and language models (e.g. [1,2]) to medical data.

Responsive adaptation to a variety of multi-step clinician inputs may require on-the-fly AI model training and fine-tuning.

[1] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y. and Dollár, P., 2023. Segment anything. arXiv preprint arXiv:2304.02643.

[2] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S. and Payne, P., 2023. Large language models encode clinical knowledge. Nature, 620(7972), pp.172-180.