Multimodal AI
People do not experience the world one channel at a time. We read a face and a tone of voice together; we understand a photograph partly through the caption sitting under it. Multimodal AI is the attempt to build systems that do the same — models that hold vision, text and audio in one representation rather than treating each as a separate problem.
This is the lab's centre of gravity. Most of our projects are, underneath, an exercise in getting two modalities into a shared space and then generating one from the other: images into music, text into sequential images, visual stimuli into audio.
In practice
- Visual question answering. Answering questions about an image, which requires grounding language in what is actually visible.
- Image captioning. Producing description that is specific to the image rather than plausible for any image.
- Cross-modal alignment. Learning embedding spaces where related content in different modalities lands in the same neighbourhood.
- Cross-modal generation. Conditioning a decoder in one modality on an encoder in another — the shape of three of the four current projects.