Multimodal used to be a specialisation. A researcher worked on vision or on language or on speech, and the people combining them were doing something slightly niche at the intersection. That framing is dead. Every frontier model released in the last two years handles images and text together, most handle audio, several handle video, and the question of how information from different modalities gets represented, aligned and fused is no longer a side topic.
It is the centre. CMU 11-777, taught by Louis-Philippe Morency with Paul Liang, is the course that took the subject seriously before that shift and is now positioned exactly where the field arrived. The intellectual contribution of the course is its taxonomy. Rather than organising the material as a tour of architectures, it identifies six core challenges that any multimodal system must address.
Representation covers how to encode information from heterogeneous sources in a shared or coordinated space. Alignment covers identifying correspondences between elements of different modalities, which words correspond to which region of an image, which gesture accompanies which phrase. Reasoning covers combining evidence across modalities to reach conclusions neither supports alone. Generation covers producing one modality conditioned on another.
Transference covers transferring knowledge between modalities, including cases where one is data-poor. Quantification covers understanding and measuring what multimodal models are actually doing, including how much each modality contributes. That framework is the thing I would take away even if I never watched a lecture. It converts a confusing literature into a structured map, and a paper you read afterward slots into it immediately.
Most students of this area have an implicit sense that fusion and alignment are different problems without being able to say precisely how. The taxonomy makes it precise, and precision is what lets you ask good questions about a new architecture. The technical content covers multimodal transformers, tensor fusion approaches, neuro-symbolic models, mutual information methods and multimodal graph networks, with applications in affect recognition and vision-language navigation among others. Morency's own research background in affective computing and multimodal human communication shows in the choice of applications, and the emphasis on human communicative signals, prosody, gesture, facial expression, gives the course a grounding that a purely vision-language treatment would lack.
The logistics are unusually good for a public course. It runs every semester, Spring and Fall, with offerings documented back to 2020 and continuing through Spring 2026. Lectures are recorded and published to the MultiComp Lab YouTube channel, and course materials are maintained in a public GitHub repository. Running twice a year means the reading list actually keeps up with a field where a six month old survey is out of date, and the public repository means you can compare what was taught in 2022 against what is taught now and see the field move, which is a genuinely interesting exercise in itself.
What an outside learner cannot replicate is the research project, and I want to be honest that this is where a large part of the course's value sits. It is a semester-long team project producing something close to a workshop paper, with milestone feedback from staff who review papers in this area professionally. Watching lectures gives you the map. Doing the project is what turns the map into research skill.
Someone working alone gets perhaps two-thirds of what an enrolled student gets, which is still a lot, and the honest mitigation is to pick a project anyway and hold yourself to the milestones. The prerequisite bar is real. This expects you to already understand transformers, attention, standard training practice and enough of the vision and NLP literature to read papers in both. It is a second or third machine learning course, not a first.
The reading load is heavy and much of the learning happens in papers rather than in lecture, which suits some learners and defeats others. Four point four. Rigorous, current, structured around a taxonomy that is the most useful thing anyone has published for organising this area, run often enough to stay honest, and covering what has quietly become the default shape of a modern model rather than a specialisation at the edge of one.