Learn AI Series (#138) - Multimodal AI

avatar

Learn AI Series (#138) - Multimodal AI

variant-a-10-skyblue.png

What will I learn

  • What multimodal AI actually means, and why gluing modalities together is genuinely more than the sum of the parts;
  • the three fusion architectures -- early fusion, late fusion, and cross-attention -- and when to reach for each;
  • the current landscape (GPT-4o, Gemini, Claude, LLaVA) and how these systems handle images, text and audio at a high level;
  • any-to-any models: the frontier where a system generates in any modality, not just text;
  • embodied AI: what happens when a model reaches out and touches the physical world through several senses at once;
  • how to actually BUILD multimodal applications with the models that already exist -- without training a giant yourself ;-)

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ with PyTorch installed (pip install torch) -- the code here is illustrative, so nothing needs a GPU;
  • You've been through the attention/transformer arc (#51-53), the first multimodal episode (#75, CLIP and friends), and last episode's foundation-models piece (#137). We lean on all of them today.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#138) - Multimodal AI

I left you with a loose thread at the end of last episode. We had spent the whole of #137 talking about foundation models one modality at a time -- text OR vision OR audio -- and I said the most interesting ones increasingly refuse to pick a lane. They read an image and a paragraph and a snippet of audio TOGETHER, in one shared representation, and reason across all of it at once. Today we pull on that thread properly.

Here is the thing worth sitting with for a second: you do not experience the world one sense at a time either. You read text, you see faces, you hear tone of voice, you feel textures -- and your brain fuses all of it into a single understanding without you ever noticing the seams. When someone says "the cat is on the mat" while you glance at a photo of a living room, your language system and your visual system quietly collaborate to find the cat. Neither one, on its own, is enough.

AI spent most of its life being deaf, or blind, or illiterate -- one sense at a time. A text model that only ever saw text. A vision model that only ever saw pixels. A speech model that only ever heard sound. The whole point of multimodal AI is to knock down those walls and build systems that process (and generate) across several modalities at once -- and, crucially, where the combination genuinely beats any single channel on its own. We got a first taste of this back in #75 with CLIP. Today we go deeper: the architectures, the models actually leading the field in 2026, and how YOU build on top of them ;-)

Solutions to episode #137's exercises

As promised, we open with the solutions. Last time was foundation models -- freezing a giant, few-shot prompting, and picking an adaptation method. Let's clean those up before we move on.

Exercise 1 -- Count the giant. Load bert-base-uncased, freeze the whole base, bolt on a 3-class head, and stare at the trainable-to-total ratio.

# Solution 1: freeze the giant, train only a sliver.
import torch
from transformers import AutoModel

base = AutoModel.from_pretrained("bert-base-uncased")

class ThreeClassHead(torch.nn.Module):
    def __init__(self, base, n_classes=3):
        super().__init__()
        self.base = base
        for p in self.base.parameters():   # freeze EVERY base weight
            p.requires_grad = False
        self.head = torch.nn.Linear(base.config.hidden_size, n_classes)

    def forward(self, input_ids, attention_mask):
        out = self.base(input_ids=input_ids, attention_mask=attention_mask)
        cls = out.last_hidden_state[:, 0, :]   # the [CLS] token
        return self.head(cls)

model = ThreeClassHead(base, n_classes=3)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"trainable {trainable:,} / total {total:,} = {trainable/total:.4%}")
# ~2,307 / ~109,484,547 = 0.0021%

You are training roughly two thousand parameters on top of 110 million frozen ones. That microscopic ratio IS the entire selling point of the foundation-model paradigm: the giant already encodes language, you only learn the thin mapping from that knowledge onto your three labels.

Exercise 2 -- Few-shot by hand. Take the in-context learning pattern and point it at a brand-new task: label a sentence as question, command, or statement. No training whatsoever.

# Solution 2: in-context learning for a task the model was never tuned on.
prompt = """Label each sentence as question, command, or statement.

Sentence: "What time does the shop open?"
Label: question

Sentence: "Close the door on your way out."
Label: command

Sentence: "The train leaves at nine."
Label: statement

Sentence: "{s}"
Label:"""

print(prompt.format(s="Hand me that wrench."))
# A capable modern LLM answers "command" -- purely from the three examples.

The interesting bit is what happens when you starve it of examples. With three, it is rock solid. Drop to one, and ambiguous inputs start to wobble. Drop to zero, and the model has to guess your label vocabulary entirely -- it might answer "imperative" in stead of "command", which is not wrong, just not YOUR word. Those few examples pin down not only the task but your exact taxonomy.

Exercise 3 -- The decision rule for a real case. Run an honest task through the choose_adaptation function from #137.

# Solution 3: tagging support tickets. (choose_adaptation is from episode #137.)
case = ("medium", "some", "low", "relaxed")
# task_specificity = medium : support tags are domain-ish, not exotic
# data_availability = some   : a few thousand historically tagged tickets
# budget = low, latency = relaxed
method, why = choose_adaptation(*case)
print(method, "->", why)   # -> LoRA fine-tuning: parameter-efficient, cheap, usually enough

I agree with it. A few thousand labelled tickets is precisely LoRA's sweet spot -- too many to stuff into a prompt, too few to justify a full fine-tune. The single fact that would flip the answer: if I had NO labelled tickets, only a wiki of tag definitions, then data_availability becomes documents and the rule jumps straight to RAG. Data availability is the load-bearing variable here, more than budget or specificity.

Right -- housekeeping done. On to the good stuff.

Why multimodality is more than convenience

It is tempting to file multimodal AI under "nice, now I can paste a screenshot into a chatbot". That framing sells it short. Adding a second modality does not just add features -- it changes what the model is capable of understanding at all. Three reasons why.

Grounding. A pure language model can chatter happily about "a red Ferrari" without any connection to what red, or a Ferrari, actually LOOKS like. The word "red" is, to it, a statistical neighbour of other words -- "crimson", "scarlet", "stop sign". A model that has digested millions of image-caption pairs develops a grounded understanding: "red" is tied to actual pixel patterns, not just co-occurrence. That grounding demonstrably reduces hallucination and sharpens reasoning about the physical world.

Disambiguation. "Bank" is a financial institution or the edge of a river. Text context helps, but not always enough. Hand the model an accompanying image and the ambiguity simply evaporates -- the picture carries a signal that no amount of surrounding words could supply. Modalities cover for each other's blind spots.

Richer generation. A system that jointly understands text and vision can caption a photo as "a golden retriever catching a frisbee mid-air, a blur of children watching from the background" -- not the flat "a dog in a park" a weaker system coughs up. The language understanding tells it which visual details are worth mentioning.

Fusion architectures: how modalities actually meet

The central engineering question in this whole field is deceptively small: how do you combine representations coming from different modalities? There are three fundamental answers, and most real systems blend them. Let's build each one.

Early fusion

Combine at the input. Turn EVERYTHING into tokens and pour them into a single transformer. Images become patch tokens (exactly like Vision Transformers, #54), text becomes word tokens, audio becomes spectrogram tokens. One model, one shared sequence, everything mixed from the very first layer.

import torch
import torch.nn as nn

class EarlyFusionEncoder(nn.Module):
    """Combine image patches and text tokens into a single sequence."""
    def __init__(self, text_vocab, text_dim, img_patch_dim, hidden_dim, n_heads=8, n_layers=4):
        super().__init__()
        self.text_embed = nn.Embedding(text_vocab, hidden_dim)
        self.img_proj = nn.Linear(img_patch_dim, hidden_dim)
        self.modality_embed = nn.Embedding(2, hidden_dim)  # 0=text, 1=image
        layer = nn.TransformerEncoderLayer(
            hidden_dim, n_heads, dim_feedforward=hidden_dim * 4, batch_first=True
        )
        self.transformer = nn.TransformerEncoder(layer, n_layers)

    def forward(self, text_ids, img_patches):
        # text_ids: (batch, text_len), img_patches: (batch, n_patches, patch_dim)
        text_emb = self.text_embed(text_ids)   # (batch, text_len, hidden_dim)
        img_emb = self.img_proj(img_patches)    # (batch, n_patches, hidden_dim)

        # Modality embeddings so the model knows which tokens are text vs image
        text_mod = self.modality_embed(torch.zeros(text_emb.shape[:2], dtype=torch.long,
                                                    device=text_emb.device))
        img_mod = self.modality_embed(torch.ones(img_emb.shape[:2], dtype=torch.long,
                                                  device=img_emb.device))

        # Concatenate: [text tokens | image tokens]
        combined = torch.cat([text_emb + text_mod, img_emb + img_mod], dim=1)
        return self.transformer(combined)

model = EarlyFusionEncoder(text_vocab=10000, text_dim=128, img_patch_dim=768,
                           hidden_dim=256, n_heads=8, n_layers=4)
text = torch.randint(0, 10000, (2, 20))
patches = torch.randn(2, 49, 768)  # 7x7 patches from a ViT
out = model(text, patches)
print(f"Output: {out.shape}")  # (2, 69, 256) -- 20 text + 49 image tokens

Notice the little modality embedding -- without it, the transformer cannot tell a text token from an image token, and the whole thing collapses into mush. Early fusion's superpower is that every single layer can attend across modalities: an image patch can "look at" a word from layer one onward. Its cost is bluntly physical -- the sequence length is the SUM of all your modalities, attention is quadratic, and you generally have to pre-train the unified beast from scratch. Expensive.

Late fusion

The opposite philosophy. Process each modality in its own specialised encoder, all the way through, and only marry the results at the very end.

class LateFusion(nn.Module):
    """Independent encoders per modality, combined only at the end."""
    def __init__(self, text_dim, img_dim, hidden_dim, n_classes):
        super().__init__()
        self.text_encoder = nn.Sequential(
            nn.Linear(text_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim),
        )
        self.img_encoder = nn.Sequential(
            nn.Linear(img_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim),
        )
        self.classifier = nn.Linear(hidden_dim * 2, n_classes)

    def forward(self, text_feat, img_feat):
        t = self.text_encoder(text_feat)
        i = self.img_encoder(img_feat)
        combined = torch.cat([t, i], dim=-1)
        return self.classifier(combined)

Late fusion is the pragmatist's choice. It is simple, and it lets you snap together battle-tested pre-trained encoders off the shelf -- BERT (#59) for the text, a ViT for the image -- without training anything new in the middle. The tradeoff: the modalities are strangers during encoding. Each processes its input in total ignorance of the other, so any cross-modal relationship can only be discovered in those final combination layers. For tasks where the interplay between image and text is subtle, that is a real limitation.

Cross-attention fusion

The middle path -- and the one that dominates modern multimodal models. Each modality keeps its own encoder, but they are wired together through cross-attention layers so they can talk while they encode.

class CrossAttentionFusion(nn.Module):
    """Modalities interact via cross-attention."""
    def __init__(self, dim, n_heads=8):
        super().__init__()
        self.text_self_attn = nn.MultiheadAttention(dim, n_heads, batch_first=True)
        self.img_self_attn = nn.MultiheadAttention(dim, n_heads, batch_first=True)
        self.text_cross_attn = nn.MultiheadAttention(dim, n_heads, batch_first=True)
        self.img_cross_attn = nn.MultiheadAttention(dim, n_heads, batch_first=True)
        self.text_norm1 = nn.LayerNorm(dim)
        self.text_norm2 = nn.LayerNorm(dim)
        self.img_norm1 = nn.LayerNorm(dim)
        self.img_norm2 = nn.LayerNorm(dim)

    def forward(self, text_tokens, img_tokens):
        # Self-attention within each modality
        t = self.text_norm1(text_tokens + self.text_self_attn(
            text_tokens, text_tokens, text_tokens)[0])
        i = self.img_norm1(img_tokens + self.img_self_attn(
            img_tokens, img_tokens, img_tokens)[0])

        # Cross-attention: each modality attends to the OTHER
        t = self.text_norm2(t + self.text_cross_attn(t, i, i)[0])  # text queries, image keys/values
        i = self.img_norm2(i + self.img_cross_attn(i, t, t)[0])    # image queries, text keys/values
        return t, i

This is, mechanically, the very same cross-attention we met in the transformer decoder (#53) -- just repurposed. Text tokens "look at" image tokens to work out what they refer to; image tokens "look at" text tokens to work out which context matters. You get most of early fusion's rich interaction without paying the full quadratic bill on one gigantic concatenated sequence. Having said that, it is fiddlier to implement -- four attention blocks where late fusion has none.

The multimodal landscape in 2026

Enough theory -- who is actually shipping, and roughly how? A quick, honest tour of the leading systems (architectures for the closed ones are educated inference, not gospel).

GPT-4o and GPT-4V (OpenAI). Accept text, images and audio in, produce text and audio out. The internals are not published, but the widespread reading is a variant of early fusion -- everything tokenised and shoved through one large transformer. The "o" stands for omni, which tells you the ambition.

Gemini (Google). Natively multimodal from day one -- text, images, audio AND video. Google's pitch is that training on multimodal data from the very beginning (rather than bolting vision onto a finished text model) yields stronger cross-modal understanding. Plausible, and hard to verify from the outside.

Claude (Anthropic). Takes text and images, with a particular strength in document understanding, chart reading and careful visual reasoning. The image understanding is baked into the language model rather than living in a bolted-on side module.

LLaVA and the open-source swarm. This is the one YOU can actually build. LLaVA -- Large Language and Vision Assistant -- connects a pre-trained vision encoder (CLIP's ViT) to a pre-trained language model (Llama) through a laughably simple projection layer. And it works startlingly well.

class LLaVAStyle(nn.Module):
    """Simplified LLaVA: vision encoder + projection + LLM."""
    def __init__(self, vision_dim=1024, llm_dim=4096, n_patches=576):
        super().__init__()
        # In real LLaVA: self.vision_encoder = CLIPViTModel(...)
        # In real LLaVA: self.llm = LlamaForCausalLM(...)
        # The key insight: a simple linear projection connects them
        self.vision_projection = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),
        )
        # The projected image tokens are prepended to the text tokens
        # and fed through the LLM as if they were plain text tokens

    def forward(self, image_features, text_input_ids):
        # Project image features into the LLM's embedding space
        visual_tokens = self.vision_projection(image_features)
        # Concatenate: [visual_tokens | text_embedding_tokens]
        # Feed through the LLM, which generates text conditioned on both
        return visual_tokens  # simplified

llava = LLaVAStyle()
fake_image = torch.randn(1, 576, 1024)   # 576 patch features from CLIP
tokens = llava(fake_image, text_input_ids=None)
print(f"visual tokens projected into LLM space: {tokens.shape}")  # (1, 576, 4096)

The LLaVA insight is almost embarrassing in its simplicity: you do NOT need a bespoke multimodal architecture. You need a good vision encoder, a good language model, and a learned bridge between them. The vision encoder turns an image into "visual tokens" that the language model then treats exactly as if they were words. Two short training stages -- first the projection alone, then the whole stack lightly -- and bam, you have a multimodal model. This single trick spawned dozens of open-weight descendants (Qwen-VL, InternVL, and the rest).

Any-to-any: the frontier

Almost everything deployed commercially today is "many inputs, text output" -- you feed it text and images, it answers with text. The frontier is any-to-any: models that generate in ANY modality, in any direction.

  • Text to image: Stable Diffusion, DALL-E 3, Midjourney -- we dissected the diffusion machinery in #84-85.
  • Text to audio: MusicGen, AudioLDM, Bark -- the music-generation episode (#96) covered the shape of this.
  • Text to video: Sora, Runway Gen-3, Kling -- these extend diffusion into the time dimension, generating coherent motion from a description.
  • Image to text: visual question answering, captioning, document extraction -- any decent multimodal LLM does this in its sleep.

The real prize is unified models that handle every direction at once. You ask in text, it replies with an image AND a spoken narration. You upload a video, it hands back a written summary and suggests a soundtrack. These are not science fiction -- early versions exist -- and the underlying recipe is exactly the one from last episode: tokenise everything, run a transformer over all the tokens, decode into whatever modality you asked for.

Embodied AI: models that reach into the world

The most ambitious use of multimodal AI is embodied agents -- robots and systems that act on the physical world through multiple sensors and actuators.

Picture a robot in a kitchen. It gets visual input (a camera), force feedback (touch sensors), audio (a microphone), and a spoken command: "pick up the red cup." To act well it must fuse all of it. A cup that LOOKS empty might be full -- the weight sensor and the camera disagree, and the model has to reconcile them. "Put it next to the other one" is meaningless until "it" and "the other one" are grounded in the actual visual scene.

Google's RT-2 showed something genuinely startling here: a single multimodal model, trained on BOTH internet data and robot demonstrations, can follow novel language commands in the real world. The internet knowledge supplies commonsense ("a cup is a thing you can pick up"); the robot data supplies the physical grounding ("here is what picking up a cup actually involves, joint by joint"). That still slightly blows my mind, honestly.

And this is precisely where multimodal AI shakes hands with reinforcement learning (the whole #102-116 arc). The agent perceives through many senses, takes an action, receives a reward, and learns a policy mapping multimodal observation to physical action. Perception fuses; control learns.

Building multimodal applications (the part you'll actually do)

For the overwhelming majority of us, "doing multimodal AI" means USING existing models through APIs or open weights -- not training a giant from scratch. Same lesson as #137: you are standing on somebody else's pre-training bill.

# Pattern: a multimodal pipeline built from existing models.
class MultimodalAnalyzer:
    """Analyze content across text, images, and audio."""
    def __init__(self):
        self.tasks = {
            "image_caption": self._caption_image,
            "visual_qa": self._visual_qa,
            "document_extract": self._extract_document,
        }

    def _caption_image(self, image_path):
        """Describe an image with a multimodal model."""
        # In production: call GPT-4o, Claude, or a local LLaVA
        return {"task": "caption", "image": image_path, "method": "multimodal_llm"}

    def _visual_qa(self, image_path, question):
        """Answer a question about an image."""
        # The question provides focus; the image provides grounding
        # "What colour is the car?" + image -> "The car is blue"
        return {"task": "vqa", "question": question, "image": image_path}

    def _extract_document(self, image_path):
        """Extract structured data from a document image."""
        # OCR (episode #82) + multimodal understanding
        # Receipt image -> {"store": "...", "total": "...", "items": [...]}
        return {"task": "extraction", "image": image_path}

    def analyze(self, inputs):
        results = {}
        for task_name, task_fn in self.tasks.items():
            if task_name in inputs:
                results[task_name] = task_fn(**inputs[task_name])
        return results

analyzer = MultimodalAnalyzer()
report = analyzer.analyze({
    "image_caption": {"image_path": "photo.jpg"},
    "visual_qa": {"image_path": "car.jpg", "question": "What colour is the car?"},
})
for k, v in report.items():
    print(k, "->", v)

The practical playbook, distilled:

  1. Start with API-based multimodal models. GPT-4o, Claude, Gemini handle image+text natively. Your prototype is a handful of API calls, not a research project.
  2. Reach for open-weight models when cost or privacy bites. LLaVA, Qwen-VL, InternVL run locally and cover image+text perfectly well for most needs.
  3. Compose specialists for complex pipelines. Whisper for audio-to-text (#93), CLIP for image search (#63), an LLM for the reasoning, a TTS model for the spoken reply. A relay team of specialists routinely beats one generalist trying to do everything.
  4. Evaluate on YOUR task, not a leaderboard. Multimodal benchmarks are a weak predictor of real-world behaviour. Build a small domain-specific eval set of examples that actually matter to you -- the same discipline we hammered in #73.

The limits -- read this before you ship anything

Modern multimodal systems are genuinely impressive, and they will also embarrass you in production if you trust them blindly. The honest failure list:

Spatial reasoning. "Is the cup to the LEFT or the RIGHT of the book?" gets fumbled far more often than you'd believe, despite being trivial for a toddler. Precise spatial relationships are a known soft spot.

Counting. "How many people are in this image?" is unreliable much past five or six. The models approximate; they do not truly count.

Fine-grained detail. Small text buried in an image, subtle differences between near-identical objects, details in a cluttered scene -- frequently missed, or worse, hallucinated with total confidence.

Temporal reasoning in video. Understanding what happened BEFORE versus AFTER, or catching a brief event, is still shaky. Most "video understanding" today is closer to "understanding a handful of frames" than to real reasoning over time.

Visual hallucination. The same disease we know from language models crosses straight over into vision. The model will cheerfully describe an object that is not there, read text that does not exist, or invent a detail to paper over an ambiguity. It never sounds unsure while doing it.

None of these are fundamental laws of nature -- they are being chipped away at, fast -- but they are very real TODAY. So design accordingly: fallbacks, sanity checks, and a human in the loop for anything high-stakes. Do not let a confidently-wrong caption sign off on a medical or financial decision.

Exercises

Before the next episode, get your hands properly dirty. Three tasks, climbing in difficulty:

  1. Trace the shapes. Take the EarlyFusionEncoder above and feed it a batch with 15 text tokens and 64 image patches (an 8x8 grid). Predict the output sequence length ON PAPER first, then run it and confirm. Now add a second, LATER image (another 64 patches) as a third modality -- extend the class with a third modality embedding id and make it work. What is the new sequence length, and why does early fusion's cost worry you as you keep adding modalities?

  2. Late vs cross-attention, by hand. Instantiate both LateFusion and CrossAttentionFusion on the same fake text/image tensors. Print the parameter count of each. Write two or three sentences on WHERE the extra parameters in the cross-attention version live, and what capability you are buying with them.

  3. Design a real pipeline. Pick a genuine multimodal task you'd actually like to solve -- captioning your holiday photos, extracting totals from receipts, whatever. Sketch (in code comments or a diagram) which existing models you'd chain together, in what order, and -- most important -- WHERE you'd put a human check or a fallback for the failure modes listed above. The architecture reasoning matters more than any code here.

We'll open the next episode with full solutions, as always.

Quick recap

  • Multimodal AI fuses text, images, audio and more into a single understanding that beats any one modality alone -- through grounding, disambiguation, and richer generation;
  • there are three fusion architectures: early fusion (one model over all tokens, rich but expensive), late fusion (separate encoders joined at the end, simple but shallow interaction), and cross-attention (modality-specific encoders that talk while they encode -- the modern default);
  • the LLaVA trick -- vision encoder + a small projection + an LLM -- is almost suspiciously simple and it launched the entire open-weight multimodal ecosystem;
  • 2026's frontier models (GPT-4o, Gemini, Claude) take multiple input modalities and shine at visual reasoning and document understanding;
  • any-to-any generation (text to image, video, audio and back) is the active frontier, and embodied AI wires multimodal perception to physical action, shaking hands with reinforcement learning;
  • current limits are real: weak spatial reasoning, shaky counting, fine-detail and visual hallucination, and shallow temporal understanding in video -- design with fallbacks;
  • for practitioners, the move is the same as with foundation models: start with an API, compose specialists for the hard pipelines, and evaluate on YOUR task.

And here is the thread for next time. We have spent this episode letting models SEE and HEAR the world. But there is one "modality" that is oddly special -- it is simultaneously human language and rigid machine logic, it comes with instant, brutal feedback (it either runs or it does not), and models that master it start to genuinely amplify the people who build everything else in this series. That is where we head next ;-)

Bedankt en tot de volgende keer -- now go trace some tensor shapes and feel, for yourself, how a picture becomes just another handful of tokens! De groeten! ;-)

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments