How to Teach Large Multimodal Models New Skills?
Large multimodal models (LMMs) such as LLaVA and Qwen2.5-VL answer questions about images in natural language, and they are good at a remarkable range of things. But they are not good at everything. Ask one to count the people in a crowd, read an analog clock, or diagnose a pathology slide, and it will often fail. Training a new LMM from scratch costs millions of dollars and weeks of compute, so the practical route is to fine-tune the model you have. The problem is that naive fine-tuning on one narrow skill tends to erode the broad abilities the model came with, the familiar catastrophic forgetting. So: how do we teach an LMM something new without it forgetting the old?
Our ECCV 2026 paper takes this question literally. We build a controlled test bed, fine-tune three families of LMMs on a sequence of five new skills, and watch what happens to eight general benchmarks along the way. Two findings surprised us, and chasing down why led to two very simple training recipes that learn almost as well as full fine-tuning while forgetting almost nothing.
Two surprises
Our first experiments simply asked where in the model to put the gradient. An LMM has three parts: a vision encoder that turns the image into tokens, a projector that maps them into the language model’s width, and a decoder-only language model. Inside each transformer block of the language model there are, in turn, two sub-blocks: the self-attention projections (the query, key, value and output matrices, which we call SA Proj.) and the feed-forward MLP. We fine-tuned each of these in isolation, and the full model, on a sequence of five target tasks.
The first surprise is in the hero figure above. Tuning only the self-attention projections learns the target task nearly as well as tuning the whole language model, yet across the entire five-task sequence its held-out accuracy never moves. We were not aware of any other instance of strong learning without forgetting in the absence of replay, model expansion or heavy regularization.
The second surprise is the shape of the other curves. Held-out performance does not decline monotonically as more tasks are learned. It crashes after the counting task (PixmoCount) and then largely recovers after training on the next task, medical VQA, which has nothing to do with the benchmarks that were lost. With full-model tuning the held-out average falls from 76 to 63 and then climbs back to 74. Whatever was “forgotten” was evidently not gone.
The test bed
To study this carefully we needed a protocol that measures learning and forgetting on equal footing. We chose five target skills that are common in real use and that LMMs typically struggle with, each with a different answer format: fine-grained bird classification (CUB-200), counting (PixmoCount), medical VQA (PathVQA), OCR reading (TextVQA) and reading analog clocks (TimeClock). Together they contain about 108K training samples. The model is fine-tuned on these one after another, with no rehearsal of earlier tasks, and after each stage we evaluate both the target tasks and eight held-out benchmarks that cover broad vision–language ability: AI2D, ChartQA, DocVQA, InfoVQA, RealWorldQA, SeedBench, ScienceQA and MMStar.
From this we compute four numbers per method, all as percentage-point changes from the base model: target learning (how much each task improves when it is the one being trained), target forgetting (how much earlier target tasks drop by the end of the sequence), target overall (the net change on all target tasks at the end) and held-out forgetting (the change in the held-out average at the end). Main results on LLaVA-OneVision are averaged over three different task orders.
Where to tune
| Tuned component | Target learning ↑ | Target forgetting ↑ | Target overall ↑ | Held-out forgetting ↑ |
|---|---|---|---|---|
| Full model | +29.9 | −25.9 | +9.2 | −27.4 |
| Vision encoder | +9.6 | −12.7 | −0.5 | −10.8 |
| Projector | +2.3 | −0.8 | +1.7 | −1.3 |
| LLM | +31.8 | −23.5 | +13.0 | −23.3 |
| SA Proj. | +24.9 | −2.3 | +23.1 | −0.6 |
| SA Proj. (QKV only) | +14.9 | −0.5 | +14.5 | +0.2 |
| MLP | +31.1 | −19.5 | +15.5 | −15.7 |
| MLP (Gate & Up) | +30.5 | −4.2 | +27.1 | −2.1 |
LLaVA-OneVision, percentage-point change from the base model (43.9 on target tasks, 76.4 on held-out), averaged over three five-task orders.
A few patterns are clear. Learning lives in the language model: tuning the LLM gives the strongest gains, while tuning the vision encoder learns little and still damages general ability, and the projector barely moves anything. But full-LLM tuning also forgets the most. Opening up the LLM reveals the two sub-blocks behave very differently. Tuning the MLP learns about as much as the whole LLM and forgets somewhat less. Tuning the self-attention projections learns a bit less but forgets essentially nothing.
The last row is the one we ended up recommending most often. The MLP in these models computes W_down(SiLU(x W_gate) ⊙ (x W_up)). If we tune the gate and up matrices but freeze the down projection, we keep almost all of the learning of full-MLP tuning (+30.5 versus +31.1) while cutting held-out forgetting from −15.7 to −2.1, and we get the best end-of-sequence target score of any method.
Why: forgetting as output distribution shift
Why would the self-attention projections be so robust, and why would the down projection of the MLP matter so much? Mechanistic studies of transformers offer a division of labor. Attention heads mostly route: they decide where in the context to read from and mix the corresponding values. MLPs behave like key–value memories: the gate and up matrices detect input patterns, and the down projection writes features back into the residual stream that are aligned with groups of vocabulary tokens, directly shaping the next-token distribution.
That suggested a hypothesis. When you fine-tune on a narrow task whose answers all look alike (every counting answer is “There are N …”), the model has a shortcut available: instead of learning to count better, it can simply bias its output distribution toward numeric tokens. The write-back paths, the MLP down projection above all, are exactly where such a bias would be installed. If much of what we measure as forgetting is really this kind of response bias, then (a) restricting updates to routing (SA Proj.) or to feature detection (MLP gate and up) should limit it, and (b) training on a subsequent task with a different answer format should undo it, which is the recovery we saw.
To test this we designed a counting-bias probe. Take a fixed batch of ordinary captioning images, let the model generate captions with greedy decoding, and at every decoding position record the maximum probability the model assigns to any numeric token. Averaged over positions and images, this number tells us how strongly the model is being pulled toward saying numbers when it has no reason to.
The probe co-varies with forgetting remarkably well: across training steps and across methods the Spearman correlation between numeric-token likelihood and held-out drop is 0.84. Full-LLM and full-MLP tuning both learn to count and both start sprinkling numbers into their captions. SA Proj. and MLP (Gate & Up) learn to count without doing so. A layer-wise analysis in the appendix attributes most of the induced logit shift to the MLP blocks, and extending the probe across three sequential stages shows the numeric bias spiking when counting is learned and receding when the next task is learned, in lockstep with the held-out accuracy falling and recovering.
The qualitative behavior is telling. After 1K steps of counting on PixmoCount, both the MLP-tuned and the SA Proj.-tuned model answer counting questions correctly. But ask the MLP-tuned model what is in a photo and it replies “There are 2 photos in the photo.” The visual understanding is intact, since it still recognizes the objects and relations, but the output has been reframed as a counting statement. The SA Proj.-tuned model still writes a detailed paragraph.
Two recipes
This gives a clear answer to the title question. To teach an LMM a new skill without forgetting, restrict the update to components that do not directly write to the output distribution:
- SA Proj.: tune the query, key, value and output projections of every attention block, freeze everything else. Prefer this when stability on general benchmarks is paramount.
- MLP (Gate & Up): tune the gate and up projections of every MLP block, freeze the down projections. Prefer this when you want near-maximal learning with limited forgetting.
Either is a few lines of code on top of any existing training script:
for p in model.parameters():
p.requires_grad = False
# Recipe 1: self-attention projections only
sa_keys = ("self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj", "self_attn.o_proj")
for name, p in model.named_parameters():
if any(k in name for k in sa_keys):
p.requires_grad = True
# Recipe 2 (alternative): MLP gate & up only, down projection stays frozen
for name, p in model.named_parameters():
if "mlp.gate_proj" in name or "mlp.up_proj" in name:
p.requires_grad = True
No extra modules, no replay buffer, no per-stage hyperparameters, and no extra forward passes. On four H100 GPUs, SA Proj. tuning runs at 1.46 samples per second, faster than LoRA (1.27) and much faster than a mixture-of-experts variant (0.44), because there are no adapters or routers in the compute graph.
Does it hold beyond one model?
We repeated the five-task curriculum on LLaVA-NeXT (LLaMA-3 8B) and Qwen2.5-VL (7B). The numbers differ, but the picture does not.
| Tuned component | LLaVA-NeXT: target overall ↑ | LLaVA-NeXT: held-out ↑ | Qwen2.5-VL: target overall ↑ | Qwen2.5-VL: held-out ↑ |
|---|---|---|---|---|
| Full model | +15.4 | −32.0 | +13.1 | −17.5 |
| Vision + projector | −1.3 | −13.4 | +4.9 | −6.2 |
| LLM | +19.3 | −35.9 | +12.1 | −24.6 |
| SA Proj. | +21.9 | −7.7 | +14.9 | +0.6 |
| MLP | +26.6 | −16.3 | +13.9 | −10.9 |
| MLP (Gate & Up) | +20.9 | −8.7 | +17.1 | −4.6 |
Percentage-point change from each base model after the full five-task sequence.
LLaVA-NeXT is far more prone to forgetting in general, yet SA Proj. and MLP (Gate & Up) still forget less than half as much as full-MLP tuning and less than a quarter as much as full-LLM tuning. On Qwen2.5-VL, SA Proj. actually shows a little positive transfer to the held-out suite, and MLP (Gate & Up) gives the best overall target score with essentially zero target forgetting. Its absolute gains look smaller only because its base model starts much higher.
Comparison to the usual mitigations
Selective tuning is not the only way to limit forgetting, so we also benchmarked the standard pillars of continual learning, all applied to the MLP layers: Learning without Forgetting (a KL distillation loss against the previous-stage model), LoRA, a mixture-of-experts expansion, and WiSE-FT weight interpolation.
Distillation does curb forgetting reliably, which fits the output-shift story since it penalizes exactly that, but it can cost target performance. WiSE-FT preserves held-out accuracy well but needs task-dependent blending coefficients chosen per stage. LoRA learns noticeably less, and the mixture-of-experts neither learns nor retains as well as the simple recipes. SA Proj. and MLP (Gate & Up) match or beat all of them on the learning–stability trade-off while being the simplest option on the table.
Limitations
Our study covers 7B to 8B open models. We have not tested whether the same recipes hold for much larger or commercial LMMs, or when the adaptation data scales to billions of samples. We did observe a little forgetting when we kept pushing with very large long-tailed tasks such as iNaturalist classification, so “no forgetting” should be read as “very little forgetting under realistic skill-adaptation workloads,” not as a guarantee.
Read more
The paper appears at ECCV 2026 as a spotlight. The code includes the full evaluation suite (five target tasks and eight held-out benchmarks via lmms-eval), the component-wise tuning scripts, the counting-bias probe, and reference implementations of the LwF, LoRA, MoE and WiSE-FT baselines.
@inproceedings{zhu2026teach,
title = {How To Teach Large Multimodal Models New Skills?},
author = {Zhen Zhu and Yiming Gong and Yao Xiao and Yaoyao Liu and Derek Hoiem},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}
This work is supported in part by ONR award N00014-23-1-2383 and the U.S. DARPA ECOLE program (HR00112390060). GPUs were provided by NCSA Delta through ACCESS allocation CIS230397.