Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation.
Key Findings
1
Standard PEFT outperforms image-to-video PEFT
Standard PEFT methods (e.g. LoRA, AdaptFormer) outperform image-to-video PEFT approaches
when paired with an attentive probe. This gap widens with a video-pretrained backbone.
2
Attention for motion, MLP for spatial
On video backbones, attention-based adaptation excels on motion-heavy tasks, while
MLP-based adaptation is best for spatial prediction.
3
A simple attentive probe is sufficient
Although probe performance differs substantially per task, this variability largely
disappears once combined with PEFT, suggesting that a simple attentive probe is sufficient
and removes the need for elaborate task-specific decoders.
4
Limited temporal modeling in image-to-video PEFT
Temporal modeling is driven by the backbone, which also dominates throughput.
Current temporal PEFT methods can only partially compensate for reduced temporal context in the backbone.
Distributing Temporal Context
Video adaptation requires reasoning over multiple frames, making a central design choice not just
whether to model temporal information, but where this temporal reasoning should
occur. Existing approaches make this choice implicitly by assigning temporal modeling to a single
component, leaving a broader design space underexplored in which multiple components may benefit
from temporal reasoning at different temporal scales. We treat temporal context as a flexible resource that can be distributed across components with
different temporal extents, enabling a systematic study of where performance gains arise.
Distributing Temporal Context. Existing works place temporal modeling in a single
component: either the PEFT modules (a), where a frame-wise backbone is combined with temporal
adaptation, or in the backbone (b), where temporally-aware representations are adapted by static
PEFT and probes. We instead allow temporal modeling across backbone, PEFT and probe simultaneously
(c) and study how temporal context should be distributed across components (d).
Results
We benchmark adaptation methods on naturally low-resource video tasks spanning appearance
(CAER, NurViD), motion (IndustReal, MammAlps) and spatial understanding (ScanNet, VSPW). The
datasets span diverse annotation granularities, viewpoints and domains, and study how temporal
context affects adaptation across the backbone, probe and PEFT modules.
Appearance
Motion
Spatial
CAER top-1 ↑
NurViD top-1 ↑
IndustReal top-1 ↑
MammAlps top-1 ↑
ScanNet AbsRel ↓
VSPW mIoU ↑
SigLIP2
Attentive Probe
52.1
79.0
47.4
66.9
0.237
33.8
LoRA
63.0
87.0
67.7
73.4
0.136
53.5
AdaptFormer
61.2
87.1
64.2
72.4
0.122
54.9
ST-Adapter
62.5
87.5
66.9
72.4
0.131
55.7
DINOv3
Attentive Probe
58.9
85.2
58.1
71.5
0.120
56.4
LoRA
62.8
87.3
69.8
72.6
0.111
60.6
AdaptFormer
61.8
87.1
67.5
72.6
0.092
61.2
ST-Adapter
62.4
86.9
69.2
75.6
0.106
61.1
VJEPA2
Attentive Probe
54.3
82.4
60.2
74.0
0.158
35.4
LoRA
59.1
84.0
67.6
77.1
0.131
46.9
AdaptFormer
57.8
83.8
64.4
77.2
0.128
45.7
IV-NEXT
Attentive Probe
62.7
88.5
67.0
74.0
0.135
47.3
LoRA
63.1
88.5
72.8
75.9
0.142
55.2
AdaptFormer
63.4
88.6
70.0
76.3
0.118
57.6
Adapting Image vs. Video Models. Video models consistently outperform image-to-video
adaptation, especially for motion, while DINOv3 excels on spatial tasks. Best and
best per backbone.
Appearance
Motion
Spatial
Params (M) ↓
CAER top-1 ↑
NurViD top-1 ↑
IndustReal top-1 ↑
MammAlps top-1 ↑
ScanNet AbsRel ↓
VSPW mIoU ↑
Base
Full Finetuning
87.1
59.1
87.3
66.7
71.8
0.164
40.9
Attentive Probe
7.4
60.3
85.5
63.1
69.9
0.157
40.4
Selective
BitFit
7.5
61.1
85.9
63.8
69.9
0.147
44.8
LayerNorm
7.5
60.4
85.7
64.6
69.6
0.150
43.5
SSF
7.6
60.7
86.2
65.3
69.6
0.136
47.0
Additive
VPT
8.1
61.2
85.6
65.8
70.9
0.152
48.7
VFPT
8.2
60.9
86.1
64.8
69.9
0.146
48.6
BPT
8.5
60.7
86.5
65.5
69.9
0.144
49.3
Adapt
AdaptFormer
11.3
61.5
86.2
66.7
71.9
0.137
50.1
ST-Adapter
12.5
61.4
85.9
67.1
70.6
0.148
49.4
Rank
LoRA
12.1
62.1
86.9
68.7
72.6
0.156
48.2
DoRA
13.7
60.3
85.7
66.9
68.8
0.159
45.9
Comparison of PEFT Methods for Video. Low-rank and adapter methods perform best while
keeping parameters limited. No single method dominates: LoRA excels on motion and appearance, while
AdaptFormer performs best on spatial tasks. Best and second-best
are highlighted.
Comparison of Probe Methods for Video. In isolation, the best probe varies per task.
Combining probe and PEFT reduces this variability, with attentive probe performing best overall.
Disentangling Temporal Modeling from Input Frames. Performance-throughput trade-offs
when varying temporal context jointly with the input (matched) or only within the backbone, PEFT, and
probe while keeping the input fixed at 16 frames (input-fixed). This separates performance changes due
to temporal-modeling and input-information.
How to Distribute Temporal Context. Fixing one component while varying the other two
at Tc isolates each component's contribution to temporal modeling. Curves above input-fixed
indicate that the corresponding component improves with increased temporal context.
Recommendations
Appearance
Video-pretrained models consistently outperform image-pretrained alternatives, but only lightweight
adaptation is required. In particular, image-style adapter and low-rank PEFT methods, combined with a
backbone with limited temporal context (e.g. T=2) and simple temporal aggregation in
the prediction head, provide a good efficiency–accuracy trade-off.
Motion
Temporal modeling in the backbone is critical. Video-pretrained backbones are
substantially more effective than image models, and increasing temporal reasoning in the backbone
leads to the largest performance gains. Extensive temporal modeling in the PEFT and probe is not
necessary, although it can partially compensate for reduced backbone context when higher throughput
is required.
Spatial
Adapting DINOv3 is most effective, likely due to large-scale
dense pretraining. However, we find benefits from increased temporal modeling within video backbones. Conversely,
low-capacity PEFT and probes struggle to preserve spatial detail while integrating temporal context,
meaning temporal modeling in adaptation modules should be minimized.
BibTeX
@article{strater2026temporal,
title = {Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?},
author = {Str{\"a}ter, Luc and Doughty, Hazel},
journal = {arXiv preprint},
year = {2026}
}
