Parameter-efficient finetuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT, and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across task types and demonstrate the importance of temporal context allocation for effective video adaptation.
Key Findings
1
Standard PEFT beats probing at almost no cost
A standard PEFT method (e.g. AdaptFormer) combined with an attentive probe outperforms the
attentive probe alone on all tasks at only a 3% throughput cost. It also outperforms or matches
full finetuning, while training 8× fewer parameters.
2
A simple attentive probe is sufficient
Although probe performance differs substantially per task, this variability largely disappears
once combined with PEFT, suggesting that PEFT removes the need for elaborate task-specific
decoders and a simple attentive probe suffices.
3
Motion needs temporal modeling, in the backbone
Temporal modeling is most beneficial for motion tasks and driven by the backbone, which also
dominates throughput. For appearance and spatial tasks, the benefit of more frames comes mainly
from additional information rather than temporal reasoning.
4
Image-to-video PEFT rarely helps
Image-to-video PEFT (e.g. ST-Adapter) rarely improves over standard PEFT on both image and video
backbones, while reducing throughput. Temporal aggregation in the probe even degrades dense
prediction. Only when a video backbone's temporal context is reduced can temporal PEFT partially
compensate on motion tasks.
Distributing Temporal Context
Video adaptation requires reasoning over multiple frames, making a central design choice not just whether to model temporal information, but where this temporal reasoning should occur. Existing approaches make this choice implicitly by assigning temporal modeling to a single component, leaving a broader design space underexplored in which multiple components may benefit from temporal reasoning at different temporal scales. We treat temporal context as a flexible resource that can be distributed across components with different temporal extents, enabling a systematic study of where performance gains arise.
Distributing Temporal Context. Existing works place temporal modeling in a single
component: either in the PEFT modules (a), where a frame-wise backbone is combined with temporal
adaptation, or in the backbone (b), where temporally aware representations are adapted by static
PEFT and a linear classifier. We instead allow temporal modeling across backbone, PEFT, and probe
simultaneously (c) and study how temporal context should be distributed across components (d).
The ScarceVid Benchmark
Model adaptation is most valuable in low-resource settings, where specialized domains make adaptation
necessary and limited data makes full finetuning prone to overfitting. Simply subsampling large-scale
datasets fails to reproduce the challenges of genuinely scarce data, so ScarceVid instead consists of
naturally low-resource video tasks spanning appearance (CAER, NurViD), motion (IndustReal, MammAlps)
and spatial understanding (ScanNet, VSPW).
The tasks are selected for structural data scarcity (e.g. healthcare, wildlife
monitoring, per-pixel annotation), capability coverage across appearance, motion and
spatial understanding, a range of annotation densities from video-level labels to
per-pixel annotations, and diversity in viewpoint and domain.
Appearance
Motion
Spatial
CAER
NurViD
IndustReal
MammAlps
ScanNet
VSPW
Task
Emotion analysis
Procedure classification
Step recognition
Behavior recognition
Depth estimation
Semantic segmentation
Viewpoint
3rd
3rd
1st
3rd
1st
3rd
Domain
Television
Medical
Assembly
Wildlife
Indoor
Web
Classes
7
51
75
11
–
124
Train
9,222
3,899
3,667
4,205
1,201
2,806
Validation
1,316
586
1,928
686
312
343
Test
2,637
1,121
3,678
1,244
100
387†
The ScarceVid Benchmark. Low-resource video adaptation tasks across appearance,
motion, and spatial video understanding. The datasets span diverse annotation granularities,
viewpoints, and domains. †VSPW test labels are not public; we evaluate on the
validation set.
Results
We study model adaptation across the ScarceVid tasks along five questions: when to adapt an image or
a video model (Q1), which PEFT (Q2) and probe methods (Q3) work best, and how temporal context should
be distributed across input, backbone, PEFT and probe (Q4 and Q5).
Q1When to adapt an image vs. a video model?
Conclusion. Adapting a video-pretrained backbone is preferable for motion tasks in the
low-data regime. For appearance and spatial tasks, the pretraining modality does not predict performance
with current models. Combining attentive probing with PEFT reduces the differences between backbones, but
adding temporal modeling to the PEFT method on image backbones is not beneficial.
Appearance
Motion
Spatial
CAER top-1 ↑
NurViD top-1 ↑
IndustReal top-1 ↑
MammAlps top-1 ↑
ScanNet AbsRel ↓
VSPW mIoU ↑
SigLIP2 image
Attentive Probe
52.1
79.0
47.4
66.9
0.237
33.8
LoRA
63.0
87.0
67.7
73.4
0.136
53.5
AdaptFormer
61.2
87.1
64.2
72.4
0.122
54.9
ST-Adapter
62.5
87.5
66.9
72.4
0.131
55.7
DINOv3 image
Attentive Probe
58.9
85.2
58.1
71.5
0.120
56.4
LoRA
62.8
87.3
69.8
72.6
0.111
60.6
AdaptFormer
61.8
87.1
67.5
72.6
0.092
61.2
ST-Adapter
62.4
86.9
69.2
75.6
0.106
61.1
VJEPA2 video
Attentive Probe
54.3
82.4
60.2
74.0
0.158
35.4
LoRA
59.1
84.0
67.6
77.1
0.131
46.9
AdaptFormer
57.8
83.8
64.4
77.2
0.128
45.7
IV-Next video
Attentive Probe
62.7
88.5
67.0
74.0
0.135
47.3
LoRA
63.1
88.5
72.8
75.9
0.142
55.2
AdaptFormer
63.4
88.6
70.0
76.3
0.118
57.6
Adapting Image vs. Video Models. Video-pretrained backbones perform best on motion
tasks, while for appearance and spatial tasks the pretraining modality does not predict performance.
PEFT reduces the differences between backbones. Best and
best per backbone are highlighted. ViT-L backbones; IV-Next =
InternVideo-Next.
Q2Which PEFT methods work best for video?
Conclusion. AdaptFormer offers the best trade-off between performance and efficiency
across tasks. On a full temporal context video backbone, temporal modeling in the adapter adds training
and inference cost without improving performance.
Appearance
Motion
Spatial
Params (M) ↓
CAER top-1 ↑
NurViD top-1 ↑
IndustReal top-1 ↑
MammAlps top-1 ↑
Clips/s ↑
ScanNet AbsRel ↓
VSPW mIoU ↑
Frames/s ↑
Base
Full Finetuning
87.1
60.3±0.6
86.5±0.7
67.2±0.5
71.7±0.7
265.5
0.167±0.006
40.2±0.7
2041.3
Attentive Probe
7.4
59.7±0.8
85.6±0.3
62.9±0.9
69.4±0.8
264.9
0.160±0.004
40.9±0.2
1860.5
Selective
BitFit
7.4
60.4±0.7
85.7±1.0
63.7±0.9
69.9±1.3
265.7
0.153±0.002
44.2±0.5
1855.5
LayerNorm
7.4
59.2±1.0
85.7±0.5
64.3±0.3
69.3±0.8
264.5
0.152±0.003
44.4±0.4
1859.2
SSF
7.6
60.2±1.2
85.7±0.7
65.4±0.6
67.4±2.0
217.4†
0.146±0.003
47.0±0.2
1555.1†
Additive
VPT
8.4
59.7±1.1
85.7±0.4
65.3±0.3
70.0±1.0
243.9
0.154±0.004
47.8±0.4
1649.6
VFPT
8.8
59.4±0.7
85.8±0.6
64.4±0.3
67.2±0.8
233.0
0.161±0.005
47.5±1.2
1640.2
BPT
8.5
60.2±0.5
85.7±0.6
65.0±1.0
70.0±1.4
244.3
0.156±0.005
48.7±0.4
1549.3
Adapter
AdaptFormer
11.3
60.2±1.3
86.0±0.6
66.7±0.3
71.1±0.5
255.9
0.151±0.004
49.9±0.5
1808.8
ST-Adapter
14.1
60.9±0.8
86.2±0.8
67.0±0.3
69.6±0.9
231.7
0.148±0.003
49.2±0.5
1692.9
Low-rank
LoRA
11.7
60.7±1.0
86.0±0.6
68.1±0.2
70.0±2.5
244.4†
0.163±0.010
47.3±0.6
1728.7†
DoRA
13.3
59.6±0.4
85.2±0.4
66.2±0.3
66.6±1.1
237.4†
0.169±0.007
45.6±0.5
1683.0†
Comparison of PEFT Methods for Video. Adapter methods perform best overall. Temporal
modeling in the adapter adds training and inference cost without improving performance. Mean
±sample std over 5 seeds; best and
second-best are highlighted. †Weights can be merged to
obtain attentive probe throughput. InternVideo-Next ViT-B backbone.
Q3Which probe works best for video?
Conclusion. In isolation, probe performance is task-dependent. When combined with PEFT,
however, a generic attentive probe suffices, removing the need for task-specific probes.
Comparison of Probe Methods for Video. In isolation, the best probe varies per task,
but combined with PEFT (hatched), a simple attentive probe outperforms or is within one standard
deviation of task-specific probes. ▼ Per-frame probes predict each frame's output using only that
frame's features. Error bars show standard deviation over 5 seeds.
Q4Do input frames or temporal modeling matter more?
Conclusion. Appearance and spatial tasks benefit primarily from more input frames rather
than temporal modeling. In contrast, motion tasks depend heavily on temporal reasoning.
Disentangling Temporal Modeling from Input Frames. Performance-throughput trade-offs
when varying temporal context together with the input (matched) or only within the backbone, PEFT,
and probe while keeping the input fixed at 16 frames (input-fixed). Motion tasks depend most on
internal temporal modeling, while appearance and spatial tasks benefit most from more input frames.
Performance is relative to T = Tc = 16 and averaged per task category.
Q5How to distribute the temporal context?
Conclusion. Temporal context should be allocated selectively rather than uniformly. For
appearance, throughput is best saved by reducing backbone context. Motion benefits most from temporal
context in the backbone, though allocating temporal context to PEFT can improve performance when high
throughput is required. Spatial tasks also favor temporal context in the backbone, but with limited
adaptation data not in the probe.
How to Distribute Temporal Context. Input-fixed varies the temporal context
Tc of backbone, PEFT, and probe jointly, while each “+X Fixed” curve keeps
component X at 16 frames and varies the other two with Tc. A curve above input-fixed thus
means that X benefits from temporal context. Temporal context is most effective in the backbone, and
for spatial tasks harmful in the probe. Performance is relative to
T = Tc = 16 and averaged per task category.
Recommendations
Appearance
A video-pretrained model (InternVideo-Next) outperforms image-pretrained alternatives, but only
lightweight adaptation is required. In particular, image-style adapter PEFT methods, combined with
limited temporal context in the backbone (e.g. Tbackbone = 2)
and the probe (e.g. Tprobe = 2), provide a good
efficiency–accuracy trade-off.
Motion
Temporal modeling in the backbone is critical. Here, video-pretrained backbones are
more effective than image models and increasing temporal reasoning in the backbone leads to the
largest performance gains. In contrast, temporal modeling in the PEFT and probe is not necessary,
although it can partially compensate for reduced backbone context when higher throughput is required.
Spatial
While adapting DINOv3 is most effective, likely due to large-scale dense pretraining, we find benefits
from temporal modeling within video backbones. However, when adapting with limited data, low-capacity
PEFT and probes struggle to preserve spatial detail while integrating temporal context. This means
temporal context in PEFT and especially the probe should be minimized.
BibTeX
@inproceedings{strater2026temporal,
title = {Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?},
author = {Str{\"a}ter, Luc and Doughty, Hazel},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
