Residual Motion Splatting for Continuous Space-Time Video Super-Resolution

Vincent Herfeld^1,2 Eva Coupeté¹ Julien Mairal² Jean Ponce^1,3

¹Enhance Lab ²Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK ³École normale supérieure - PSL

Continuous space-time video super-resolution upscales a low-resolution, low-frame-rate input (left) in both space and time (right: ours, space, time).

Abstract

Continuous space-time video super-resolution produces a high-resolution, high-frame-rate video from a low-resolution, low-frame-rate one at arbitrary spatial scales and temporal instants. Classical spatial multi-frame super-resolution has shown that detail beyond a single frame’s sampling limit can be recovered by aligning complementary observations with sub-pixel precision and accumulating them on a finer grid. Existing learning-based methods largely depart from this principle: some align and fuse frames only at low resolution before regressing a continuous high-resolution representation, while others interpolate each frame to the output resolution before splatting. In both cases, the per-frame samples are resampled before fusion, so their sub-pixel positions are lost before frames are combined. Restoring this principle in a learned continuous model raises two challenges: estimating sub-pixel motion at unobserved times, and handling holes and occlusions when splatting a coarse grid onto a finer one. We address the first by predicting per-pixel continuous trajectories that refine linear optical-flow paths with smooth residual motion, using features integrated over the full video sequence. We address the second by directly splatting consecutive low-resolution features in a learned latent space onto a high-resolution grid at their predicted sub-pixel positions with predicted visibility weights, before a decoder resolves remaining ambiguities. This makes temporal interpolation and multi-frame super-resolution a single operation. Our method improves both reconstruction fidelity and perceptual quality on multiple standard benchmarks over the state of the art, while running up to an order of magnitude faster.

Method

Overview diagram: the input low-resolution video is encoded, features are splatted along spline-corrected trajectories onto a high-resolution grid at time t+tau, then decoded to the output high-resolution video.

Continuous residual trajectories

Each low-resolution feature follows the optical flow, corrected by a cubic B-spline residual predicted from features that carry context from the whole sequence. Any time between two input frames can be queried.

Trajectory from u_t to u_t+1: the linear optical-flow path (dashed blue) corrected by a B-spline residual (orange) gives the full trajectory (black).

Sub-pixel splatting

Features from both keyframes are splatted straight onto the grid at their sub-pixel landing positions , weighted by a predicted visibility and a small kernel . Temporal interpolation and multi-frame super-resolution become this single operation; a decoder then maps to RGB.

A low-resolution pixel (left grid) is splatted to a sub-pixel position on the high-resolution grid (right) and spread over its neighbours.

Results

C-STVSR at spatial scale, temporal (GoPro, Adobe240) and temporal (Vid4). We report PSNR-Y / LPIPS. *Center* evaluates frames 1, 5, 9 of each 9-frame sequence, *Average* all output frames. Methods marked have no public weights, so we report their published PSNR-Y only. We highlight the best result in **bold** and the second best with underline.
Method	# Par.	FPS	Vid4	GoProCenter	GoProAverage	AdobeCenter	AdobeAverage
MoTIF	12.6M	2.5	25.76/0.3024	31.04/0.1554	30.04/0.1587	30.63/0.1408	29.82/0.1457
BF-STVSR	13.5M	2.2	25.84/0.2931	31.17/0.1507	30.22/0.1543	30.83/0.1313	30.12/0.1350
GS-STVSR	12.7M	—	26.04/—	31.33/—	30.35/—	31.13/—	30.35/—
	13.7M	1.3	26.76/0.2649	32.96/0.1458	32.26/0.1485	32.91/0.1186	32.29/0.1223
-Large	20.6M	—	26.82/—	33.09/—	32.36/—	33.08/—	32.45/—
Ours	16.5M	14.5	26.87/0.2760	33.40/0.1261	32.54/0.1306	33.55/0.0986	32.78/0.1032

Video comparison

Sequence

Method

Ours

Adobe240 test sequences at spatial, temporal scale. Switching method keeps the playback position. Keys 1–4 also switch method.

Speed

Because no stage runs at high resolution before the splat, our method is 6–11 faster than the C-STVSR methods with public weights. FPS is measured on full-resolution GoPro test clips ( input, output) on a single RTX 6000 Ada GPU, timing only the model's forward pass.

Ours

14.5 FPS

MoTIF

2.5 FPS

BF-STVSR

2.2 FPS

1.3 FPS

Qualitative results

Adobe240 IMG_0015 ground-truth frame 1568 with two regions of interest — Fast motion on Adobe240 IMG_0015 ( space, time), at the intermediate times between input keyframes 1561 and 1569. A truck drives across the frame. Ours keeps the back of the truck sharp and the “LYDIG” text on its door readable at every intermediate time. blurs the truck and adds straight lines where the borders between its space-time processing blocks show in the output, while MoTIF and BF-STVSR produce several overlapping copies of it.

motif, frame 1563 — Fast motion on Adobe240 IMG_0015 ( space, time), at the intermediate times between input keyframes 1561 and 1569. A truck drives across the frame. Ours keeps the back of the truck sharp and the “LYDIG” text on its door readable at every intermediate time. blurs the truck and adds straight lines where the borders between its space-time processing blocks show in the output, while MoTIF and BF-STVSR produce several overlapping copies of it.

Adobe240 IMG_0004a ground-truth frame 238 with region of interest — Parallax on Adobe240 IMG_0004a ( space, time), at the intermediate times between input keyframes 233 and 241. The scene pans around with a rail in the foreground and a staircase in the background is visible through the railing loop. Our method keeps the bar intact and correctly pans the background at every intermediate time. blurs the rail and background through the loop, BF-STVSR breaks the bar where it overlaps the background, and MoTIF smears the two layers together.

motif, frame 235 — Parallax on Adobe240 IMG_0004a ( space, time), at the intermediate times between input keyframes 233 and 241. The scene pans around with a rail in the foreground and a staircase in the background is visible through the railing loop. Our method keeps the bar intact and correctly pans the background at every intermediate time. blurs the rail and background through the loop, BF-STVSR breaks the bar where it overlaps the background, and MoTIF smears the two layers together.

BibTeX

@article{herfeld2026residual,
  title   = {Residual Motion Splatting for Continuous Space-Time Video Super-Resolution},
  author  = {Herfeld, Vincent and Coupet{\'e}, Eva and Mairal, Julien and Ponce, Jean},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}