Temporal Residual Jacobians for Rig-free Motion Transfer

[ECCV 2024]

1University College London 2Adobe Research 3University of Montreal

Abstract

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume access to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two dedicated neural networks that individually predict the local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its effectiveness in generating realistic and natural-looking animations on unseen body shapes.

Motion sequences

We showcase our method on a wide variety of sequences from AMASS and humanoid meshes with significantly different shape and topology. All results show transferred motion to an unseen shape without rigging.


Running (in-place)

Number of frames: 278

woman

holey man

alien

Punching

Number of frames: 187

superhero

holey man

alien

Jumping jacks

Number of frames: 460

Notice secondary motion on the chubby man around the chest and belly regions.

chubby man

superhero

cyclops

Walking

Number of frames: 1845

holey man

superhero

skeleton zombie

Dancing

Number of frames: 1088

original capture (unsynchronized)

skeleton zombie

maynard

Generalization beyond humanoids

Temporal Residual Jacobians generalizes to scans (FAUST) and in the wild meshes (Mixamo).


FAUST scan (running)

FAUST scan (punching)

Mixamo skeleton zombie (running)

Mixamo skeleton zombie (punching)

Motion transfer from human to animal

Temporal Residual Jacobians can generalize motion transfer to animals even when it's trained only on humans.


Source

Target (reptile)

Motion transfer on animals

DeformingThings4D

DeformingThings4D is a synthetic dataset of animations including animals.

Source (bear)

Target (triceraptor)

Target (wolf)

Source (dog)

Target (triceratop)

Target (wolf)

COP3D: Common Pets in 3D

COP3D dataset contains in the wild animal reconstructions from consumer devices.

Original capture

Source (dog)

Target (wolf)

Target (reptile)

Target (triceratop)

Comparison

Running

In comparison to ours, NJF has jitters and artefacts especially at later time steps.

Vertex ODE (holey man)

NJF (holey man)

Ours (holey man)

Vertex ODE (scan)

NJF (scan)

Ours (scan)

Limitations

We do not impose physics constraints, therefore, our animation can have self-intersections. An interesting direction is to incorporate constraints for collision detection.

BibTeX


                @inproceedings{10.1007/978-3-031-73636-0_6,
                    author = {Muralikrishnan, Sanjeev and Dutt, Niladri and Chaudhuri, Siddhartha and Aigerman, Noam and Kim, Vladimir and Fisher, Matthew and Mitra, Niloy J.},
                    title = {Temporal Residual Jacobians for Rig-Free Motion Transfer},
                    year = {2024},
                    isbn = {978-3-031-73635-3},
                    publisher = {Springer-Verlag},
                    address = {Berlin, Heidelberg},
                    url = {https://doi.org/10.1007/978-3-031-73636-0_6},
                    doi = {10.1007/978-3-031-73636-0_6},
                    booktitle = {Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LVIII},
                    pages = {93–109},
                    numpages = {17},
                    location = {Milan, Italy}
                    }
          

Template adapteed from Nerfies