4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

This is the official implementation of 4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction.

The current release includes inference code for HO3D. Stay tuned for more!

Environment setup

You need an NVIDIA GPU with a compatible driver, Conda, Git, a C++ compiler, and internet access.

Run these commands to install the dependencies and activate the environment:

bash setup_env.sh
conda activate 4dhof

Download the MANO models from the official MANO website (sign up and accept the license). Extract the download and place MANO_RIGHT.pkl at _DATA/mano/MANO_RIGHT.pkl in this repository.

HO3D Experiments

From the repository root, download the checkpoint and HO3D initialization:

bash download_assets.sh

Files are saved in this layout:

checkpoints/4dhof.pth
_DATA/ho3dv3_init/<sequence>/
    hand.pkl
    mesh.pkl
    pose.pkl
    mask.pkl
    contact.pkl
    intrinsics.pkl

Download HO3D_v3 and set --ho3d-root to its directory containing train/, train.txt, and models/.

Run inference on HO3D sequences from the repository root:

python infer_ho3d.py --ho3d-root /path/to/HO3D_v3

Results are saved to results/ho3d.

Evaluate all sequences:

python eval_ho3d.py --ho3d-root /path/to/HO3D_v3

Results are saved to eval_ho3d.json in the prediction directory.

Cite

If you find our work useful, please consider citing:

@article{li2026_4dhof,
  title   = {4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction},
  author  = {Li, Shiqi and Cho, Sean and Li, Yijie and Guo, Fengzhi and Wen, Bowen and Zhang, Cheng},
  journal = {arXiv preprint arXiv:2610.08782},
  year    = {2026}
}

Acknowledgments

We thank the authors of HOLD, MagicHOI, AGILE, CHOIR, SAM, MoGe, Hunyuan3D, MV-Adapter, FoundationPose, WiLoR, HACO, DINO, and Point Transformer V3 for sharing their work and code.