Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization (CVPR2026)

This is the code for the paper "Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization" (CVPR2026).

paper, arXiv, project page

Getting started

This project is built on ProgMoGen.

1. Setup environment

Follow the instruction of PriorMDM (or MDM) to create a conda environment and install necessary packages in order to use the MDM model.

2. Download necessary data files

The data files required for running experiments are the same as MDM. The required files include:

• glove: Download files from progmogen/prepare/download_glove.sh and place the directory as progmogen/glove.

cd progmogen
bash prepare/download_glove.sh

• body_models: Download files from progmogen/prepare/download_smpl_files.sh (a folder named smpl) and place it under progmogen/body_models.

bash prepare/download_smpl_files.sh

You can also refer to paths in progmogen/config_data.py to check whether files are placed correctly.

3. Get HumanML3D data

Since the raw HumanML3D motion representation does not support direct recovering of SMPL mesh, in this work we adopt RoHM representation and obtain 284-dim motion representation for each frame. So the dataset has to be pre-processed for this new representation. (You can skip this if only for running demo and main experiments)

Place HumanML3D folder under progmogen/dataset as progmogen/dataset/HumanML3D.

a. The easy way (for demo)

HumanML3D - If you wish to run the demo only, motion data is not needed and we only need Mean.npy and Std.npy. We have provided these files and some retrieved samples under progmogen/dataset/HumanML3D.

b. Full data (text + motion capture for retrieval)

HumanML3D - Follow the instructions in HumanML3D, prepare motion capture data, and then copy the result dataset to our repository:

cd ..
git clone https://github.com/EricGuo5513/HumanML3D.git
unzip ./HumanML3D/HumanML3D/texts.zip -d ./HumanML3D/HumanML3D/
cp -r HumanML3D/HumanML3D progmogen/dataset/HumanML3D
cd progmogen

Then create motion data in the new representation.

python -m data_loaders.humanml.scripts_smpl_2.motion_representation_smpl_to_feature_v4_createdataset

5. Pre-trained Model

Get pre-trained motion model under the above new representation. We have already provided this model.

• pretrained MDM model: model000600000.pt and args.json. Place the folder as progmogen/save/my_dim284_diffstep1000_humanml_trans_enc_512.

Get MotionCritic pre-trained model for reward function. Follow MotionCritic document for the download link.

• place motioncritic_pre.pth under progmogen/MotionCritic/MotionCritic/critic

Using MotionCritic as reward function is optional.

5. Install blender for visualization (optional)

We use blender code from project TEMOS for visualization. Follow the instruction to install blender and bpy dependencies. In progmogen/script/*.sh scripts, replace blender_app path with your own path to blender application, and replace project_dir with your own absolute path to this github project.

Experiments

We provide code for the main generation tasks:

Task-1: very narrow gap (we consider two motion lengths of 100 and 150 frames)

Task-2: very low barrier

Task-3: assign number of walking steps.

We seperate the retrieval and optimization part in this code repository. Examples of relational task parsing via LLM are provided under progmogen/result_task_parsing/.

The task files which define constraints and task parsing results are provided in progmogen/task_configs_smpl_exp/.

cd progmogen

Retrieval

Run retrieval for each task, which requires the full dataset. This part is optional and we also indicate the sample to be used from retrieval in the optimization part.

sh script_retrieve/run_retrieve_avoid_barrier_ours.sh
sh script_retrieve/run_retrieve_narrow_gap_ours.sh  
sh script_retrieve/run_retrieve_walksteps_ours_6step.sh

Optimization

Ours:

sh script/run_opt_narrow_gap_ours.sh 100
sh script/run_opt_narrow_gap_ours.sh 150
sh script/run_opt_avoid_barrier_ours.sh
sh script/run_opt_walksteps_ours2.sh

ProgMoGen+DNO baseline:

sh script/run_opt_narrow_gap_dno.sh 100
sh script/run_opt_narrow_gap_dno.sh 150
sh script/run_opt_avoid_barrier_dno.sh
sh script/run_opt_walksteps_dno2.sh

Evaluation

The generated motion results will be saved to result. See scripts under script for the evaluation part. The reported experimental result for Task-1 is averaged over motion lengths of 100 and 150.

Visualization

See scripts under script for the visualization part. Video results will also be saved to result along with the generated motions.

Acknowledgements

Our code is heavily built on: PriorMDM, MDM, TEMOS, GMD, HumanML3D, ProgMoGen, DNO, MotionCritic. We thank them for kindly releasing their code.

Bibtex

If you find this code useful in your research, please consider citing:

@inproceedings{liu2026towards,
title={Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization},
author={Hanchao Liu, Fang-Lue Zhang, Shining Zhang, Tai-Jiang Mu, Shi-Min Hu},
booktitle={CVPR},
year={2026}
}

License

This code is distributed under an MIT LICENSE.

Note that our code depends on other libraries, including CLIP, SMPL, SMPL-X, PyTorch3D, and uses datasets that each have their own respective licenses that must also be followed.