Weili Nie

I am a Senior Research Scientist in Fundamental Generative AI Research Team at NVIDIA Research. I got my PhD from the ECE Department at Rice University, advised by Ankit B. Patel. Prior to joining Rice, I got my BS degree from Huazhong University of Science and Technology (HUST) and MS degree from Southeast University. During my PhD studies, I have done several internships at NVIDIA Research and VMware Research.

Email  /  Google Scholar  /  GitHub  /  Twitter

profile photo
Research

My current research mainly focuses on deep generative models and their applications to real-world problems. I have also been working on model robustness and compositional generalization.

I'm looking for research interns to work on both fundamental and applied aspects of deep generative models.


Publications

Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization
Siyi Gu, Minkai Xu, Alexander S Powers, Weili Nie, Tomas Geffner, Karsten Kreis, Jure Leskovec, Arash Vahdat, Stefano Ermon
Advances in Neural Information Processing Systems (NeurIPS), 2024
arXiv

Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola Kovachki, Arash Vahdat
Advances in Neural Information Processing Systems (NeurIPS), 2024
arXiv

Molecule Generation with Fragment Retrieval Augmentation
Seul Lee, Karsten Kreis, Srimukh Veccham, Meng Liu, Danny Reidenbach, Saee Paliwal, Arash Vahdat†, Weili Nie†,
Advances in Neural Information Processing Systems (NeurIPS), 2024
(†Equal Advising)
arXiv

BlobGEN-3D: Compositional 3D-Consistent Freeview Image Generation with 3D Blobs
Chao Liu, Weili Nie, Sifei Liu, Abhishek Badki, Hang Su, Morteza Mardani, Benjamin Eckart, Arash Vahdat,
ACM SIGGRAPH Asia, 2024
arXiv / project page

DiffUHaul: A Training-Free Method for Object Dragging in Images
Omri Avrahami, Rinon Gal, Gal Chechik, Ohad Fried, Dani Lischinski, Arash Vahdat†, Weili Nie†
ACM SIGGRAPH Asia, 2024
(†Equal Advising)
arXiv / project page

Compositional text-to-image generation with dense blob representations
Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, Arash Vahdat
International Conference on Machine Learning (ICML) , 2024
arXiv / project page

Fast training of diffusion models with masked transformers
Hongkai Zheng*, Weili Nie*, Arash Vahdat, Anima Anandkumar
Transactions on Machine Learning Research (TMLR), 2024
(*Equal Contribution)
arXiv / code

Efficient video diffusion models via content-frame motion-latent decomposition
Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, Anima Anandkumar
International Conference on Learning Representations (ICLR), 2024
openreview / project page / code

State-specific protein-ligand complex structure prediction with a multiscale deep generative model
Zhuoran Qiao, Weili Nie, Arash Vahdat, Thomas F Miller III, Anima Anandkumar
Nature Machine Intelligence, 2024
arXiv / code

Unsupervised discovery of steerable factors when graph deep generative models are entangled
Shengchao Liu, Chengpeng Wang, Jiarui Lu, Weili Nie, Hanchen Wang, Zhuoxinran Li, Bolei Zhou, Jian Tang
Transactions on Machine Learning Research (TMLR) , 2024
arXiv / code / project page

Multi-modal molecule structure-text model for text-based retrieval and editing
Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, Anima Anandkumar
Nature Machine Intelligence, 2023
arXiv / code / project page

Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Ming-Yu Liu, Yuke Zhu, Mohammad Shoeybi, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
Findings of Empirical Methods in Natural Language Processing (EMNLP) , 2023
arXiv

Fast sampling of diffusion models via operator learning
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, Anima Anandkumar
International Conference on Machine Learning (ICML) , 2023
arXiv / code

I2SB: Image-to-image Schrodinger bridge
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou, Weili Nie†, Anima Anandkumar†
International Conference on Machine Learning (ICML) , 2023
(†Equal Advising)
arXiv / project page / code

A critical revisit of adversarial robustness in 3D point cloud recognition with diffusion-driven purification
Jiachen Sun, Jiongxiao Wang, Weili Nie, Zhiding Yu, Chaowei Xiao, Zhuoqing Mao
International Conference on Machine Learning (ICML) , 2023
arXiv / code

Dr-Fairness: Dynamic data ratio adjustment for fair training on real and generated data
Yuji Roh, Weili Nie, De-An Huang, Steven Euijong Whang, Arash Vahdat, Anima Anandkumar
Transactions on Machine Learning Research (TMLR) , 2023
openreview / code

Retrieval-based controllable molecule generation
Zichao Wang*, Weili Nie*, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, Anima Anandkumar
International Conference on Learning Representations (ICLR) , 2023 (Spotlight)
(*Equal Contribution)
arXiv / code

DensePure: understanding diffusion models towards adversarial robustness
Zhongzhu Chen*, Kun Jin*, Chaowei Xiao*, Jiongxiao Wang*, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, Dawn Song
International Conference on Learning Representations (ICLR), 2023
(*Equal Contribution)
arXiv / project page / code

Defending against adversarial audio via diffusion model
Shutong Wu, Jiongxiao Wang, Wei Ping, Weili Nie, Chaowei Xiao
International Conference on Learning Representations (ICLR), 2023
arXiv / code

Test-time prompt tuning for zero-shot generalization in vision-language models
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao
Advances in Neural Information Processing Systems (NeurIPS), 2022
arXiv / project page / code

Diffusion models for adversarial purification
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, Anima Anandkumar
International Conference on Machine Learning (ICML), 2022
arXiv / project page / code / slides

Bongard-HOI: Benchmarking few-shot visual reasoning for human-object interactions
Huaizu Jiang*, Xiaojian Ma*, Weili Nie, Zhiding Yu, Yuke Zhu, Anima Anandkumar
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022 (Oral)
(*Equal Contribution)
arXiv / code / slides

RelViT: Concept-guided vision transformer for visual relational reasoning
Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, Anima Anandkumar
International Conference on Learning Representations (ICLR), 2022
OpenReview / arXiv / code / slides

Controllable and compositional generation with latent-space energy-based models
Weili Nie, Arash Vahdat, Anima Anandkumar
Advances in Neural Information Processing Systems (NeurIPS), 2021
arXiv / project page / code / slides / poster

Bongard-LOGO: A new benchmark for human-level concept learning and reasoning
Weili Nie, Zhiding Yu, Lei Mao, Ankit B. Patel, Yuke Zhu, Anima Anandkumar
Advances in Neural Information Processing Systems (NeurIPS), 2020 (Spotlight)
arXiv / code / slides

Semi-supervised StyleGAN for disentanglement learning
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit B. Patel, Anima Anandkumar
International Conference on Machine Learning (ICML), 2020
arXiv / project page / datasets

Towards a better understanding and regularization of the GAN training dynamics
Weili Nie, Ankit B. Patel
Uncertainty in Artificial Intelligence (UAI), 2019 (Oral)
arXiv / slides / code

RelGAN: Relational generative adversarial networks for text generation
Weili Nie, Nina Narodytska, Ankit B. Patel
International Conference on Learning Representations (ICLR), 2019
OpenReview / poster / code

A theoretical explanation for perplexing behaviors of backpropagation-based visualizations
Weili Nie, Yang Zhang, Ankit B. Patel
International Conference on Machine Learning (ICML), 2018
arXiv / slides / code

QG-Net: A data-driven question generation model for educational content
Zichao Wang, Andrew Lan, Weili Nie, Andrew E. Waters, Phillip J. Grimaldi, Richard Baraniuk
Learning at Scale (L@S) , 2018
code


Conference Reviewer

- ICML 2020-2023 (Outstanding reviewer in 2022)
- NeurIPS 2020-2022 (Top 10% of high-scoring reviewers in 2020)
- ICLR 2021-2023
- AAAI 2020-2022
- IJCAI 2021

Template from here