Xiaodan Du

SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction

Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, Alexander H. Liu

arXiv | Code
The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

We introduce SHuBERT, a self-supervised representation learning approach that adapts the masked prediction for multi-stream visual sign language input, learning to predict multiple targets for corresponding to clustered hand, face, and body pose streams. SHuBERT achieves state-of-the-art performance on multiple sign language benchmarks.

SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale

Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu

arXiv
The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

We introduce SignMusketeers for Sign Language Translation at Scale. Using only 3% of compute, 41x less pretraining data, and 160x less pretraining epochs, it achieves competitive performance(-0.4 BLEU) compared to the recent ASL-English Translation SOTA.

Generative Models: What do they know? Do they know things? Let's find out!

Previous title: Intrinsic LoRA: A Generalist Approach for Discovering Knowledge in Generative Models

Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, Anand Bhattad

Project Page | arXiv | Code
IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) workshops, 2024

We introduce Intrinsic LoRA (I-LoRA), a universal, plug-and-play approach that transforms any generative model into a scene intrinsic predictor, capable of extracting intrinsic scene maps directly from the original generator network.

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

Haochen Wang*, Xiaodan Du*, Jiahao Li*, Raymond A. Yeh, Greg Shakhnarovich

Project Page | arXiv | Code
IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 2023

Generating 3D objects from 2D diffusion models by chaining scores with NeRF gradients.

Text-Free Learning of a Natural Language Interface for Pretrained Face Generators

Xiaodan Du, Raymond A. Yeh, Nicholas Kolkin, Eli Shechtman, Greg Shakhnarovich

arXiv | Code
arXiv, 2022

We propose Fast text2StyleGAN, a natural language interface that adapts pre-trained GANs for text-guided human face synthesis.