Publications

2024

1. CVPR

   LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

   Yunsheng Ma*, Can Cui*, Xu Cao*, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, James M. Rehg, and Ziran Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024

   arXiv
2. CVPR

   MAPLM: A Real-World Large-Scale Vision-Language Dataset for Map and Traffic Scene Understanding

   Xu Cao*, Tong Zhou*,  Yunsheng Ma*, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M. Rehg, and Chao Zheng

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024

   Abs

   Vision-language generative AI has demonstrated remarkable promise for empowering cross-modal scene understanding of autonomous driving and high-definition (HD) map systems. However, current benchmark datasets lack multi-modal point cloud, image, and language data pairs. Recent approaches utilize visual instruction learning and cross-modal prompt engineering to expand vision-language models into this domain. In this paper, we propose a new vision-language benchmark that can be used to finetune traffic and HD map domain-specific foundation models. Specifically, we annotate and leverage large-scale, broad-coverage traffic and map data extracted from huge HD map annotations, and use CLIP and LLaMA-2 / Vicuna to finetune a baseline model with instruction-following data. Our experimental results across various algorithms reveal that while visual instruction-tuning large language models (LLMs) can effectively learn meaningful representations from MAPLM-QA, there remains significant room for further advancements.

3. CVPR

   Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture

   Juanwu Lu*, Can Cui*,  Yunsheng Ma, Aniket Bera, and Ziran Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024

   arXiv
4. ITSM

   Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

   Can Cui,  Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang

   IEEE Intelligent Transportation Systems Magazine (ITSM) , 2024
5. WACV

   MACP: Efficient Model Adaptation for Cooperative Perception

   Yunsheng Ma*, Juanwu Lu*, Can Cui, Sicheng Zhao, Xu Cao, Wenqian Ye, and Ziran Wang

   In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2024

   Abs arXiv Code Website

   Vehicle-to-vehicle (V2V) communications have greatly enhanced the perception capabilities of connected and automated vehicles (CAVs) by enabling information sharing to “see through the occlusions", resulting in significant performance improvements. However, developing and training complex multi-agent perception models from scratch can be expensive and unnecessary when existing single-agent models show remarkable generalization capabilities. In this paper, we propose a new framework termed MACP, which equips a single-agent pre-trained model with cooperation capabilities. We approach this objective by identifying the key challenges of shifting from single-agent to cooperative settings, adapting the model by freezing most of its parameters and adding a few lightweight modules. We demonstrate in our experiments that the proposed framework can effectively utilize cooperative observations and outperform other state-of-the-art approaches in both simulated and real-world cooperative perception benchmarks while requiring substantially fewer tunable parameters with reduced communication costs.

6. T-IV

   Driver Digital Twin for Online Recognition of Distracted Driving Behaviors

   Yunsheng Ma, Runjia Du, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, and Ziran Wang

   IEEE Transactions on Intelligent Vehicles (T-IV) , 2024

   Website

2023

1. T-IV

   REDFormer: Radar Enlightens the Darkness of Camera Perception With Transformers

   Can Cui,  Yunsheng Ma, Juanwu Lu, and Ziran Wang

   IEEE Transactions on Intelligent Vehicles (T-IV) , 2023
2. UAI

   Mitigating Transformer Overconfidence via Lipschitz Regularization

   Wenqian Ye,  Yunsheng Ma, Xu Cao, and Kun Tang

   In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI) , 2023

   Abs arXiv

   Though Transformers have achieved promising results in many computer vision tasks, they tend to be over-confident in predictions, as the standard Dot Product Self-Attention (DPSA) can barely preserve distance for the unbounded input domain. In this work, we fill this gap by proposing a novel Lipschitz Regularized Transformer (LRFormer). Specifically, we present a new similarity function with the distance within Banach Space to ensure the Lipschitzness and also regularize the term by a contractive Lipschitz Bound. The proposed method is analyzed with a theoretical guarantee, providing a rigorous basis for its effectiveness and reliability. Extensive experiments conducted on standard vision benchmarks demonstrate that our method outperforms state-of-the-art single forward pass approaches in prediction, calibration, and uncertainty estimation.

3. ITSC

   CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers

   Yunsheng Ma, Wenqian Ye, Xu Cao, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, and Ziran Wang

   In IEEE International Conference on Intelligent Transportation Systems (ITSC) , 2023

   arXiv
4. CVPRW

   M2DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer

   Yunsheng Ma, Liangqi Yuan, Amr Abdelraouf, Kyungtae Han, Rohit Gupta, Zihao Li, and Ziran Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , 2023
   Outstanding Speaker Award at NGTS [News]

   arXiv Code
5. CVPRW

   Peer-to-Peer Federated Continual Learning for Naturalistic Driving Action Recognition

   Liangqi Yuan,  Yunsheng Ma, Lu Su, and Ziran Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , 2023

   arXiv

2020

1. AAAI Oral

   An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

   Sicheng Zhao*,  Yunsheng Ma*, Yang Gu, Jufeng Yang, Tengfei Xing, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI Oral) , 2020

   arXiv Code Website