A Survey on Multimodal Large Language Models for Autonomous Driving

Published: Jan. 1, 2024

With the emergence of Large Language Models (LLMs) and Vision Foundation (VFMs), multimodal AI systems benefiting from large models have potential to equally perceive real world, make decisions, control tools as humans. In recent months, LLMs shown widespread attention in autonomous driving map systems. Despite its immense potential, there is still a lack comprehensive understanding key challenges, opportunities, future endeavors apply LLM this paper, we present systematic investigation field. We first introduce background Multimodal (MLLMs), development using LLMs, history driving. Then, overview existing MLLM for driving, transportation, together with datasets benchmarks. Moreover, summarized works The 1st WACV Workshop on Autonomous Driving (LLVM-AD), which workshop kind regarding To further promote field, also discuss several important problems MLLMs that need be solved by both academia industry.

Language: Английский

---

A Systematic Survey of Control Techniques and Applications in Connected and Automated Vehicles

IEEE Internet of Things Journal, Journal Year: 2023, Volume and Issue: 10(24), P. 21892 - 21916

Published: Aug. 21, 2023

Vehicle control is one of the most critical challenges in autonomous vehicles (AVs) and connected automated (CAVs), it paramount vehicle safety, passenger comfort, transportation efficiency, energy saving. This survey attempts to provide a comprehensive thorough overview current state technology, focusing on evolution from estimation trajectory tracking AVs at microscopic level collaborative CAVs macroscopic level. First, this review starts with key estimation, specifically sideslip angle, which pivotal for control, discuss representative approaches. Then, we present symbolic approaches AVs. On top that, further frameworks corresponding applications. Finally, concludes discussion future research directions challenges. aims contextualized in-depth look art CAVs, identifying areas focus pointing out potential exploration.

Language: Английский

---

Collaborative Perception in Autonomous Driving: Methods, Datasets, and Challenges

IEEE Intelligent Transportation Systems Magazine, Journal Year: 2023, Volume and Issue: 15(6), P. 131 - 151

Published: Sept. 12, 2023

Collaborative perception is essential to address occlusion and sensor failure issues in autonomous driving. In recent years, theoretical experimental investigations of novel works for collaborative have increased tremendously. So far, however, few reviews focused on systematical collaboration modules large-scale datasets. This article achievements this field bridge gap motivate future research. We start with a brief overview schemes. After that, we systematically summarize the methods ideal scenarios real-world issues. The former focuses efficiency, latter devoted addressing problems actual application. Furthermore, present public datasets quantitative results these benchmarks. Finally, highlight gaps overlooked challenges between current academic research applications.

Language: Английский

---

FusionPlanner: A multi-task motion planner for mining trucks via multi-sensor fusion

Mechanical Systems and Signal Processing, Journal Year: 2024, Volume and Issue: 208, P. 111051 - 111051

Published: Jan. 3, 2024

Language: Английский

---

Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges

IEEE Transactions on Intelligent Vehicles, Journal Year: 2023, Volume and Issue: 9(1), P. 119 - 137

Published: Nov. 14, 2023

Machine learning (ML) is widely used for key tasks in Connected and Automated Vehicles (CAV), including perception, planning, control. However, its reliance on vehicular data model training presents significant challenges related to in-vehicle user privacy communication overhead generated by massive volumes. Federated (FL) a decentralized ML approach that enables multiple vehicles collaboratively develop models, broadening from various driving environments, enhancing overall performance, simultaneously securing local vehicle security. This survey paper review of the advancements made application FL CAV (FL4CAV). First, centralized frameworks are analyzed, highlighting their characteristics methodologies. Second, diverse sources, security techniques relevant CAVs reviewed, emphasizing significance ensuring confidentiality. Third, specific applications explored, providing insight into base models datasets employed each application. Finally, existing FL4CAV listed potential directions future investigation further enhance effectiveness efficiency context discussed.

Language: Английский

---

Multi-Sensor Fusion and Cooperative Perception for Autonomous Driving: A Review

IEEE Intelligent Transportation Systems Magazine, Journal Year: 2023, Volume and Issue: 15(5), P. 36 - 58

Published: Aug. 4, 2023

Autonomous driving (AD), including single-vehicle intelligent AD and vehicle–infrastructure cooperative AD, has become a current research hot spot in academia industry, multi-sensor fusion is fundamental task for system perception. However, the process faces problem of differences type dimensionality sensory data acquired using different sensors (cameras, lidar, millimeter-wave radar, so on) as well performance environmental perception caused by strategies. In this article, we study multiple papers on field address that category division not detailed clear enough more subjective, which makes classification strategies differ significantly among similar algorithms. We innovatively propose taxonomy, divides into two categories—symmetric asymmetric fusion—and seven subcategories strategy combinations, such data, features, results. addition, reliability limited its insufficient environment capability robustness data-driven methods dealing with extreme situations (e.g., blind areas). This article also summarizes innovative applications

Language: Английский

---

Vehicle-to-everything (V2X) in the autonomous vehicles domain – A technical review of communication, sensor, and AI technologies for road user safety

Transportation Research Interdisciplinary Perspectives, Journal Year: 2023, Volume and Issue: 23, P. 100980 - 100980

Published: Dec. 15, 2023

Autonomous vehicles (AV) are rapidly becoming integrated into everyday life, with several countries anticipating their inclusion in public transport networks the coming years. Safety measures context of Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication have been extensively investigated. However, ensuring safety for Vulnerable Road Users (VRUs) such as pedestrians, cyclists, e-scooter riders remains an area that requires more focused research effort. The existing AV sensor suites offer diverse capabilities, covering blind spots, longer ranges, resilience to weather conditions , benefiting V2V V2I scenarios. Nevertheless, predominant emphasis has on communicating identifying other vehicles, leveraging advanced infrastructure efficient status information exchange. identification VRUs introduces challenges localization difficulties, limitations, a lack network coverage. This review critically assesses state-of-the-art domains V2X technologies, aiming enhance identification, tracking, VRUs. Additionally, it proposes end-to-end autonomous vehicle motion control architecture based temporal deep learning algorithm. algorithm incorporates dynamic behaviors both visible non-line-of-sight (NLOS) road users. work also provides critical evaluation various AI technologies improve VRU message sharing, tracking domains.

Language: Английский

---

Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments

2022 International Joint Conference on Neural Networks (IJCNN), Journal Year: 2023, Volume and Issue: unknown, P. 1 - 10

Published: June 18, 2023

In federated learning, all networked clients contribute to the model training cooperatively. However, with sizes increasing, even sharing trained partial models often leads severe communication bottlenecks in underlying networks, especially when communicated iteratively. this paper, we introduce a learning framework FedD3 requiring only one-shot by integrating dataset distillation instances. Instead of updates other approaches, allows connected distill local datasets independently, and then aggregates those decentralized distilled (e.g. few unrecognizable images) from networks for training. Our experimental results show that significantly outperforms frameworks terms needed volumes, while it provides additional benefit be able balance trade-off between accuracy cost, depending on usage scenario or target dataset. For instance, an AlexNet CIFAR-10 10 under non-independent identically distributed (Non-IID) setting, can either increase over 71% similar volume, save 98% reaching same accuracy, compared approaches.

Language: Английский

---

Sora-Based Parallel Vision for Smart Sensing of Intelligent Vehicles: From Foundation Models to Foundation Intelligence

IEEE Transactions on Intelligent Vehicles, Journal Year: 2024, Volume and Issue: 9(2), P. 3123 - 3126

Published: Feb. 1, 2024

installed on the modern intelligent vehicles. Many Artificial Intelligence based foundation models have been proposed for smart sensing to recognize known object classes in new but similar scenarios. However, it is still challenging of detect all both seen and unseen This letter aims at pushing boundary research We first summarize current widely-used intelligence needed then explain Sora-based Parallel Vision boost from basic (1.0) enhanced (2.0) final generalized (3.0). Several representative case studies are discussed show potential usages Vision, followed by its future direction.

Language: Английский

---

Towards the Next Level of Vehicle Automation Through Cooperative Driving: A Roadmap From Planning and Control Perspective

IEEE Transactions on Intelligent Vehicles, Journal Year: 2024, Volume and Issue: 9(3), P. 4335 - 4347

Published: Feb. 8, 2024

Cooperative Driving Automation (CDA) stands at the forefront of evolving landscape vehicle automation, elevating driving capabilities within intricate real-world environments. This research aims to navigate path toward future CDA by offering a thorough examination from perspective Planning and Control (PnC). It classifies state-of-the-art literature according classes defined Society Automotive Engineers (SAE). The strengths, weaknesses, requirements PnC for each class are analyzed. analysis helps identify areas that need improvement provides insights into potential directions. further discusses evolution directions CDA, providing valuable enhancement enrichment research. suggested include: robustness against disturbance; Risk-aware planning in mixed environment Connected Automated Vehicles (CAVs) Human-driven (HVs); Vehicle-signal coupled modeling coordination enhancement; Vehicle grouping enhance mobility platooning.

Language: Английский

---