Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception

2021 IEEE/CVF International Conference on Computer Vision (ICCV), Journal Year: 2023, Volume and Issue: unknown, P. 23326 - 23335

Published: Oct. 1, 2023

Multi-agent collaborative perception as a potential application for vehicle-to-everything communication could significantly improve the performance of autonomous vehicles over single-agent perception. However, several challenges remain in achieving pragmatic information sharing this emerging research. In paper, we propose SCOPE, novel frame-work that aggregates spatio-temporal awareness characteristics across on-road agents an end-to-end manner. Specifically, SCOPE has three distinct strengths: i) it considers effective semantic cues temporal context to enhance current representations target agent; ii) perceptually critical spatial from heterogeneous and overcomes localization errors via multi-scale feature interactions; iii) integrates multi-source agent based on their complementary contributions by adaptive fusion paradigm. To thoroughly evaluate consider both real-world simulated scenarios 3D object detection tasks datasets. Extensive experiments show superiority our approach necessity proposed components. The project link is https://ydk122024.github.io/SCOPE/.

Language: Английский

---

HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative Perception with Vision Transformer

2021 IEEE/CVF International Conference on Computer Vision (ICCV), Journal Year: 2023, Volume and Issue: unknown, P. 284 - 295

Published: Oct. 1, 2023

Vehicle-to-Vehicle technologies have enabled autonomous vehicles to share information see through occlusions, greatly enhancing perception performance. Nevertheless, existing works all focused on homogeneous traffic where are equipped with the same type of sensors, which significantly hampers scale collaboration and benefit cross-modality interactions. In this paper, we investigate multi-agent hetero-modal cooperative problem agents may distinct sensor modalities. We present HM-ViT, first unified framework that can collaboratively predict 3D objects for highly dynamic (V2V) collaborations varying numbers types agents. To effectively fuse features from multi-view images LiDAR point clouds, design a novel heterogeneous graph transformer jointly reason inter-agent intra-agent The extensive experiments V2V dataset OPV2V demonstrate HM-ViT outperforms SOTA methods perception. Our code will be released at https://github.com/XHwind/HM-ViT.

Language: Английский

---

MACP: Efficient Model Adaptation for Cooperative Perception

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Journal Year: 2024, Volume and Issue: unknown, P. 3361 - 3370

Published: Jan. 3, 2024

Vehicle-to-vehicle (V2V) communications have greatly enhanced the perception capabilities of connected and automated vehicles (CAVs) by enabling information sharing to "see through occlusions", resulting in significant performance improvements. However, developing training complex multi-agent models from scratch can be expensive unnecessary when existing single-agent show remarkable generalization capabilities. In this paper, we propose a new framework termed MACP, which equips pre-trained model with cooperation We approach objective identifying key challenges shifting cooperative settings, adapting freezing most its parameters adding few lightweight modules. demonstrate our experiments that proposed effectively utilize observations outperform other state-of-the-art approaches both simulated real-world benchmarks while requiring substantially fewer tunable reduced communication costs. Our ource code is available at https://github.com/PurdueDigitalTwin/MACP.

Language: Английский

---

S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from Simulation to Reality

Published: May 13, 2024

Language: Английский

---

TUMTraf V2X Cooperative Perception Dataset

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2024, Volume and Issue: unknown, P. 22668 - 22677

Published: June 16, 2024

Language: Английский

---

What2comm: Towards Communication-efficient Collaborative Perception via Feature Decoupling

Published: Oct. 26, 2023

Multi-agent collaborative perception has received increasing attention recently as an emerging application in driving scenarios. Despite advancements previous approaches, challenges remain due to redundant communication patterns and vulnerable collaboration processes. To address these issues, we propose What2comm, end-to-end framework achieve a trade-off between performance bandwidth. Our novelties lie three aspects. First, design efficient mechanism based on feature decoupling transmit exclusive common maps among heterogeneous agents provide perceptually holistic messages. Secondly, spatio-temporal module is introduced integrate complementary information from collaborators temporal ego cues, leading robust procedure against transmission delay localization errors. Ultimately, common-aware fusion strategy refine final representations with informative features. Comprehensive experiments real-world simulated scenarios demonstrate the effectiveness of What2comm.

Language: Английский

---

FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems

IEEE Transactions on Intelligent Vehicles, Journal Year: 2023, Volume and Issue: 9(1), P. 958 - 969

Published: Aug. 31, 2023

Bird's eye view (BEV) perception is becoming increasingly important in the field of autonomous driving. It uses multi-view camera data to learn a transformer model that directly projects road environment onto BEV perspective. However, training often requires large amount data, and as for traffic are private, they typically not shared. Federated learning offers solution enables clients collaborate train models without exchanging but parameters. In this paper, we introduce FedBEVT, federated approach perception. order address two common heterogeneity issues FedBEVT: (i) diverse sensor poses, (ii) varying numbers systems, propose approaches - Learning with Camera-Attentive Personalization (FedCaP) Adaptive Multi-Camera Masking (AMCM), respectively. To evaluate our method real-world settings, create dataset consisting four typical use cases. Our findings suggest FedBEVT outperforms baseline all cases, demonstrating potential improving

Language: Английский

---

Multi-Vehicle Cooperative Simultaneous LiDAR SLAM and Object Tracking in Dynamic Environments

IEEE Transactions on Intelligent Transportation Systems, Journal Year: 2024, Volume and Issue: 25(9), P. 11411 - 11421

Published: Sept. 1, 2024

Simultaneous localization and mapping (SLAM) moving object detection tracking (MODT) are two fundamental problems for autonomous driving systems. Multi-vehicle cooperative SLAM perception, which take advantage of multi-vehicle information sharing, can overcome inherent limitations single vehicle such as view occlusion. Solutions to MODT usually rely on certain assumptions, the static environment assumption accurate ego-vehicle pose MODT. However, it is difficult or even impossible have these assumptions hold in complex dynamic environments. We propose a LiDAR-based coupled simultaneous (C-SLAMMODT) strategy, not only handles environments but also overcomes perception. The proposed C-SLAMMODT outperforms both This method includes module that augment estimation by shared from neighbouring vehicles, applies state-of-the-art adaptive feature-level fusion model fuse data, improving precision overcoming perception occlusion situations. Furthermore, unified factor graph optimization integrates obtained states, neighbor-vehicle states realize tracking. Various comparative experiments demonstrate performance advantages solution terms accuracy robustness.

Language: Английский

---

Image Analysis in Autonomous Vehicles: A Review of the Latest AI Solutions and Their Comparison

Published: July 24, 2024

The integration of advanced image analysis using artificial intelligence (AI) is pivotal for the evolution autonomous vehicles (AVs). This article provides a thorough review most significant datasets and latest state-of-the-art AI solutions employed in AVs. Datasets such as Cityscapes, NuScenes, CARLA form benchmarks training evaluating different models, with unique characteristics catering to various aspects driving. Key methodologies, including Convolutional Neural Networks (CNNs), Recurrent (RNNs), Transformer Generative Adversarial (GANs), are discussed. also presents comparative techniques real-world scenarios, focusing on semantic segmentation, 3D object detection, vehicle control virtual environments. Simultaneously, role multisensor simulation platforms like AirSim, TORCS, SUMMIT enriching data testing environments AVs highlighted. By synthesizing information datasets, solutions, performance evaluations, serves crucial resource researchers, developers, industry stakeholders. Offering clear view current landscape future directions technologies.

Language: Английский

---

BEV perception for autonomous driving: State of the art and future perspectives

Expert Systems with Applications, Journal Year: 2024, Volume and Issue: 258, P. 125103 - 125103

Published: Aug. 24, 2024

Language: Английский

---