K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2023, Volume and Issue: unknown, P. 12479 - 12488

Published: June 1, 2023

We introduce k-planes, a white-box model for radiance fields in arbitrary dimensions. Our uses planes to represent d-dimensional scene, providing seamless way go from static (d = 3) dynamic (d= 4) scenes. This planar factorization makes adding dimension-specific priors easy, e.g. temporal smoothness and multi-resolution spatial structure, induces natural decomposition of components scene. use linear feature decoder with learned color basis that yields similar performance as nonlinear black-box MLP decoder. Across range synthetic real, dynamic, fixed varying appearance scenes, k-planes competitive often state-of-the-art recon- struction fidelity low memory usage, achieving 1000x compression over full 4D grid, fast optimization pure PyTorch implementation. For video results code, please see sarafridov.github.io/K-Planes.

Language: Английский

---

HexPlane: A Fast Representation for Dynamic Scenes

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2023, Volume and Issue: unknown

Published: June 1, 2023

Modeling and re-rendering dynamic 3D scenes is a challenging task in vision. Prior approaches build on NeRF rely implicit representations. This slow since it requires many MLP evaluations, constraining real-world applications. We show that can be explicitly represented by six planes of learned features, leading to an elegant solution we call HexPlane. A HexPlane computes features for points spacetime fusing vectors extracted from each plane, which highly efficient. Pairing with tiny regress output colors training via volume rendering gives impressive results novel view synthesis scenes, matching the image quality prior work but reducing time more than 100×. Extensive ablations confirm our design robust different feature fusion mechanisms, coordinate systems, decoding mechanisms. simple effective representing 4D volumes, hope they broadly contribute modeling scenes. ¹ Project page: https://caoang327.github.io/HexPlane.

Language: Английский

---

4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2024, Volume and Issue: 38, P. 20310 - 20320

Published: June 16, 2024

Language: Английский

---

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

2021 IEEE/CVF International Conference on Computer Vision (ICCV), Journal Year: 2023, Volume and Issue: unknown, P. 3272 - 3283

Published: Oct. 1, 2023

Recent methods for neural surface representation and rendering, example NeuS [59], have demonstrated the remarkably high-quality reconstruction of static scenes. However, training takes an extremely long time (8 hours), which makes it almost impossible to apply them dynamic scenes with thousands frames. We propose a fast approach, called NeuS2, achieves two orders magnitude improvement in terms acceleration without compromising quality. To accelerate process, we parameterize by multi-resolution hash encodings present novel lightweight calculation second-order derivatives tailored our networks leverage CUDA parallelism, achieving factor speed up. further stabilize expedite training, progressive learning strategy is proposed optimize from coarse fine. extend method scenes, incremental global transformation prediction component, allow handle challenging sequences large movements deformations. Our experiments on various datasets demonstrate that NeuS2 significantly outperforms state-of-the-arts both accuracy The code available at website: https://vcai.mpi-inf.mpg.de/projects/NeuS2/.

Language: Английский

---

NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields

IEEE Transactions on Visualization and Computer Graphics, Journal Year: 2023, Volume and Issue: 29(5), P. 2732 - 2742

Published: Feb. 22, 2023

Visually exploring in a real-world 4D spatiotemporal space freely VR has been long-term quest. The task is especially appealing when only few or even single RGB cameras are used for capturing the dynamic scene. To this end, we present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. First, propose to decompose according temporal characteristics. Points associated with probabilities belonging three categories: static, deforming, new areas. Each area represented regularized by separate neural field. Second, hybrid representations based feature streaming scheme efficiently modeling fields. Our approach, coined NeRFPlayer, evaluated on scenes captured hand-held multi-camera arrays, achieving comparable superior rendering performance terms quality speed recent state-of-the-art methods, reconstruction 10 seconds per frame interactive Project website: https://bit.ly/nerfplayer.

Language: Английский

---

Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2024, Volume and Issue: 34, P. 20331 - 20341

Published: June 16, 2024

Language: Английский

---

Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2023, Volume and Issue: unknown, P. 16632 - 16642

Published: June 1, 2023

We present Tensor4D, an efficient yet effective approach to dynamic scene modeling. The key of our solution is 4D tensor decomposition method so that the can be directly represented as a spatio-temporal tensor. To tackle accompanying memory issue, we decompose hierarchically by projecting it first into three time-aware volumes and then nine compact feature planes. In this way, spatial information over time simultaneously captured in memory-efficient manner. When applying Tensor4D for reconstruction rendering, further factorize fields different scales sense structural motions detailed changes learned from coarse fine. effectiveness validated on both synthetic real-world scenes. Extensive experiments show able achieve high-quality rendering sparse-view camera rigs or even monocular camera. code dataset will released at https://github.com/DSaurus/Tensor4D.

Language: Английский

---

HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

ACM Transactions on Graphics, Journal Year: 2023, Volume and Issue: 42(4), P. 1 - 12

Published: July 26, 2023

Representing human performance at high-fidelity is an essential building block in diverse applications, such as film production, computer games or videoconferencing. To close the gap to production-level quality, we introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance motion from multi-view video input, and enables playback novel, unseen viewpoints. Our novel acts encoding fine details high compression rates by factorizing space-time into temporal matrix-vector decomposition. This allows us obtain temporally coherent reconstructions of actors for long sequences, while representing high-resolution even context challenging motion. While most research focuses on synthesizing resolutions 4MP lower, address challenge operating 12MP. this end, ActorsHQ, dataset provides 12MP footage 160 cameras 16 sequences with high-fidelity, per-frame mesh reconstructions. We demonstrate challenges emerge using data show our newly introduced HumanRF effectively leverages data, making significant step towards quality view synthesis.

Language: Английский

---

Robust Dynamic Radiance Fields

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2023, Volume and Issue: unknown

Published: June 1, 2023

Dynamic radiance field reconstruction methods aim to model the time-varying structure and appearance of a dynamic scene. Existing methods, however, assume that accurate camera poses can be reliably estimated by Structure from Motion (SfM) algorithms. These thus, are unreliable as SfM algorithms often fail or produce erroneous on challenging videos with highly objects, poorly textured surfaces, rotating motion. We address this robustness issue jointly estimating static fields along parameters (poses focal length). demonstrate our approach via extensive quantitative qualitative experiments. Our results show favorable performance over state-of-the-art view synthesis methods.

Language: Английский

---

Compact 3D Gaussian Representation for Radiance Field

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Journal Year: 2024, Volume and Issue: 41, P. 21719 - 21728

Published: June 16, 2024

Language: Английский

---