HUSH: Holistic Panoramic 3D Scene Understanding using Spherical Harmonics

2025-07-18 16:32:00
vision3d-lab.github.io

TL;DR: HUSH conducts various panorama image-based 3D perception tasks by utilizing task-relevant and geometrically aligned spherical harmonics basis functions for each task.

🌟 Key insight: SH basis functions seem geometrically aligned with the signals (e.g., depth/normal) on the unit sphere!

Abstract

Motivated by the efficiency of spherical harmonics (SH) in representing various physical phenomena, we propose a Holistic panoramic 3D scene Understanding framework using Spherical Harmonics, dubbed as HUSH.
Our approach focuses on a unified framework adaptable to various 3D scene understanding tasks via SH bases.
To achieve this, we first estimate SH coefficients, allowing for the adaptive configuration of the SH bases specific to each scene.
HUSH then employs a hierarchical attention module that uses SH bases as queries to generate comprehensive scene features by integrating these scene-adaptive SH bases with image features.
Additionally, we introduce an SH basis index module that adaptively emphasizes relevant SH bases to produce task-relevant features, enhancing the versatility of HUSH across different scene understanding tasks.
Finally, by combining the scene features with task-relevant features in the task-specific heads, we perform various scene understanding tasks, including depth, surface normal and room layout estimation.
Experiments demonstrate that HUSH achieves state-of-the-art performance on depth estimation benchmarks, highlighting the robustness and scalability of using SH in panoramic 3D scene understanding.

Methods

HUSH first extracts multi-scale image features $f_i$ and scene-wise SH bases via feature extractor and SH coefficient network.
These image features and the scene-wise SH bases are then fed into the SH-based hierarchical attention module and the SH basis index module to estimate comprehensive scene feature $f_S$ and task-relevant features ($f_D$, $f_N$, $f_L$).
Finally, various scene-understanding tasks are performed as these features pass through task-specific heads.

Task-relevant SH bases

To validate the effectiveness of using SH basis functions as queries, we compare the results from conventional learnable (LR) queries and our SH queries.
We visualize the frequently referred queries both on 2D and 3D domains w.r.t target task (depth/normal estimation).
As we can see below, using the SH basis function as a query can keep better geometric consistency of the scene than the LR query.

Visualization on 2D

Visualization on 3D

Results

Depth estimation

Surface normal estimation

Layout estimation

Comparison on 3D

    
      @InProceedings{Lee_2025_CVPR,
        author    = {Lee, Jongsung and Park, Harin and Lee, Byeong-Uk and Joo, Kyungdon},
        title     = {HUSH: Holistic Panoramic 3D Scene Understanding using Spherical Harmonics},
        booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
        month     = {June},
        year      = {2025},
        pages     = {16599-16608}
       }

Source Link

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Unlock unlimited streaming with a free Amazon Prime trial!
Sign up today!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo