VAST-AI Releases HoloPart: Generative 3D Part Amodal Segmentation Technology
The VAST-AI and Hong Kong University joint research team has recently released HoloPart, a new type of 3D part amodal segmentation technology that can decompose 3D models into complete, semantically meaningful parts. This technology not only identifies visible surfaces of the model but can also infer the geometric structure of occluded parts, achieving truly complete part segmentation.
Solving Core Pain Points in 3D Content Creation
When creating 3D content, editing downloaded, scanned, or AI-generated 3D models often presents significant challenges. These models are typically “one solid piece” of geometry, making it extremely difficult to adjust, animate, or apply different materials to individual parts (such as chair legs or eyeglass frames).
While existing 3D part segmentation techniques can identify visible surfaces belonging to different parts, they typically produce fragmented, incomplete pieces, greatly limiting their practical value in content creation.
HoloPart introduces a new task: 3D Part Amodal Segmentation—it not only decomposes 3D shapes into visible surface fragments but into complete parts with semantic information, generating complete components even when portions are occluded.
How HoloPart Works
HoloPart is a new diffusion-based technology inspired by amodal perception (humans’ ability to perceive complete objects even when parts are occluded). The technology is implemented through a two-stage approach:
-
Initial Segmentation: First, initial surface fragments (incomplete parts) are obtained using existing advanced methods (like SAMPart3D).
-
Part Completion: The incomplete part fragment, along with context information from the entire shape, is input into the novel HoloPart model. Based on a Diffusion Transformer architecture, HoloPart can generate complete and reasonable 3D geometry for the part.
HoloPart is built on the TripoSG 3D generation foundation model, developed through extensive pre-training on large datasets (like Objaverse) and specialized fine-tuning on part-whole data, giving it a deep understanding of 3D geometry.
Its key innovation lies in a dual attention mechanism:
- Local Attention: Focuses on the fine geometric details of the input surface fragments, ensuring seamless integration between the completed part and visible geometry.
- Context-Aware Attention: Focuses on the overall shape and the part’s position within it, ensuring that the completed part maintains proportion, semantics, and overall shape consistency.
This enables HoloPart to intelligently reconstruct hidden geometric details, respecting the object’s overall structure even for complex parts or severe occlusions.
Application Scenarios
By generating complete parts, HoloPart unlocks multiple powerful applications:
- Intuitive Editing: Easily grab, scale, move, or replace complete parts
- Convenient Material Assignment: Clearly assign textures or materials to complete components
- Animation-Ready Assets: Generate parts suitable for rigging and animation
- Smart Geometric Processing: Achieve more robust geometric operations such as remeshing through coherent part processing
- Part-Aware Generation: Laying the foundation for future generative models that can create or manipulate 3D shapes at the part level
- Geometric Super-Resolution: Enhancing part details by representing parts with a high number of tokens
Online Demo
You can experience HoloPart’s functionality through this interactive demo: