SenseNova-Vision: A Unified Multimodal Model for All Vision Tasks

SenseTime (商汤科技) has open-sourced SenseNova-Vision-7B-MoT, a unified multimodal model (7B parameters) that reformulates heterogeneous computer vision tasks — detection, segmentation, depth estimation, surface normal prediction, OCR, keypoints, and multi-view geometry — as native text, image, or mixed text-image generation, eliminating the need for task-specific heads or decoders.

The model is released under CC BY-NC 4.0 on both Hugging Face and ModelScope, accompanied by the SenseNova-Vision-Corpus-50M dataset and a detailed arXiv paper.

What Makes SenseNova-Vision Different

Traditional computer vision systems bolt on separate prediction heads for each task — one for detection, another for segmentation, yet another for depth. SenseNova-Vision takes a radically different approach by casting all vision tasks into the native input-output spaces of a unified multimodal model:

1. Unified task formulation — Detection, segmentation, depth, normal, OCR, GUI grounding, keypoints, camera pose estimation, and multi-view point map reconstruction are all expressed through natural-language instructions and optional visual prompts, with outputs generated as text records, dense image maps, or mixed text-image responses.

2. No task-specific heads — The model relies on a single architecture without separate detection, segmentation, depth, or geometry heads. Outputs are decodable back into benchmark-compatible formats (bounding boxes, masks, depth maps, normal maps, point maps, camera parameters).

3. Broad task coverage — The same model handles structured visual understanding (detection, referring localization, OCR, GUI grounding, keypoints), dense geometric prediction (depth, surface normals), segmentation (semantic, referring, reasoning, interactive, grounded conversation), and multi-view visual geometry (point map reconstruction, camera pose estimation).

4. Language-defined task variants — Natural-language instructions enable flexible task definitions beyond fixed benchmark schemas, allowing users to define custom task variants on the fly.

Architecture & Availability

Quick Start

git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
cd SenseNova-Vision
bash setup.sh sensenova-vision
conda activate sensenova-vision

Download the model weights:

from huggingface_hub import snapshot_download
model_path = snapshot_download("sensenova/SenseNova-Vision-7B-MoT")

Run the curated example to verify setup:

bash scripts/run_sensenova_vision.sh example

Run a single inference request — for example, binary segmentation for "person":

bash scripts/run_sensenova_vision.sh inference binary_seg "person" examples/images/2.jpg

Launch the Gradio web demo:

bash scripts/run_sensenova_vision.sh web_demo

Benchmarking Highlights

SenseNova-Vision was evaluated across four task families and achieves state-of-the-art or competitive results against both specialist and generalist models:

• Object Detection (COCO): 56.6 mAP — tied with Grounding DINO-Swin-T and surpassing Bagel, Qwen3-VL-8B, Rex-Omni, and LocateAnything
• Referring Detection (RefCOCOg): 80.5 Acc@0.5 — best among all compared methods
• Dense Object Detection (Dense200): 66.8 mAP — significantly outperforming all competitors
• OCR (HierText): 31.2 bbox — best among generalist models
• GUI Grounding (ScreenSpot-V2): 49.5 bbox — best among all methods
• Keypoint (COCO-Kpt): 34.6 point Acc@0.5 — best among all comparable methods
• Depth (NYUv2): 0.040 AbsRel / 98.1 δ1 — competitive with DepthAnything and FE2E
• Surface Normal (ScanNet): 12.8 Mean / 68.9 11.25° — best among all methods including DSINE
• Segmentation (ReasonSeg): 63.2 / 60.7 gIoU — best among compared methods
• Multi-View Camera Pose (CO3Dv2): 97.4 RRA@30 / 95.4 RTA@30 / 80.1 AUC@30 — best among unified models

Links

• GitHub Repository
• arXiv Paper
• Hugging Face Model
• ModelScope Model
• Training Dataset