A Novel method for open-vocabulary panoptic segmentation

Kormushev, Nikolay

A Novel method for open-vocabulary panoptic segmentation

License

Citation

Kormushev, N. A Novel method for open-vocabulary panoptic segmentation. 2025. handle: http://hdl.handle.net/10230/71838

This citation was generated automatically.

Abstract

Open-vocabulary panoptic segmentation aims to segment and classify visual content into both known and unseen categories using natural language supervision. While class-agnostic mask generators produce reasonably high quality masks, this thesis identifies two main bottlenecks limiting performance: mask quality assessment, where valid masks are often mistakenly discarded as background, and semantic classification, which remains challenging especially for unseen categories. To address these, we propose a two part solution: a novel background mask reclassification module that recovers valid masks misclassified as background, and a CLIP fine-tuning strategy that preserves alignment between visual and textual embeddings. Together, these methods improve panoptic quality (PQ) on ADE20K from a baseline of 26.6 to 27.94, with analysis showing that addressing errors in mask quality and semantic classification could theoretically increase PQ to 65.9. These findings offer practical advancements and valuable insights toward bridging the gap between open- and closed-vocabulary segmentation.