NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
  Patch Neighbor Consistency

Valentinos Pariza; Mohammadreza Salehi; Gertjan Burghouts; Francesco Locatello; Yuki M. Asano

返回文献探索

Paper · arXiv 2408.11054

NeCo: Improving DINOv2's spatial representations in 19 GPU hours with Patch Neighbor Consistency

Valentinos Pariza, Mohammadreza Salehi, Gertjan Burghouts, Francesco Locatello, Yuki M. Asano

14 upvotesAugust 20, 2024arXiv 预印本

AI 摘要

NeCo, a novel self-supervised learning method using patch neighbor consistency, improves pretrained representations with minimal computational cost and achieves state-of-the-art results in semantic segmentation.

self-supervised learningpatch neighbor consistencyNeCotraining lossstudent modelteacher modelreference batchesdifferentiable sortingDINOv2dense feature encoderssemantic segmentationADE20kPascal VOCCOCO-ThingsCOCO-Stuff

Abstract

We propose sorting patch representations across views as a novel self-supervised learning signal to improve pretrained representations. To this end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that enforces patch-level nearest neighbor consistency across a student and teacher model, relative to reference batches. Our method leverages a differentiable sorting method applied on top of pretrained representations, such as DINOv2-registers to bootstrap the learning signal and further improve upon them. This dense post-pretraining leads to superior performance across various models and datasets, despite requiring only 19 hours on a single GPU. We demonstrate that this method generates high-quality dense feature encoders and establish several new state-of-the-art results: +5.5% and + 6% for non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and +7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.