Jia Gu and Shuying Liu will present their master thesis "LightGlu3D: Feature Matching between 2D and 3D".

Examinator: Kalle Åström
Advisors: Viktor Larsson, Ludvig Dillén & Johanna Lidholm

Abstract. 

This thesis proposes LightGlu3D, a transformer-based direct 2D-3D feature matcher. The query keypoints and features are extracted from SuperPoint. The subset of 3D point clouds where the query is likely to be located is captured by covisibility expansion, and the 3D features are presented by averaged descriptors.

The architecture of LightGlu3D is adpated from LightGlue by doubling self-attention blocks, which include a standard 2D one and a dedicated 3D one with quantile normalization and 3D positional encoding. The redesigned bidirectional cross-attention block applies unshared weights to explicitly bridge geometric difference. LightGlu3D is trained on MegaDepth using a classification strategy withmargin that isolates ambiguous matches into an unsupervised “ignore” label based on reprojection and depth error. Data augment tion techniques are applied to effectively ensure convergence without overfitting.

LightGlu3D achieves high match metrics on MegaDepth, indicating the effectiveness of training. In downstream localization tasks, our model achieves performance superior or comparable to the state-of-the-art HLOC pipeline in standard lighting conditions and urban environments, while a domain sensitivity and degradation issue appears in challenging low-light conditions.

This thesis demonstrates the feasibility of direct 2D-3D feature matching with spatial information, while defining the generalization limitations for future research on real-world localization.