Structure from Motion with a Neural Network

This project delves into the 3D reconstruction of both single and multiple rigid motions, examining the potential of deep learning methods, such as that proposed by Moran et al., to supplant traditional geometry-based approaches. The project is structured into two main parts. In the first part, we focus on the 3D reconstruction of a single rigid motion, building on the work of Moran et al. In addition to using the dataset they used in their paper, we expand it with a new one named BlendedMVS and evaluate the generalization performance of the network on this enriched dataset. The network, in general, performs commendably in both single-scene optimization and multi-view learning settings. Furthermore, by exploring different architectures and enhancing the network, we manage to slightly improve the success rate of single-scene optimization. In the second part, our attention shifts to multiple rigid motions. We initially employ YOLOV7 and Deep Sort for motion segmentation, using a portion of the Hopkins 155 dataset. In total, there are nine video files, each corresponding to a segmentation accuracy. The results reveal five instances of 100% accuracy, with the lowest accuracy standing at 99.55%. In terms of single-scene optimization and multi-view learning settings, there are no failed 3D reconstructions, indicating that all the reprojection errors fall below two pixels.