OpticFusion: Fusing White Light Interferometry and Optical Microscopy for 3D Color Reconstruction of Microstructures

In a significant advancement in the field of computer vision and microscopy, researchers from Zhejiang University have introduced OpticFusion, a novel multi-modal neural implicit method that combines White Light Interferometry (WLI) and Optical Microscopy (OM) to achieve detailed 3D reconstructions with natural color textures. This work addresses a longstanding limitation in WLI, which traditionally provides precise 3D topography but lacks the ability to capture surface colors.

Technical Overview

OpticFusion leverages the strengths of both WLI and OM to produce high-quality 3D reconstructions that include color information. Here’s a breakdown of how it works:

• Data Acquisition:

  • WLI Data: Captures precise 3D topography with sub-micron resolution.
  • OM Data: Provides natural color textures at the microscale.

• Two-Step Data Association Process:

  • Initial Pose Estimation: Uses a coarse-to-fine approach to estimate the initial poses of WLI and OM data. This involves aligning the images based on feature matching and iterative refinement.
  • Refined Pose Optimization: Applies bundle adjustment techniques to refine the poses, ensuring accurate alignment between the two modalities.

• Neural Implicit Representation:

  • The method employs a neural implicit function to represent the 3D geometry. This allows for continuous and differentiable representation of the microstructure, which is crucial for high-fidelity reconstructions.
  • The neural network is trained on multi-view WLI and OM images, learning to fuse the geometric and color information effectively.

• Color Decomposition:

  • To extract natural colors from the OM data, OpticFusion uses a color decomposition technique. This involves separating the color channels and integrating them with the 3D geometry to produce a colored 3D model.

Implementation Details

• Dataset:

  • The researchers created a multi-modal dataset consisting of various microscale samples, including biological tissues and engineered materials.
  • Each sample was captured using both WLI and OM, providing a rich set of data for training and evaluation.

• Benchmarks:

  • OpticFusion was tested on this dataset, demonstrating its ability to produce detailed 3D reconstructions with accurate color textures.
  • The method outperformed existing single-modal approaches in terms of both geometric accuracy and color fidelity.

Why It Matters

For practitioners in fields such as materials science, biology, and microelectronics, the ability to capture both high-precision 3D geometry and natural colors is invaluable. Traditional WLI systems have been limited by their inability to provide color information, which can be crucial for understanding surface properties and material composition.

OpticFusion offers a cost-effective solution by leveraging existing hardware and advanced computer vision techniques. This approach not only enhances the quality of 3D reconstructions but also opens up new possibilities for microscale research and applications.

Conclusion

The introduction of OpticFusion marks a significant step forward in multi-modal 3D reconstruction. By fusing WLI and OM data, this method provides a powerful tool for researchers and practitioners who need detailed, color-accurate 3D models of microstructures. The open-source nature of the project further encourages collaboration and innovation in the field.