Real-time surfaces reconstruction using coded structured light

Partners

ICube, Region Alsace, Karl Storz Company, IRCAD.

Description

In this project, the robust matching technique based on coded structured lighting (SL) is addressed to achieve real-time surface reconstructions, like for a stero-like vision system. To that purpose, most existing approaches involve color or grey levels coding (Figure 1) but they are well-known to be sensitive to spectral properties and texture of the viewed surfaces.

Figure 1 : A 5-ary, (14 x 14) PSM with a min Hamming distance of 5 between all the (3 x 3) neighbourhoods, represented with a) digits, b) shapes, c) grey levels, d) colors.

Therefore, the overall robustness of the proposed technique comes first from the geometrical features used in cunjonction with the SL neighbourhood scheme to carry out pattern coding. Second, a desired minimum Hamming distance between features' codewords drives the pattern design. This is suited for autonomous navigation in unknown environment as this parameter enables impressive codewords correction capabilities during the decoding stage and a new pattern can be produced in (video) real-time.

Furthermore, we take advantage of the known epipolar geometry (Figure 2) by including projective invariants between corresponding epipolar lines directly in the pattern components. Thus, the final pattern displays relocated and reoriented (cuneiform) features along the epipolar lines (Figure 3).

Figure 2 :Experimental validation by a projection of epipolar lines. In the general case lines projection on the IPC depends on the scene (left).But the epipolar lines homography is independent of the scene (right).

Figure 3 : Features orientation is controlled independently to the scene variations. (left) Projection of cuneiform features along the epipolar lines (in red). (middle) Rectification 1: epipolar lines are now horizontal. (right) Rectification 2: horizontal distortion due to rectification 1 is corrected.

This grouping also contribute to reduce the search space more and results in a significant less constrained coding. This technique provides very big patterns (100 x 150) with high correction capabilities (Hmin > 1). For practical considerations, each numerical codeword symbol is associated to a unique visual feature embedding the local orientation of the pattern, which is helpful for the neighbourhood retrieval during the decoding process. Finally, we show how this coding and this non-grid based pattern offer efficient and fast correction of mislabeled features due to blurring, spectral harmful effects and surfaces discontinuities prior to the 3-D reconstruction of real scenes (Figures 4 and 5).

Figure 4 : projection of epipolar-based pattern on a surfaces with specularities and texture changes

Figure 5 : surfaces reconstruction with only one shot and rendrings (shaded)

In vivo experiments at IRCAD France

Together with the endoscopic device, in vivo real-time reconstructions (in mini-invasive surgical conditions) have been performed in the surgery room of IRCAD, Strasbourg. Renderings are displayed on Figures 6-a and 6-b.

This allows to assess both the efficiency of the proposed pattern design, the decoding process with the 3-D laparoscope setup realized in the lab.

Figure 6 : Reconstructions and rendering (shaded) from in vivo experiments at IRCAD Strasbourg,with one-trocard based 3D active laparoscope with coded strcutre light (on board video-projector).

Figure 6 : Reconstructions and rendering (texture mapping) from in vivo experiments at IRCAD Strasbourg,with one-trocard based 3D active laparoscope with coded strcutre light (on board video-projector).

Keywords

Perfect map theory (PSM),
Pattern design,
Information theory,
Epipolar lines invariance,
Real-Time Segmentation,
Surfaces reconstruction

Alumni

Chadi Albitar (2005-2009)
Xavier Maurice (2008-2012)

Contact

Christophe Doignon
Professeur, Télécom Physique Strasbourg
[1]