Depth–Normals Fusion and Photometric Stereo Errors Self-Correction Using Joint Optimization

We address the problem of depth-normals fusion under conditions where the input depth map contains holes and
the photometric stereo (PS) normals are degraded by modeling
and calibration imperfections. To improve depth inpainting within
these holes—where reconstruction depends entirely on surface
normals—we introduce a joint optimization approach that reconstructs the depth while simultaneously computing a parametric
correction for the surface gradients derived from the PS normals.
This correction is estimated from pixel locations where reliable
depth information is available but is applied globally, ensuring the
gradients used for inpainting inside the holes are also corrected.
We formulate the correction as the sum of a global polynomial
in gradient space and a spatially smooth polynomial in image
coordinates, targeting only the low-frequency component of the
PS gradients. The resulting bi-convex objective function is solved
by alternating between a convex depth reconstruction step and a
closed-form least squares update for the correction parameters.
We demonstrate the performance of our approach both on real
data from our own sensor system and on data derived from the
DiLiGenT-MV dataset.