Occlusion in Augmented Reality

1.Occlusion in Augmented Reality

笔记来源：
1.Occlusion handling in Augmented Reality context
2.Occlusion in Augmented Reality
3.Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach
4.Occlusion Matting: Realistic Occlusion Handling for Augmented Reality Applications

声明：本篇内容大部分来自上述论文，仅做学习使用

1.1 遮挡问题

如果遮挡处理不当，遮挡对象的可视化会导致用户对空间属性的误解，在极端情况下，可能会破坏沉浸式AR体验

本篇论文目标是在未知环境中仅仅使用RGB-D相机处理AR背景下的遮挡问题，更具体的说是聚焦于解决真实物体遮挡了虚拟物体的这种情况（较难解决的情况），不包括虚拟物体遮挡真实物体等其他情况。
对于当虚拟物体遮挡真实物体这种情况，为了处理遮挡，增加虚拟物体的亮度就足够了。
对于真实物体遮挡了虚拟物体这种情况，则需要使用图像处理方法来解决这个问题。更详细地说，在该方法中，应用图像处理方法来分析和合并来自深度和RGB传感器的数据。

1.2 遮挡问题的类型

（1）真实环境中物体之间的遮挡（单单只发生在真实环境中）
（2）虚拟物体之间的遮挡（单单只发生在虚拟环境中）
（3）虚拟物体和真实环境中物体之间的遮挡（发生在虚实共存的人为创造的环境中）

1.3 相关方法

1.3.1 Object-based method

来自论文：Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach
使用物体的轮廓处理遮挡问题
这种方法的前提假设是静态场景，在该场景中真实物体和虚拟物体的遮挡关系是固定的。
处理步骤：
（1）遮挡物体的选择：用户使用交互式分割方法选择遮挡对象
（2）物体跟踪：在后续帧中实时跟踪所选对象轮廓
（3）遮挡处理：将被跟踪物体上的所有像素重新绘制到未处理的增强图像上，生成真实和虚拟物体之间相对位置正确的新合成图像。

1.3.2 Model-based Method

对于简单的静态场景，可以直接对该场景进行三维重建，从而把真实场景转换为虚拟场景，相当于把虚-实遮挡问题转变为了虚-虚遮挡问题
对于没有任何先验的未知静态环境，可以使用实时稠密三维重建来解决虚实遮挡问题，但对于高动态场景则效果并不理想，
如果重建无法还原真实细节，则渲染遮挡的质量也会大幅下降

1.3.3 Depth-based Method

什么是trimap？
A trimap is a type of image used in image processing, particularly in tasks like image matting and segmentation. It helps to delineate different regions of an image, typically into three categories:

Foreground: The object of interest.
Background: The area behind the object of interest.
Unknown or Transition Area: The region where the object blends with the background and is not easily classified as either foreground or background.

来自论文：Occlusion Matting: Realistic Occlusion Handling for Augmented Reality Applications

3D Rendering

目的： 3D 渲染阶段旨在使用颜色 (RGB) 和深度信息创建虚拟场景的视觉表示。这一步至关重要，因为它为后续阶段将虚拟元素与现实世界场景融合提供了基础。

Input: The 3D Rendering stage receives an RGB image and a depth map as inputs.
Output: The output includes a rendered RGB texture (showing color details) and a rendered depth texture (showing depth values).

场景的深度和场景纹理的深度是同一个概念，只是在计算机图形学二者的目的不同，前者为了捕捉场景的结构，后者为了处理纹理效果
对比 depth map 和 depth texture

Adaptive Trimap Generation

目的：将图像的像素分类为三个区域：foreground、background、unknown。trimap 有助于指导后续阶段，以实现准确的前景和背景传播以及 alpha 估计。
Input:
Rendered Depth Map: Depth information of the virtual scene generated in the 3D rendering stage.
Depth Map from Sensor: Depth information captured by the sensor, indicating distances of real-world objects from the camera.
Color Image (RGB): Color data captured by the sensor, providing visual details of the real-world scene.
Output:
Trimap: An image with three regions:
（1）White (Foreground): Pixels definitely belonging to the foreground.
（2）Black (Background): Pixels definitely belonging to the background.
（3）Gray (Unknown): Pixels whose classification is uncertain.

详细步骤
假设有一个虚拟平面（蓝色正方形）在真实物体（椅子）的背面
（1）深度图含噪声，我们首先使用low pass filter进行平滑去噪
（2）由真实物体的深度图经过coarse segmentation得到coarse trimap
（3）对区域U中的edge进行label，标注其属于前半部分还是后半部分，也就是属于前景F还是背景B
（4）对区域U进行固定大小的膨胀

coarse segmentation具体细节
该过程具体包括：depth map经过depth test 分割出前景图（F）和背景图（B），在有效区域内（虚实共存的区域内）利用3×3 sobel kernel进行卷积得到未知区域（U）

label的具体细节
图（a）是粗分割深度图覆盖到了RGB物体边缘图上的效果，以RGB物体边缘为界，决定未知区域属于前景还是背景
RGB中的物体边界（红色）、由深度图粗分割出来的前景图（F，白色）背景图（B，黑色）和未知区域（U，蓝色）
现在看未知区域到底属于前景还背景，根据下图一目了然，
例如橙黄色区域（b）中，U位于红色边界的后面，也就是说这一小段未知区域属于背景
例如浅蓝色区域（c）中，U位于红色边界的前面，也就是说这一小段未知区域属于前景

dilation的具体细节
检查在前景和背景中的每个像素k，在该像素周围的一个小窗口内是否有在未知区域的像素i，如果有，则将该像素i标记为未知像素并且标记该像素属于前景还是背景中
为什么需要膨胀处理？
未知区域（蓝色）位于深度图像的边界（黑白交界）周围，而不一定位于彩色图像的边界（红色）周围。这对alpha估计有很大的影响，因为一些背景区域被错误地当作前景，反之亦然。为了克服这些问题，我们向彩色图像的边界扩大未知区域。然而，大的膨胀意味着大的未知区域和颜色边界可能被覆盖，但这也意味着已知的前景区域可能会缩小。要保证在解决以上问题的同时保证未知区域尽可能的小。

adaptive dilation
不同形态的边界需要不同的膨胀量
膨胀的量取决于在未知区域中被标记为no edge的点的个数，如果这个数量超过给定阈值，则增加膨胀量

Foreground and Background Propagation

目的：将前景和背景的已知区域扩展到trimap中识别的未知区域。此步骤有助于细化像素的分类（该未知区域内像素到底归属于前景还是背景）使后续的 alpha 估计更加准确可靠。
Input:
Trimap: An image from the Adaptive Trimap Generation stage, with pixels labeled as foreground, background, or unknown.
Color Image (RGB): The color data from the sensor, providing visual details of the real-world scene.
Output:
Propagated Foreground Image: An image where the foreground regions have been extended into the unknown areas.
Propagated Background Image: An image where the background regions have been extended into the unknown areas.

复制前景图片（未知区域透明度设置为0，其他设置为1），自底向上创建该前景图片的金字塔，而后自顶向下开始进行模糊处理，到最底层时将未知区域便有了上层模糊处理过的包含前景的颜色，进行该项操作的次数，论文中成文diffusion次数。同样对背景图片进行此操作。

Algorithm 1: Foreground Propagation
Input: Foreground image F, levels l, iterations i
Output: Propagated foreground image F

1: S = copy(F)  // Step 1
2: Create pyramid with l levels:
3: for each level from finest to coarsest:
4:   Initialize finest level with known foreground (α = 1)
5:   Set unknown pixels α = 0
6:   for each coarser level:
7:     Apply Gaussian filter to previous finer level
8:     Store in current level
9: end for

10: Top-down blurring:
11: for each level from coarsest to finest:
12:   Apply quadratic B-Spline interpolation
13:   Calculate new pixel colors from coarser level
14:   Weight interpolation by α values
15: end for

16: Write back blurred colors:
17: for each pixel in F:
18:   if pixel is unknown:
19:     Write blurred color with linear interpolation
20:     Regularize with value n
21:   end if
22: end for

23: Repeat steps 1-22 for i iterations

24: Return F

Alpha Estimation

目的： Alpha 估计阶段的目标是确定 trimap 未知区域中每个像素的 alpha 值（透明度）。 Alpha 值定义了前景与背景的像素比例，从而实现最终合成图像中的无缝混合。
Input: The stage receives the propagated foreground and background images and the trimap.
Output: It produces an alpha matte, visually showing the transparency levels of the unknown regions, ready for the final compositing step.

Algorithm 2: Alpha Estimation
Input: Color image I, propagated foreground F, propagated background B, neighborhood size n, weight w
Output: Alpha matte α

1: for each pixel p in I do
2:   Collect samples F_samples and B_samples from n × n neighborhood
3:   Initialize min_cost to infinity
4:   for each pair (F_i, B_j) in (F_samples, B_samples) do
5:     Estimate α_hat using equation (2)
6:     Calculate color cost C_col using equation (3)
7:     Calculate propagation cost C_pro using equation (4)
8:     Calculate total cost: cost = w * C_col + C_pro
9:     if cost < min_cost then
10:      min_cost = cost
11:      best_pair = (F_i, B_j)
12:   end for
13:   Set α_p = α_hat for best_pair
14: end for

15: Set α values for known foreground pixels to 1 and for background pixels to 0
16: Calculate alpha values for virtual objects using 1 - α_p

17: Return α

Compositing

目的：最后阶段使用 alpha matte将渲染的虚拟场景和真实场景的彩色图像组合成单个合成图像，以实现无缝混合。
Input: RGB image from the sensor and the rendered RGB image, along with the alpha matte.
Output: Final composite image.

Algorithm 3: Compositing
Input: Color image from real scene C_foreground, color image from virtual scene C_background, alpha matte α
Output: Composite image C_composite

1: for each pixel p in C_composite do
2:   α_p = α(p)  # Alpha value at pixel p
3:   C_fg = C_foreground(p)  # Foreground color at pixel p
4:   C_bg = C_background(p)  # Background color at pixel p
5:   C_composite(p) = α_p * C_fg + (1 - α_p) * C_bg  # Calculate composite color
6: end for

7: Apply anti-aliasing to C_composite to smooth edges
8: Perform color correction on C_composite if needed
9: Adjust lighting on C_composite for consistency

10: Return C_composite

Code（由gpt生成）
Step 1: 3D Rendering
This step involves rendering the color and depth values of the virtual scene. For simplicity, we’ll assume these are already provided as images.

import cv2
import numpy as np

# Load color and depth images from the virtual scene
color_image_virtual = cv2.imread('color_virtual.png')
depth_image_virtual = cv2.imread('depth_virtual.png', cv2.IMREAD_UNCHANGED)

Step 2: Adaptive Trimap Generation
Generate a trimap that specifies foreground, background, and unknown regions.

def generate_trimap(depth_image, threshold=10):
    trimap = np.zeros_like(depth_image)
    trimap[depth_image < threshold] = 255  # Background
    trimap[depth_image > 255 - threshold] = 128  # Foreground
    trimap[(depth_image >= threshold) & (depth_image <= 255 - threshold)] = 0  # Unknown
    return trimap

depth_image = cv2.imread('depth_image.png', cv2.IMREAD_GRAYSCALE)
trimap = generate_trimap(depth_image)

Step 3: Foreground and Background Propagation
Propagate known foreground and background colors to unknown regions.

def propagate_colors(image, trimap, num_levels=4, num_iterations=5):
    foreground = np.zeros_like(image)
    background = np.zeros_like(image)

    foreground[trimap == 128] = image[trimap == 128]
    background[trimap == 255] = image[trimap == 255]

    for _ in range(num_iterations):
        blurred_foreground = cv2.pyrDown(foreground)
        blurred_foreground = cv2.pyrUp(blurred_foreground)
        foreground[trimap == 0] = blurred_foreground[trimap == 0]

        blurred_background = cv2.pyrDown(background)
        blurred_background = cv2.pyrUp(blurred_background)
        background[trimap == 0] = blurred_background[trimap == 0]

    return foreground, background

foreground, background = propagate_colors(cv2.imread('color_image.png'), trimap)

Step 4: Alpha Estimation
Estimate the alpha matte based on the propagated foreground and background colors.

def estimate_alpha(image, foreground, background, trimap, weight=0.5):
    alpha = np.zeros(image.shape[:2], dtype=np.float32)
    rows, cols = image.shape[:2]

    for r in range(rows):
        for c in range(cols):
            if trimap[r, c] == 0:
                best_cost = float('inf')
                best_alpha = 0
                for i in range(max(0, r-1), min(rows, r+2)):
                    for j in range(max(0, c-1), min(cols, c+2)):
                        if trimap[i, j] in [128, 255]:
                            alpha_value = np.dot(image[r, c] - background[i, j], foreground[i, j] - background[i, j]) / np.linalg.norm(foreground[i, j] - background[i, j])**2
                            color_cost = np.linalg.norm(image[r, c] - (alpha_value * foreground[i, j] + (1 - alpha_value) * background[i, j]))
                            propagation_cost = (i + j) / 2  # Simplified propagation cost
                            total_cost = weight * color_cost + (1 - weight) * propagation_cost

                            if total_cost < best_cost:
                                best_cost = total_cost
                                best_alpha = alpha_value

                alpha[r, c] = best_alpha
            elif trimap[r, c] == 128:
                alpha[r, c] = 1
            elif trimap[r, c] == 255:
                alpha[r, c] = 0

    return alpha

alpha_matte = estimate_alpha(cv2.imread('color_image.png'), foreground, background, trimap)

Step 5: Compositing
Combine the color images of the rendered and real scene into a single image using the alpha matte.

def composite_images(foreground_image, background_image, alpha_matte):
    composite_image = np.zeros_like(foreground_image)
    alpha_matte_expanded = np.expand_dims(alpha_matte, axis=2)
    
    composite_image = alpha_matte_expanded * foreground_image + (1 - alpha_matte_expanded) * background_image
    return composite_image

color_image_real = cv2.imread('color_image_real.png')
composite_image = composite_images(color_image_real, color_image_virtual, alpha_matte)

# Save or display the composite image
cv2.imwrite('composite_image.png', composite_image)
cv2.imshow('Composite Image', composite_image)
cv2.waitKey(0)
cv2.destroyAllWindows()