System and methods for predicting and removing interference patterns when capturing images from behind a partially transparent display
AssigneeVeeo Technology, Inc.
InventorsTimothy Barnes, Matthew Wehner
Life of this patent
Estimating the term…
Abstract
A camera located behind a partially transparent display allows for a participant in a video conference to appear to be looking directly at other participants, creating a natural eye-to-eye setting similar to an in-person meeting. However, the camera picks up moiré patterns and other distortion from the display. A machine learning model is trained to predict the interference from a frame of video on the display. The predicted interference is then subtracted from a corresponding video frame from the camera, resulting in a clean image. The clean image is then used in place of the camera's video frame in output video, resulting in high quality video useful for a video conference.
Claims11
A method for video display and capture, comprising the steps of:
- providing a display apparatus comprising a partially transparent display, a camera located behind the partially transparent display and configured to capture images through the partially transparent display, and a controller;
- preparing an interference prediction model;
- loading the interference model onto the controller of the display apparatus;
- obtaining an input video frame;
- obtaining a camera frame;
- sending the input video frame to the interference prediction model;
- receiving a predicted delta image from the interference prediction model;
- and
- subtracting the predicted delta image from the camera frame to create a clean image,
- wherein the step of preparing an inference prediction model comprises the steps of:
- preparing a plurality of datasets, each dataset comprising a display image, a camera image with interference, and a clean camera image;
- for each dataset, performing the steps of:
- predicting, using a neural network model, a distortion pattern image delta,
- subtracting the predicted distortion pattern image delta from the camera image with interference to create a cleaned image, and
- comparing the cleaned image with the clean camera image;
- and
- finalizing the neural network model for use as the interference prediction model.
Dependencies are inferred from the claim text.