Back to results
US 12,744,874 B2Granted
Misaligned vantage point mitigation for computer stereo vision
AssigneeSnap Inc.
InventorsOlha Borys, Georg Halmetschlager-Funek, Matthias Kalkgruber, Daniel Wolf, Jakob Zillner
Life of this patent
Estimating the term…
Abstract
Disclosed are systems, methods, and non-transitory computer-readable media for misaligned vantage point mitigation for computer stereo vision. A misaligned vantage point mitigation system determines whether vantage points of the optical sensors are misaligned from an expected vantage point and, if so, determines an adjustment variable to mitigate the misalignment based on the location of matching features identified in images captured by both optical sensors.
Claims20
A system for mitigating misaligned vantage points in computer stereo vision, comprising:
- one or more computer processors;
- an inertial measurement unit (IMU);
- a first optical sensor of a head-worn device and a second optical sensor of the head-worn device, the first and second optical sensors having at least partially overlapping fields of view;
- one or more computer readable mediums storing instructions that, when executed by the one or more computer processors, causes the system to perform operations comprising:
- accessing a first image generated by the first optical sensor and a second image generated by the second optical sensor during a rolling shutter image capture;
- calculating depth data from the first image and the second image;
- detecting an image feature depicted in both the first image and the second image;
- determining an expected location of the image feature in the second image based on IMU data taken during the rolling shutter image capture, wherein the expected location is determined based on a predicted pose of the head-worn device derived from IMU measurements captured during the rolling shutter image capture;
- constraining the expected location of the image feature to a horizontal correspondence line in the second image based on a horizontal displacement between the first optical sensor and the second optical sensor;
- calculating a variance between the expected location and an actual location of the image feature in the second image relative to the horizontal correspondence line;
- determining that the first and second optical sensors are misaligned based on the calculated variance exceeding a threshold value;
- generating an adjustment variable based on the calculated variance to compensate for the misalignment, wherein the adjustment variable modifies one or more parameters of a stereo geometric model used to compute the depth data;
- adjusting subsequent depth calculations of images captured by the first and second optical sensors based on the adjustment variable;
- and
- causing display of a presentation of Augmented-Reality (AR) content based on the adjusted depth calculations, wherein the AR content is positioned relative to real-world objects detected by the first and second optical sensors.
12 more elements
A method for mitigating misaligned vantage points in computer stereo vision, the method comprising:
- accessing a first image generated by a first optical sensor of a head-worn device and a second image generated by the second optical sensor of the head-worn device during a rolling shutter image capture, the first and second optical sensors having at least partially overlapping fields of view;
- calculating depth data from the first and second image data;
- detecting an image feature depicted in both the first image and the second image;
- determining an expected location of the image feature in the second image based on IMU data taken during the rolling shutter image capture, wherein the expected location is determined based on a predicted pose of the head-worn device derived from IMU measurements captured during the rolling shutter image capture;
- constraining the expected location of the image feature to a horizontal correspondence line in the second image based on a horizontal displacement between the first optical sensor and the second optical sensor;
- calculating a variance between the expected location and an actual location of the image feature in the second image relative to the horizontal correspondence line;
- determining that the first and second optical sensors are misaligned based on the calculated variance exceeding a threshold value;
- generating an adjustment variable based on the calculated variance to compensate for the misalignment, wherein the adjustment variable modifies one or more parameters of a stereo geometric model used to compute the depth data;
- adjusting subsequent depth calculations of images captured by the first and second optical sensors based on the adjustment variable;
- and
- causing display of a presentation of Augmented-Reality (AR) content based on the adjusted depth calculations, wherein the AR content is positioned relative to real-world objects detected by the first and second optical sensors.
8 more elements
A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to perform operations comprising:
- accessing a first image generated by a first optical sensor of a head-worn device and a second image generated by the second optical sensor of the head-worn device during a rolling shutter image capture, the first and second optical sensors having at least partially overlapping fields of view;
- calculating depth data from the first and second image data;
- detecting an image feature depicted in both the first image and the second image;
- determining an expected location of the image feature in the second image based on IMU data taken during the rolling shutter image capture, wherein the expected location is determined based on a predicted pose of the head-worn device derived from IMU measurements captured during the rolling shutter image capture;
- constraining the expected location of the image feature to a horizontal correspondence line in the second image based on a horizontal displacement between the first optical sensor and the second optical sensor;
- calculating a variance between the expected location and an actual location of the image feature in the second image relative to the horizontal correspondence line;
- determining that the first and second optical sensors are misaligned based on the calculated variance exceeding a threshold value;
- generating an adjustment variable based on the calculated variance to compensate for the misalignment, wherein the adjustment variable modifies one or more parameters of a stereo geometric model used to compute the depth data;
- adjusting subsequent depth calculations of images captured by the first and second optical sensors based on the adjustment variable;
- and
- causing display of a presentation of Augmented-Reality (AR) content based on the adjusted depth calculations, wherein the AR content is positioned relative to real-world objects detected by the first and second optical sensors.
8 more elements
Dependencies are inferred from the claim text.