# US 11,451,760 B2 · Systems and methods for correcting rolling shutter artifacts

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,760 B2
- Document ID: US11451760B2
- Status: Granted
- Kind: B2
- Application number: 17/319,553
- Filed: 2021-05-13
- Published: 2021-12-16
- Granted: 2022-09-20
- Earliest claimed: 2020-06-15
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Michael Bleyer; Christopher Douglas Edmonds; Raymond Kirk Price
- Web page: https://patentgrep.com/patent/US11451760B2

## CPC

- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/593: from stereo images
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/15: for colour aspects of image signals
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 23/689: {Motion occurring during a rolling shutter mode}
- H04N 25/677: for reducing the column or line fixed pattern noise
- H04N 25/701: Line sensors
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/30244: Camera pose
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.

## Claims

### Claim 1

A system for compensating for rolling shutter artifacts, comprising:
a rolling shutter sensor comprising a plurality of sensor rows, wherein different sensor rows in the plurality of sensor rows output sensor data at different times, such that the sensor data output from different sensor rows in the plurality of sensor rows are temporally offset;
one or more processors; and
one or more hardware storage devices having stored computer-executable instructions that are operable, when executed by the one or more processors, to configure the system to:
obtain a 3D representation of an environment comprising a plurality of points that represent the environment;
identify a plurality of readout timepoints comprising a different readout timepoint for each different sensor row of the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of the environment;
identify a plurality of readout poses comprising a separate readout pose associated with the system for each different readout timepoint of the plurality of readout timepoints and for each different sensor row of the plurality of sensor rows;
obtain a plurality of 2D coordinates comprising a corresponding 2D coordinate for each point of a set of points of the 3D representation of the environment by projecting each point of the set of points based on a default pose associated with a default sensor row of the plurality of sensor rows of the rolling shutter sensor;
identify a plurality of estimated sensor rows comprising a corresponding estimated sensor row of the plurality of sensor rows for each point of the set of points, wherein the corresponding estimated sensor row for any particular point of the set of points is based on the corresponding 2D coordinate for the particular point;
identify a plurality of projection poses comprising a corresponding projection pose for each point of the set of points, wherein the corresponding projection pose for any particular point of the set of points corresponds to the separate readout pose that is associated with the corresponding estimated sensor row for the particular point; and
project one or more points of the set of points based on the corresponding projection pose for each point in the set of points.

### Claim 2 (depends on 1)

The system of claim 1, wherein the projecting of one or more points comprises projecting a plurality of points of the set of points based on the corresponding projection pose for each point in the set of points.

### Claim 3 (depends on 1)

The system of claim 1, wherein the default sensor row is a center sensor row of the plurality of sensor rows.

### Claim 4 (depends on 1)

The system of claim 1, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
obtain a plurality of updated 2D coordinates comprising a corresponding updated 2D coordinate for each point of the set of points based on the projection of each point of the set of points based on the corresponding projection pose for each point of the set of points;
identify a plurality of updated estimated sensor rows comprising a corresponding updated estimated sensor row of the plurality of sensor rows for each point of the set of points, wherein the corresponding updated estimated sensor row for any particular point of the set of points is based on the corresponding updated 2D coordinate for the particular point;
identify a plurality of updated projection poses comprising a corresponding updated projection pose for each point of the set of points, wherein the corresponding updated projection pose for any particular point of the set of points corresponds to the separate readout pose that is associated with the corresponding updated estimated sensor row for the particular point; and
project one or more points of the set of points based on the corresponding updated projection pose for each point in the set of points.

### Claim 5 (depends on 1)

The system of claim 1, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to iteratively obtain newly updated corresponding projection poses for at least some points of the set of points until a difference between a most recently identified updated corresponding estimated sensor row and a preceding corresponding estimated sensor row for one or more points of the set of points is below a predetermined difference threshold.

### Claim 6 (depends on 1)

The system of claim 1, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
obtain texture information for at least a portion of the 3D representation of the environment based on the image of the environment captured by the rolling shutter sensor and based on the projection of each point of the set of points based on the corresponding projection pose for each point in the set of points.

### Claim 7 (depends on 6)

The system of claim 6, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
obtain second texture information for at least a portion of the 3D representation of the environment based on an additional image of the environment captured by an additional sensor and based on a projection of at least some points of the set of points based on one or more poses associated with the additional sensor.

### Claim 8 (depends on 7)

The system of claim 7, wherein the rolling shutter sensor comprises a low light camera or a thermal camera, and wherein the additional sensor comprises a thermal camera.

### Claim 9 (depends on 6)

The system of claim 6, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to cause the system to:
project the at least a portion of the 3D representation of the environment using a display pose associated with the system for a display timepoint; and
render an image of the environment at the display timepoint based on the texture information and based on the projection of at least the portion of the 3D representation of the environment using the display pose.

### Claim 10 (depends on 1)

The system of claim 1, wherein the system is a wearable HMD system.

### Claim 11

A system for compensating for rolling shutter artifacts, comprising:
a rolling shutter sensor comprising a plurality of sensor rows, wherein different sensor rows in the plurality of sensor rows output sensor data at different times, such that the sensor data output from different sensor rows in the plurality of sensor rows are temporally offset;
one or more processors; and
one or more hardware storage devices having stored computer-executable instructions that are operable, when executed by the one or more processors, to configure the system to:
identify a plurality of different readout timepoints comprising a different readout timepoint for each of at least two different sensor rows of the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment;
identify a plurality of readout poses comprising a readout pose associated with the system for each different readout timepoint of the plurality of different readout timepoints;
obtain a depth map of the environment based on the image of the environment captured using the rolling shutter sensor, the depth map comprising a plurality of different rows of depth data, each different row of depth data corresponding to a different sensor row of the plurality of sensor rows of the rolling shutter sensor; and
generate a 3D representation of the environment by using the readout poses associated with the system for each different readout timepoint of the plurality of different readout timepoints.

### Claim 12 (depends on 11)

The system of claim 11, wherein the system comprises a stereo pair of rolling shutter sensors, and wherein the depth map of the environment is based on a stereo pair of images captured by the stereo pair of rolling shutter sensors.

### Claim 13 (depends on 11)

The system of claim 11, wherein generating the 3D representation includes unprojecting two or more different rows of depth data of the plurality of different rows of depth data using a same readout pose of the plurality of readout poses.

### Claim 14 (depends on 11)

The system of claim 11, wherein there is an equal number of different readout timepoints of the plurality of different readout timepoints as different sensor rows of the plurality of sensor rows.

### Claim 15 (depends on 14)

The system of claim 14, wherein generating the 3D representation includes unprojecting the two or more different rows of depth data by using a separate readout pose of the plurality of readout poses.

### Claim 16 (depends on 14)

The system of claim 14, wherein the number of different readout timepoints of the plurality of readout timepoints is determined based on a detected motion attribute of the system.

### Claim 17 (depends on 1)

The system of claim 1, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
project the 3D representation of the environment using a single pose.

### Claim 18 (depends on 17)

The system of claim 17, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
obtain texture information for at least a portion of the 3D representation of the environment based on the projection of the 3D representation of the environment using the single pose and based on one or more images of the environment.

### Claim 19 (depends on 18)

The system of claim 18, wherein the computer-executable instructions are further operable, when executed by the one or more processors, to configure the system to:
generate a corrected image of the environment based on the texture information and based on the projection of the 3D representation of the environment using the single pose.

### Claim 20 (depends on 18)

The system of claim 18, wherein the one or more images of the environment are captured by a global shutter sensor subsequent to capturing the image of the environment with the rolling shutter sensor, the global shutter sensor being positioned at the single pose.