# US 2024/0005460 A1 · Self-learning distortion correction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005460 A1
- Document ID: US20240005460A1
- Status: Published application
- Kind: A1
- Application number: 18/326,555
- Filed: 2023-05-31
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Axis AB
- Applicants: Axis AB
- Inventors: Joakim ERICSON; Song YUAN; Johan STENING
- Web page: https://patentgrep.com/patent/US20240005460A1

## CPC

- G06T 5/80: Geometric correction
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 3/047: Fisheye or wide-angle transformations
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06V 10/243: {by compensating for image skew or non-uniform image deformations}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- B60R 2300/607: from a bird's eye viewpoint
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

A method of distortion correction in an image captured by a non-rectilinear camera includes obtaining multiple images of a scene captured by the camera over time, determining where bottom portions of objects having moved over a horizontal surface in the scene are located in the images, determining a boundary of the horizontal surface in the scene based on the determined locations of the bottom portions, generating a three-dimensional model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and correcting a distortion of at least one of the images by projecting the image onto the three-dimensional model of the scene. A corresponding device, computer program and computer program product are also provided.

## Claims

### Claim 1

A method of distortion correction in an image of a scene captured by a camera having a non-rectilinear lens, including:
i) obtaining multiple images of the scene captured by the camera over time;
ii) determining where bottom portions of one or more objects having moved over a horizontal surface in the scene are located in at least some of the multiple images;
iii) determining a boundary of the horizontal surface in the scene based on the determined locations of the bottom portions;
iv) generating a three-dimensional model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and
v) correcting a distortion of at least one of the multiple images by projecting at least one image of the multiple images onto the three-dimensional model of the scene.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the non-rectilinear lens is a fisheye lens.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the bottom portions include feet of the objects.

### Claim 4 (depends on 1)

The method according to claim 1, wherein determining where bottom portions of one or more objects have moved further includes identifying one or more keypoints of the one or more objects.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the scene is of a room, and wherein the horizontal surface of the scene is the floor of the room.

### Claim 6 (depends on 1)

The method according to claim 1, wherein at least some of the multiple images are captured during a training period, and wherein an area of said at least some of the multiple images is considered to belong to the horizontal surface of the scene if a number of bottom portions having moved in the area during a finite time interval exceeds a threshold value.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the three-dimensional model includes a horizontal surface bounded by the determined boundary, and wherein the one or more vertical surfaces project in a direction perpendicular to said horizontal surface.

### Claim 8

A device for distortion correction in an image of a scene captured by a camera having a non-rectilinear lens, the device including processing circuitry configured to cause the device to:
i) obtain multiple images of the scene captured by the camera over time;
ii) determine where bottom portions of one or more objects having moved over a horizontal surface in the scene are located in at least some of the multiple images;
iii) determine a boundary of the horizontal surface in the scene based on the determined locations of the bottom portions;
iv) generate a three-dimensional model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and
v) correct a distortion of at least one of the multiple images by projecting the at least one image onto the three-dimensional model of the scene.

### Claim 9 (depends on 8)

The device according to claim 8, wherein the non-rectilinear lens is a fisheye lens.

### Claim 10 (depends on 8)

The device according to claim 8, wherein the bottom portions include feet of the objects.

### Claim 11 (depends on 8)

The device according to claim 8, wherein determining where bottom portions of one or more objects having moved further includes identifying one or more keypoints of the one or more objects.

### Claim 12 (depends on 8)

The device according to claim 8, wherein the scene is of a room, and wherein the horizontal surface of the scene is the floor of the room.

### Claim 13 (depends on 8)

The device according to claim 8, wherein at least some of the multiple images are captured during a training period, and wherein an area of said at least some of the multiple images is considered to belong to the horizontal surface of the scene if a number of bottom portions having moved in the area during a finite time interval exceeds a threshold value.

### Claim 14 (depends on 8)

The device according to claim 8, wherein the three-dimensional model includes a horizontal surface bounded by the determined boundary, and wherein the one or more vertical surfaces project in a direction perpendicular to said horizontal surface.

### Claim 15 (depends on 8)

The device according to claim 8, wherein the device is the camera having the non-rectilinear lens.

### Claim 16

A non-transitory computer readable storage medium having stored thereon a computer program for distortion correction in an image of a scene captured by a camera having a non-rectilinear lens, wherein the computer program includes computer code that, when running on processing circuitry of a device, causes the device to:
i) obtain multiple images of the scene captured by the camera over time;
ii) determine where bottom portions of one or more objects having moved over a horizontal surface in the scene are located in at least some of the multiple images;
iii) determine a boundary of the horizontal surface in the scene based on the determined locations of the bottom portions;
iv) generate a three-dimensional model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and
v) correct a distortion of at least one of the multiple images by projecting the at least one image onto the three-dimensional model of the scene.
