# US 2021/0342600 A1 · Structure annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342600 A1
- Document ID: US20210342600A1
- Status: Published application
- Kind: A1
- Application number: 17/279,784
- Filed: 2019-09-26
- Published: 2021-11-04
- Earliest claimed: 2018-09-26
- Assignees: Five Al Limited
- Applicants: Five Al Limited
- Inventors: THOMAS WESTMACOTT; Joel Jakubovic; John Redford; Robert Chandler
- Web page: https://patentgrep.com/patent/US20210342600A1

## CPC

- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06T 7/251: {involving models}
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G08G 1/056: with provision for distinguishing direction of travel
- G06T 2200/04: involving 3D image data
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A method of annotating frames of a time sequence of frames captured by at least one travelling vehicle comprises, in a frame processing system: determining a three-dimensional (3D) road model for an area captured in the time sequence of frames; receiving first annotation data denoting a known 3D location of a moving object for a first frame of the time sequence of frames; and automatically generating second annotation data for marking an expected moving object location in at least a second frame of the time sequence of frames, by assuming the moving object moves along an expected path determined from the known 3D location and the 3D road model.

## Claims

### Claim 1

A method of annotating frames of a time sequence of frames captured by at least one travelling vehicle, the method comprising, in a frame processing system:
determining a three-dimensional (3D) road model for an area captured in the time sequence of frames;
receiving first annotation data denoting a known 3D location of a moving object for a first frame of the time sequence of frames; and
automatically generating second annotation data for marking an expected moving object location in at least a second frame of the time sequence of frames, by assuming the moving object moves along an expected path determined from the known 3D location and the 3D road model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the frames are 2D images.

### Claim 3 (depends on 1)

The method of claim 1, wherein the 3D location of the moving object is determined for the first frame based on an imposed geometric restriction on an elevation of the known 3D location relative to a road surface of the 3D road model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the geometric restriction restricts the known 3D location to a predetermined zero or non-zero elevation relative to the road surface of the 3D road model.

### Claim 5 (depends on 3)

The method of claim 3, wherein a 3D annotation element is located at an initial 3D location, the 3D annotation element is geometrically projected into the image plane of the first frame, and the 3D location is adjusted whilst said geometric restriction is maintained to match the projection of the 3D annotation element to the moving object in the first frame, thereby determining the known 3D location of the moving object.

### Claim 6 (depends on 5)

The method of claim 5, wherein the 3D location is manually adjusted to manually match the projection to the moving object in the image plane whilst maintaining said geometric restriction.

### Claim 7 (depends on 5)

The method of claim 5, wherein the second annotation data is determined by transforming the 3D annotation element to move it along the expected path to the expected moving object location in 3D space.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second annotation data comprises 2D annotation data for marking the expected moving object location in the image plane of the second frame, the 2D annotation data determined by projecting the 3D annotation element at the expected moving object location into the image plane of the second frame.

### Claim 9 (depends on 5)

The method of claim 5, wherein first 2D annotation data is determined for the first frame by projecting the 3D annotation element at the known 3D location into the image plane of the first frame.

### Claim 10 (depends on 1)

The method of claim 1, wherein the expected path is determined based on a speed of the moving object and a time interval between the first and second frames.

### Claim 11 (depends on 10)

The method of claim 10, wherein the speed is assumed to match a speed of the travelling vehicle.

### Claim 12 (depends on 10)

The method of claim 10, wherein an additional known 3D location of the moving object is determined for a third of the frames, and the known 3D location and the additional known 3D location are used to estimate the speed based on a time interval between the first and third frames.

### Claim 13 (depends on 1)

The method of claim 1, wherein the expected road model is determined by reconstructing a path travelled by the vehicle whilst capturing the time sequences of images, the 3D road model exhibiting a road shape that corresponds to the shape of the reconstructed vehicle path.

### Claim at position 14

-15. (canceled)

### Claim 16 (depends on 13)

The method of claim 13, wherein the expected path of the moving object is defined as having zero vertical offset from the reconstructed vehicle path in the direction of a road surface normal and a fixed lateral offset from the reconstructed vehicle path.

### Claim 17

(canceled)

### Claim 18 (depends on 1)

The method of claim 1, wherein a direction of movement is inferred for the moving object based on its location relative to a road structure of the 3D road model.

### Claim at position 18

19-20. (canceled)

### Claim 21 (depends on 1)

The method of claim 1, wherein the annotated frames are used to train a machine learning model in a supervised training process.

### Claim 22 (depends on 1)

The method of claim 1, wherein location data captured in association with the time sequence of frames is used to match the time sequence of frames to an area of a map in order to determine a road shape for the 3D road model.

### Claim 23 (depends on 1)

The method of claim 1, wherein the second annotation data takes the form of a 3D annotation element located at the expected moving object location in 3D space.

### Claim at position 22

-44. (canceled)

### Claim 45

A processing computer system comprising:
one or more processors configured to:
receive a time sequence of frames captured by at least one travelling vehicle;
determine a three-dimensional (3D) road model for an area captured in the time sequence of frames;
receive first annotation data denoting a known 3D location of a moving object for a first frame of the time sequence of frames, and
automatically generate second annotation data for marking an expected moving object location in at least a second frame of the time sequence of frames, by assuming the moving object moves along an expected path determined from the known 3D location and the 3D road model.

### Claim 46

A computer program comprising executable instructions which, when executed on one or more processors of a processing computer system, cause the computer system to:
receive a time sequence of frames captured by at least one travelling vehicle;
determine a three-dimensional (3D) road model for an area captured in the time sequence of frames;
receive first annotation data denoting a known 3D location of a moving object for a first frame of the time sequence of frames; and
automatically generate second annotation data for marking an expected moving object location in at least a second frame of the time sequence of frames, by assuming the moving object moves along an expected path determined from the known 3D location and the 3D road model.
