# US 2023/0065922 A1 · Self-organized learning of three-dimensional motion data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0065922 A1
- Document ID: US20230065922A1
- Status: Published application
- Kind: A1
- Application number: 17/460,038
- Filed: 2021-08-27
- Published: 2023-03-02
- Earliest claimed: 2021-08-27
- Assignees: Rapsodo Pte. Ltd.
- Applicants: Rapsodo Pte. Ltd.
- Inventors: Batuhan OKUR; Roshan Gopalakrishnan
- Web page: https://patentgrep.com/patent/US20230065922A1

## CPC

- G06F 18/251: {of input or preprocessed data}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/08: Systems for measuring distance only (indirect measurement G01S13/46)
- G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems
- G01S 13/587: {using optical means}
- G01S 13/72: for two-dimensional [2D] tracking, e.g. combination of angle and range tracking, track-while-scan radar
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/88: Radar or analogous systems specially adapted for specific applications (electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00)
- G01S 13/89: for mapping or imaging
- G06T 7/215: Motion-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30224: Ball; Puck
- G06T 2207/30232: Surveillance
- G06T 2207/30241: Trajectory
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method may include capturing image data associated with an object in a defined environment at one or more points in time. The method may include capturing radar data associated with the object in the defined environment at the same points in time. The method may include obtaining, by a machine learning model, the image data and the radar data associated with the object in the defined environment. The method may include pairing each image datum with a corresponding radar datum based on a chronological occurrence of the image data and the radar data. The method may include generating, by the machine learning model, a three-dimensional motion representation associated with the object that is associated with the image data and the radar data.

## Claims

### Claim 1

A method, comprising:
capturing image data associated with an object in a defined environment at a plurality of points in time;
capturing radar data associated with the object in the defined environment at the plurality of points in time;
obtaining, by a machine learning model, the image data and the radar data associated with the object in the defined environment;
pairing each image datum with a corresponding radar datum based on a chronological occurrence of the image data and the radar data; and
generating, by the machine learning model, a three-dimensional motion representation associated with the object that is associated with the image data and the radar data.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the three-dimensional motion representation associated with the object that is associated with the image data and the radar data comprises:
organizing the image data chronologically;
generating an optical flow based on the chronologically organized image data;
identifying radar data collected over a same time period as the image data; and
applying the identified radar data to the optical flow based on chronological occurrence of the radar data.

### Claim 3 (depends on 1)

The method of claim 1, wherein the three-dimensional motion representation associated with the object includes at least one of: velocity analysis of the object, body-movement analysis, or movement simulation of the object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the image data includes a plurality of two-dimensional continuous frames of the object.

### Claim 5 (depends on 1)

The method of claim 1, wherein the radar data includes at least one of: distance data, velocity data, or frequency data associated with the object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the object includes at least one of: a ball, a sports apparatus, a human appendage, or a human.

### Claim 7 (depends on 1)

The method of claim 1, wherein the three-dimensional motion representation associated with the object includes simulating virtual image data, wherein simulating the virtual image data includes:
identifying a location and an angle associated with a virtual camera;
estimating one or more extrinsic parameters associated with the virtual camera based on the captured image data and the captured radar data; and
generating the virtual image data at the identified location and angle of the virtual camera.

### Claim 8

A system for capturing and analyzing motion, comprising a computing system communicatively coupled to one or more cameras and one or more radar sensors, the computing system configured to:
obtain image data associated with an object in a defined environment at a plurality of points in time from the one or more cameras;
obtain radar data associated with the object in the defined environment at the plurality of points in time from the one or more radar sensors;
pair, by a machine learning model, each image datum with a corresponding radar datum based on chronological occurrence of the image data and the radar data; and
generate, by the machine learning model, the three-dimensional motion representation associated with the object that is associated with the image data and the radar data.

### Claim 9 (depends on 8)

The system of claim 8, wherein generating the three-dimensional motion representation associated with the object that is associated with the image data and the radar data comprises:
organizing the image data chronologically;
generating an optical flow based on the chronologically organized image data;
identifying radar data collected over a same time period as the image data; and
applying the identified radar data to the optical flow based on chronological occurrence of the radar data.

### Claim 10 (depends on 8)

The system of claim 8, wherein the three-dimensional motion representation associated with the object includes at least one of: velocity analysis of the object, body-movement analysis, or movement simulation of the object.

### Claim 11 (depends on 8)

The system of claim 8, wherein the image data includes a plurality of two-dimensional continuous frames of the object.

### Claim 12 (depends on 8)

The system of claim 8, wherein the radar data includes at least one of: distance data, velocity data, or frequency data associated with the object.

### Claim 13 (depends on 8)

The system of claim 8, further comprising:
one or more cameras configured to collect image data within a defined environment; and
one or more radar sensors configured to collect radar data within the defined environment.

### Claim 14 (depends on 8)

The system of claim 8, wherein the three-dimensional motion representation associated with the object includes simulating virtual image data, wherein simulating the virtual image data includes:
identifying a location and an angle associated with a virtual camera;
estimating one or more extrinsic parameters associated with the virtual camera based on the captured image data and the captured radar data; and
generating the virtual image data at the identified location and angle of the virtual camera.

### Claim 15

One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations comprising:
capturing image data associated with an object in a defined environment at a plurality of points in time;
capturing radar data associated with the object in the defined environment at the plurality of points in time;
obtaining, by a machine learning model, the image data and the radar data associated with the object in the defined environment;
pairing each image datum with a corresponding radar datum based on chronological occurrence of the image data and the radar data; and
generating, by the machine learning model, the three-dimensional motion representation associated with the object that is associated with the image data and the radar data.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable storage media of claim 15, wherein generating the three-dimensional motion representation associated with the object that is associated with the image data and the radar data comprises:
organizing the image data chronologically;
generating an optical flow based on the chronologically organized image data;
identifying radar data collected over a same time period as the image data; and
applying the identified radar data to the optical flow based on chronological occurrence of the radar data.

### Claim 17 (depends on 15)

The one or more non-transitory computer-readable storage media of claim 15, wherein the three-dimensional motion representation associated with the object includes at least one of: velocity analysis of the object, body-movement analysis, or movement simulation of the object.

### Claim 18 (depends on 15)

The one or more non-transitory computer-readable storage media of claim 15, wherein the image data includes a plurality of two-dimensional continuous frames of the object.

### Claim 19 (depends on 15)

The one or more non-transitory computer-readable storage media of claim 15, wherein the radar data includes at least one of: distance data, velocity data, or frequency data associated with the object.

### Claim 20 (depends on 15)

The one or more non-transitory computer-readable storage media of claim 15, wherein the three-dimensional motion representation associated with the object includes simulating virtual image data, wherein simulating the virtual image data includes:
identifying a location and an angle associated with a virtual camera;
estimating one or more extrinsic parameters associated with the virtual camera based on the captured image data and the captured radar data; and
generating the virtual image data at the identified location and angle of the virtual camera.
