# US 2022/0284625 A1 · System and method for determining 3d positional coordinates of a ball

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284625 A1
- Document ID: US20220284625A1
- Status: Published application
- Kind: A1
- Application number: 17/653,499
- Filed: 2022-03-04
- Published: 2022-09-08
- Earliest claimed: 2021-03-05
- Assignees: Trackman A/S
- Applicants: Trackman A/S
- Inventors: Fredrik TUXEN; Michael UNGSTRUP
- Web page: https://patentgrep.com/patent/US20220284625A1

## CPC

- A63B 71/0616: {Means for conducting or scheduling competition, league, tournaments or rankings}
- A63B 24/0006: {Computerised comparison for qualitative assessment of motion sequences or the course of a movement}
- A63B 24/0021: {Tracking a path or terminating locations}
- A63B 69/3691: {Golf practising terrains}
- B25J 9/1692: {Calibration of manipulator}
- B25J 9/1697: {Vision controlled systems}
- B25J 19/023: {including video camera means}
- G01S 13/9029: {specially adapted for moving target detection within a single SAR image or within multiple SAR images taken at the same time}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 7/75: {involving models}
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/85: {Stereo camera calibration}
- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G06V 10/82: using neural networks
- G06V 20/42: {of sport video content}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06V 40/50: Maintenance of biometric data or enrolment thereof
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- A63B 2024/0025: {Tracking the path or location of one or more users, e.g. players of a game}
- A63B 2024/0028: {Tracking the path of an object, e.g. a ball inside a soccer pitch}
- A63B 2071/0691: {Maps, e.g. yardage maps or electronic maps}
- A63B 2102/32: Golf
- A63B 2220/05: Image processing for measuring physical parameters
- A63B 2220/12: Absolute positions, e.g. by using GPS
- A63B 2220/62: Time or time measurement used for time reference, time stamp, master time or clock signal
- A63B 2220/803: Motion sensors
- A63B 2220/806: Video cameras
- A63B 2220/89: Field sensors, e.g. radar systems
- A63B 2225/54: Transponders, e.g. RFID
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30224: Ball; Puck
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system includes a camera capturing images in a first field of view of a sports ball bouncing and rolling, a storage arrangement, and a processing arrangement. The storage arrangement includes a three-dimensional 3D model of a part of a sports play area and the processing arrangement is configured to: detect a pixel location of a ball in an image; determine, based on intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in the direction of the sports ball and to determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model. The processor outputs the intersection point as a 3D position of the ball in the image.

## Claims

### Claim 1

A system, comprising:
a tracking camera having a first field of view configured to capture images in a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith;
a storage arrangement including the intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view; and
a processing arrangement coupled to the tracking camera configured to:
perform a ball detection to detect a pixel location of the sports ball in an image;
determine, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera a direction of the sports ball in a 3D coordinate system;
determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model; and
output the intersection point as a 3D position of the sports ball in the image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processing arrangement is further configured to:
determine the sports ball is at rest by comparing the pixel location of the sports ball in successive images; and
outputting the intersection point as a 3D rest position of the sports ball in the image.

### Claim 3 (depends on 1)

The system of claim 1, wherein the captured images are associated with metadata including a time at which the image was captured, an exposure time and a crop region.

### Claim 4 (depends on 1)

The system of claim 1, further comprising:
a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball,
wherein the processing arrangement is further configured to detect the sports ball in dependence on the radar data to narrow an area searched in the images and a number of images searched in time.

### Claim 5 (depends on 4)

The system of claim 4, wherein detections of the sports ball using the radar data in combination with the images are used by the processing arrangement to improve an accuracy of 3D position determinations of the sports ball.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processing arrangement is further configured to detect the sports ball using a neural network trained to detect sports balls in images.

### Claim 7 (depends on 6)

The system of claim 6, wherein the neural network uses information for search regions from previous images to improve an accuracy of detections of the sports ball in current images.

### Claim 8 (depends on 3)

The system of claim 3, wherein the processing arrangement is further configured to determine the camera-ball line based on the crop region for the image.

### Claim 9 (depends on 1)

The system of claim 1, wherein the sports play area is a golf course and wherein the 3D model of the part of the golf course contains a surface model representing a height of terrain for a given position on the golf course.

### Claim 10 (depends on 9)

The system of claim 9, wherein the surface model further represents a terrain type for the given position on the golf course.

### Claim 11 (depends on 10)

The system of claim 10, wherein the processing arrangement is further configured to determine a type of terrain within which the sports ball is detected based on the 3D model.

### Claim 12 (depends on 9)

The system of claim 9, wherein the surface model comprises a mesh of triangles or spline surfaces.

### Claim 13 (depends on 9)

The system of claim 9, wherein the intersection point is determined using an iterative process or using a numerical solver to solve an optimization problem.

### Claim 14 (depends on 1)

The system of claim 1, wherein the 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.

### Claim 15 (depends on 1)

The system of claim 1, wherein the 3D model is provided in the 3D coordinate system for the camera-ball line.

### Claim 16

A method, comprising:
performing a ball detection to detect a pixel location of a sports ball in an image captured by a tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored to a storage arrangement;
determining, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system;
determining, based on a three-dimensional (3D) model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least part of a sports play area overlapping with the first field of view; and
outputting the intersection point as a 3D position of the sports ball in the image.

### Claim 17

A processor coupled to a tracking camera and a storage arrangement configured to perform operations comprising:
performing a ball detection to detect a pixel location of a sports ball in an image captured by the tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored to the storage arrangement;
determining, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system;
determining, based on a three-dimensional (3D) model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least part of a sports play area overlapping with the first field of view; and
outputting the intersection point as a 3D position of the sports ball in the image.

### Claim 18

A system, comprising:
a tracking camera having a first field of view configured to capture images in an image plane of a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith;
a storage arrangement including the intrinsic and extrinsic calibration parameters; and
a processing arrangement coupled to the tracking camera and the storage arrangement configured to:
perform a ball detection to detect a pixel location in the image plane of the sports ball in each of a sequence of images;
generate a first time-series of the ball detections in the image plane;
determine, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the sports ball detections in the first time-series;
generate a second time-series of an elevation angle of each camera-ball line; and
identify, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.

### Claim 19 (depends on 18)

The system of claim 18, wherein the storage arrangement further stores a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view, wherein the processing arrangement is further configured to:
determine, based on the 3D model, for each identified bounce, an intersection point of the camera-ball line with the 3D model; and
output the intersection points as 3D positions of bounces of the sports ball.

### Claim 20 (depends on 18)

The system of claim 18, further comprising:
a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball, wherein the processing arrangement is further configured to:
identify bounces in dependence on the radar data by determining velocity discontinuities in the radar data.

### Claim 21 (depends on 18)

The system of claim 18, wherein the processing arrangement is further configured to:
distinguish bounces from rolls in a second time series based on a physical model of a bounce; and
classify each minima in the second time series that has not been determined to be a bounce as being a roll.

### Claim 22 (depends on 18)

The system of claim 18, wherein a 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.

### Claim 23

A method, comprising:
performing a ball detection to detect a pixel location in an image plane of a sports ball in each of a sequence of images captured by a tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored to a storage arrangement;
generating a first time-series of the ball detections in the image plane;
determining, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series;
generating a second time-series of an elevation angle of each camera-ball line; and
identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.

### Claim 24

A processor coupled to a tracking camera and a storage arrangement configured to perform operations comprising:
performing a ball detection to detect a pixel location in an image plane of a sports ball in each of a sequence of images captured by the tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith stored to the storage arrangement;
generating a first time-series of the ball detections in the image plane;
determining, based on the intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series;
generating a second time-series of an elevation angle of each camera-ball line; and
identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
