# US 2021/0279896 A1 · Multi-cam ball location method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279896 A1
- Document ID: US20210279896A1
- Status: Published application
- Kind: A1
- Application number: 17/255,837
- Filed: 2018-09-28
- Published: 2021-09-09
- Earliest claimed: 2018-09-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Xiaofeng Tong; Chen Ling; Ming Lu; Qiang Li; Wenlong Li; Yikai Fang; Yumeng Wang
- Web page: https://patentgrep.com/patent/US20210279896A1

## CPC

- G06T 7/292: Multi-camera tracking
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/215: Motion-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01S 3/7864: {T.V. type tracking systems}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30224: Ball; Puck
- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A multi-camera architecture for detecting and tracking a ball in real-time. The multi-camera architecture includes network interface circuitry to receive a plurality of real-time videos taken from a plurality of high-resolution cameras. Each of the high-resolution cameras simultaneously captures a sports event, wherein each of the plurality of high-resolution cameras includes a viewpoint that covers an entire playing field where the sports event is played. The multi-camera architecture further includes one or more processors coupled to the network interface circuitry and one or more memory devices coupled to the one or more processors. The one or more memory devices includes instructions to determine the location of the ball for each frame of the plurality of real-time videos, which when executed by the one or more processors, cause the multi-camera architecture to simultaneously perform one of a detection scheme or a tracking scheme on a frame from each of the plurality of real-time videos to detect the ball used in the sports event and perform a multi-camera build to determine a location of the ball in 3D for the frame from each of the plurality of real-time videos using one of detection or tracking results for each of the cameras.

## Claims

### Claim at position 1

1-25. (canceled)

### Claim 26

A multi-camera architecture, comprising:
network interface circuitry to receive a plurality of real-time videos taken from a plurality of high-resolution cameras, each of the high-resolution cameras simultaneously capturing a sports event, wherein each of the plurality of high-resolution cameras includes a viewpoint that covers an entire playing field where the sports event is played;
one or more processors coupled to the network interface circuitry;
one or more memory devices coupled to the one or more processors, the one or more memory devices including instructions to determine a location of a ball for each frame of the plurality of real-time videos, which when executed by the one or more processors, cause the multi-camera architecture to:
simultaneously perform one of a detection scheme or a tracking scheme on a frame from each of the plurality of real-time videos to detect the ball used in the sports event; and
perform a multi-camera build to determine the location of the ball in 3D (3-Dimensions) for the frame from each of the plurality of real-time videos using one of detection or tracking results for each camera.

### Claim 27 (depends on 26)

The multi-camera architecture of claim 26, wherein the detection scheme comprises instructions, which when executed by the one or more processors, cause the multi-camera architecture to:
retrieve the frame for each of the cameras;
retrieve a background image from each of the cameras;
remove the background image from the frame to obtain a foreground mask for each of the cameras;
partition the foreground mask into tiles to obtain a partitioned foreground mask for each of the cameras;
perform motion filtering on the partitioned foreground mask to obtain a motion filtered foreground mask for each of the cameras;
perform detection of the ball for each tile in the frame of the motion filtered foreground mask that indicates motion is occurring for each of the cameras; and
collect detection results from all of the tiles in the frame for each of the cameras.

### Claim 28 (depends on 26)

The multi-camera architecture of claim 26, wherein the tracking scheme comprises instructions, that when executed by the one or more processors, cause the multi-camera architecture to:
perform tracking-by-detection when the ball was detected in a previous frame, wherein instructions to perform tracking-by-detection further comprise instructions to only perform detection on a single tile, the single tile being set using a ball center of the previous frame in which the ball was detected or tracked as a tile center the single tile.

### Claim 29 (depends on 27)

The multi-camera architecture of claim 27, wherein detection includes detection of the ball using one of YOLO (You Only Look Once), Faster RCNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), and any other object detection technique used to detect small objects.

### Claim 30 (depends on 28)

The multi-camera architecture of claim 28, wherein detection includes detection of the ball using one of YOLO (You Only Look Once), Faster RCNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), and any other object detection technique used to detect small objects.

### Claim 31 (depends on 26)

The multi-camera architecture of claim 26, wherein the multi-camera build comprises instructions, that when executed by the one or more processors, cause the multi-camera architecture to:
perform a multi-camera cross validation, the multi-camera cross validation including instructions to sample the detection results from a set of cameras, wherein the set of cameras are selected using a random sampling method; and calculate a matching error along an epipolar line for the set of cameras randomly selected;
when the matching error is equal to or greater than a predetermined threshold, a miss or false detection has occurred, wherein instructions further comprise to repeat the multi-camera cross validation instructions until the matching error is less than the predetermined threshold; and
when the matching error is less than the predetermined threshold, multi-camera build instructions further comprise instructions to determine a 3D ball location using the sample cameras, re-project the 3D ball location onto each of the cameras, and determine a distance between a detected position of the ball and a re-projection position of the ball for each of the cameras, wherein if the distance is less than a pre-determined threshold, the results from the detection of the ball are correct using the set of cameras, wherein the multi-camera build instructions further comprise instructions to place the set of cameras on an inner list and apply bundle adjustment to get an optimized 3D ball location.

### Claim 32 (depends on 31)

The multi-camera architecture of claim 31, wherein the multi-camera build further comprises instructions to repeat all of the multi-camera build instructions N times to obtain an optimal result with minimal re-project error.

### Claim 33 (depends on 26)

The multi-camera architecture of claim 26, wherein when the multi-camera build is successful, the tracking scheme is used in the next frame of each of the videos to locate the ball; and wherein when the multi-camera build is unsuccessful, the detection scheme is used in the next frame of each of the videos to locate the ball.

### Claim 34 (depends on 26)

The multi-camera architecture of claim 26, wherein further instructions, which when executed by the one or more processors, cause the multi-camera architecture to:
project the 3d ball location onto each of the results of the plurality of cameras in 2D (2-Dimensions);
perform the detection around a projected position to obtain a more accurate location of the ball for the frame from each of the cameras; and
continuously advance each of the plurality of real-time videos to a next frame to repeat the instructions to determine the location of the ball for the next frame until the plurality of real-time videos end.

### Claim 35 (depends on 26)

The multi-camera architecture of claim 26, wherein the plurality of high-resolution cameras comprises twelve (12) high-resolution cameras, wherein at least three (3) of the 12 high-resolution cameras capture every pixel in the entire playing field.

### Claim 36

A method comprising:
simultaneously performing one of a detection scheme or a tracking scheme on a frame from each of a plurality of videos captured from at least twelve high-resolution cameras to detect a ball used in a sports event; and
performing a multi-camera build to determine a location of the ball in 3D (3-Dimensions) for the frame from each of the plurality of videos using one of detection or tracking results for each camera.

### Claim 37 (depends on 36)

The method of claim 36, wherein the detection scheme comprises:
retrieving the frame for each of the cameras;
retrieving a background image from each of the cameras;
removing the background image from the frame to obtain a foreground mask for each of the cameras;
partitioning the foreground mask into tiles to obtain a partitioned foreground mask for each of the cameras;
performing motion filtering on the partitioned foreground mask to obtain a motion filtered foreground mask for each of the cameras;
performing detection of the ball for each tile in the frame of the motion filtered foreground mask that indicates motion is occurring for each of the cameras; and
collecting detection results from all of the tiles in the frame for each of the cameras.

### Claim 38 (depends on 36)

The method of claim 36, wherein the tracking scheme comprises:
performing tracking-by-detection when the ball was detected in a previous frame, wherein tracking-by-detection comprises only performing detection on a single tile, the single tile being set using a ball center of the previous frame in which the ball was detected or tracked as a tile center the single tile.

### Claim 39 (depends on 36)

The method of claim 36, wherein the multi-camera build comprises:
performing a multi-camera cross validation, the multi-camera cross validation includes sampling the detection results from a set of cameras, wherein the set of cameras are selected using random sampling; and calculating a matching error along an epipolar line for the set of cameras randomly selected;
when the matching error is equal to or greater than a predetermined threshold, a miss or false detection has occurred, the multi-camera build further includes repeating the multi-camera cross validation instructions until the matching error is less than the predetermined threshold; and
when the matching error is less than the predetermined threshold, the multi-camera build further includes determining a 3D ball location using the sample cameras, re-projecting the 3D ball location onto each camera, and determining a distance between a detected position of the ball and a re-projection position of the ball for each camera, wherein if the distance is less than a pre-determined threshold, the results from the detection of the ball are correct using the set of cameras, wherein the multi-camera build further includes placing the set of cameras on an inner list and applying bundle adjustment to get an optimized 3D ball location.

### Claim 40 (depends on 39)

The method of claim 39, wherein the multi-camera build further includes repeating all of the multi-camera build method N times to obtain an optimal result with minimal re-project error.

### Claim 41 (depends on 36)

The method of claim 36, wherein when the multi-camera build is successful, using the tracking scheme in the next frame of each of the videos to locate the ball; and wherein when the multi-camera build is unsuccessful, using the detection scheme in the next frame of each of the videos to locate the ball.

### Claim 42 (depends on 36)

The method of claim 36, further comprising:
projecting the 3D ball location onto each of the results of the plurality of cameras in 2D (2-Dimensions);
performing the detection around a projected position to obtain a more accurate location of the ball for the frame from each of the cameras; and
continuously advancing each of the plurality of real-time videos to a next frame to repeat determining the location of the ball for the next frame until the plurality of real-time videos end.

### Claim 43

At least one computer readable medium, comprising a set of instructions, which when executed by one or more computing devices, cause the one or more computing devices to:
simultaneously perform one of a detection scheme or a tracking scheme on a frame from each of a plurality of real-time videos captured from at least twelve high-resolution cameras to detect a ball used in a sports event; and
perform a multi-camera build to determine a location of the ball in 3D (3-Dimension) for the frame from each of the plurality of real-time videos using one of detection or tracking results for each of the cameras.

### Claim 44 (depends on 43)

The at least one computer readable medium of claim 43, wherein the detection scheme includes instructions, which when executed by the one or more computing devices, cause the one or more computing devices to:
retrieve the frame for each of the cameras;
retrieve a background image from each of the cameras;
remove the background image from the frame to obtain a foreground mask for each of the cameras;
partition the foreground mask into tiles to obtain a partitioned foreground mask for each of the cameras;
perform motion filtering on the partitioned foreground mask to obtain a motion filtered foreground mask for each of the cameras;
perform detection of the ball for each tile in the frame of the motion filtered foreground mask that indicates motion is occurring for each of the cameras; and
collect detection results from all of the tiles in the frame for each of the cameras.

### Claim 45 (depends on 43)

The at least one computer readable medium of claim 43, wherein the tracking scheme includes instructions, which when executed by the one or more computing devices, cause the one or more computing devices to:
perform tracking-by-detection when the ball was detected in a previous frame, wherein instructions to perform tracking-by-detection further comprise instructions to only perform detection on a single tile, the single tile being set using a ball center of the previous frame in which the ball was detected or tracked as a tile center the single tile.

### Claim 46 (depends on 44)

The at least one computer readable medium of claim 44, wherein detection includes detection of the ball using one of YOLO (You Only Look Once), Faster RCNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), and any other object detection technique used to detect small objects.

### Claim 47 (depends on 43)

The at least one computer readable medium of claim 43, wherein the multi-camera build includes instructions, which when executed by the one or more computing devices, cause the one or more computing devices to:
perform a multi-camera cross validation, the multi-camera cross validation including logic to sample the detection results from a set of cameras, wherein the set of cameras are selected using a random sampling method; and calculate a matching error along an epipolar line for the set of cameras randomly selected;
wherein when the matching error is equal to or greater than a predetermined threshold, a miss or false detection has occurred, the logic coupled to the substrate further includes logic to repeat the multi-camera cross validation instructions until the matching error is less than the predetermined threshold; and
wherein when the matching error is less than the predetermined threshold, multi-camera build logic further includes logic to determine a 3D ball location using the sample cameras, re-project the 3D ball location onto each camera, and determine a distance between a detected position of the ball and a re-projection position of the ball for each camera, wherein if the distance is less than a pre-determined threshold, the results from the detection of the ball are correct using the set of cameras, wherein the multi-camera build logic further includes logic to place the set of cameras on an inner list and apply bundle adjustment to get an optimized 3D ball location.

### Claim 48 (depends on 47)

The at least one computer readable medium of claim 47, wherein the multi-camera build further includes instructions, which when executed by the one or more computing devices, cause the one or more computing devices to repeat all of the multi-camera build logic N times to obtain an optimal result with minimal re-project error.

### Claim 49 (depends on 43)

The at least one computer readable medium of claim 43, wherein when the multi-camera build is successful, the tracking scheme is used in the next frame of each of the real-time videos to locate the ball; and wherein when the multi-camera build is unsuccessful, the detection scheme is used in the next frame of each of the real-time videos to locate the ball.

### Claim 50 (depends on 43)

The at least one computer readable medium of claim 43, further comprising instructions, which when executed by the one or more computing devices, cause the one or more computing devices to:
project the 3d ball location onto each of the results of the plurality of cameras in 2D (2-Dimensions);
perform the detection around a projected position to obtain a more accurate location of the ball for the frame from each of the cameras; and
continuously advance each of the plurality of real-time videos to a next frame to repeat the logic to determine the location of the ball for the next frame until the plurality of real-time videos end.
