# US 10,321,208 B2 · System and method for enhanced video image recognition using motion sensors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,321,208 B2
- Document ID: US10321208B2
- Status: Granted
- Kind: B2
- Application number: 15/334,131
- Filed: 2016-10-25
- Published: 2017-04-27
- Granted: 2019-06-11
- Earliest claimed: 2015-10-26
- Assignees: Alpinereplay, Inc.
- Applicants: AlpineReplay, Inc.
- Inventors: David J. Lokshin; Anatole M. Lokshin; Claire Louise Roberts-Thomson
- Web page: https://patentgrep.com/patent/US10321208B2

## CPC

- H04N 21/8547: involving timestamps for synchronizing content
- G06F 18/24133: {Distances to prototypes}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G11B 27/3036: {Time code signal}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06V 20/44: {Event detection}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

Disclosed are systems and methods for improving image recognition by using information from sensor data. In one embodiment, the method comprises receiving one or more sensor records, the sensor records representing timestamped sensor data collected by a sensor recording device; selecting an event based on the sensor records; identifying a time associated with the event; retrieving a plurality of timestamped video frames; synchronizing the sensor records and the video frames, wherein synchronizing the sensor records and the video frames comprises synchronizing the timestamped sensor data with individual frames of the timestamped video frames according to a common timeframe; and selecting a subset of video frames from the plurality of timestamped video frames based on the selected event.

## Claims

### Claim 1

A method for improving image recognition by using information from sensor data, the method comprising:
receiving one or more sensor records, the one or more sensor records representing timestamped sensor data collected by a sensor recording device;
selecting an event based on the one or more sensor records;
identifying a time associated with the event;
retrieving a plurality of timestamped video frames;
synchronizing the one or more sensor records and the plurality of timestamped video frames, wherein synchronizing the one or more sensor records and the plurality of timestamped video frames comprises:
synchronizing the timestamped sensor data with individual frames of the plurality of timestamped video frames according to a common timeframe,
classifying the plurality of timestamped video frames using an image recognition to obtain classified video frames, and
identifying a start video frame and end video frame in the classified video frames, the start video frame representing a start of the event and the end video frame representing an end of the event; and
selecting a subset of video frames from the plurality of timestamped video frames based on a start timestamp associated with the start video frame and an end timestamp associated with the end video frame.

### Claim 2 (depends on 1)

The method of claim 1 wherein selecting the subset of video frames further comprises selecting a portion of each of the subset of video frames, the subset of video frames selected based on a determination of a mapping between at least one three-dimensional coordinate and a two-dimensional camera image pixel space.

### Claim 3 (depends on 1)

The method of claim 1 further comprising performing the image recognition on the selected subset of video frames.

### Claim 4 (depends on 3)

The method of claim 3 wherein performing image recognition on the selected subset of video frames comprises:
training a machine learning algorithm using at least one video frame from the subset of video frames, the machine learning algorithm configured to automatically detect participants in a set of video frames;
receiving a second subset of video frames;
providing the second subset of frames to the machine learning algorithm; and
classifying one or more frames of the second subset of frames based on the detection of a participant within the one or more frames by the machine learning algorithm.

### Claim 5 (depends on 1)

The method of claim 1 wherein selecting the subset of video frames from the plurality of timestamped video frames based on the selected event comprises calculating a time period based on a time associated with the selected event and selecting one or more of the plurality of timestamped video frames occurring within the time period.

### Claim 6 (depends on 1)

The method of claim 1 wherein the identifying a time associated with the event further comprises determining a position and duration of the event based on the one or more sensor records.

### Claim 7 (depends on 1)

The method of claim 1 wherein the plurality of timestamped video frames are each associated with metadata wherein the metadata includes a user identifier representing a user present within the plurality of timestamped video frames.

### Claim 8 (depends on 1)

The method of claim 1 further comprising calibrating one or more cameras after synchronizing the sensor records and the video frames, wherein the calibration comprises:
identifying one or more unknown parameters of the operational cameras, the unknown parameters relating to the field of view of the operational cameras;
determining a three-dimensional position of a participant appearing in the subset of video frames based on the sensor record data;
determining a two-dimensional position of one or more pixels containing the participant, and
utilizing camera calibration algorithm to generate the unknown parameters of the operational cameras based on the correspondence between the three-dimensional position and the two dimensional position.

### Claim 9 (depends on 1)

The method of claim 1 further comprising:
calculating a three-dimensional position of the sensor recording device for each frame in the subset of video frames;
determining a set of image areas for each frame in the subset of video frames, the image areas framing a participant equipped with the sensor recording device; and
digitally zooming each frame in the subset of video frames based on the set of image areas.

### Claim 10

A system for improving image recognition by using information from sensor data, the system comprising:
a sensor recording device configured to capture one or more sensor records, the one or more sensor records representing timestamped sensor data collected by the sensor recording device;
one or more cameras configured to record a plurality of timestamped video frames; and
an event processing system configured to:
receive the one or more sensor records, the one or more sensor records representing timestamped sensor data collected by a sensor recording device;
select an event based on the one or more sensor records;
identify a time associated with the event;
retrieve the plurality of timestamped video frames;
synchronize the one or more sensor records and the plurality of timestamped video frames, wherein synchronizing the one or more sensor records and the plurality of timestamped video frames comprises:
synchronizing the timestamped sensor data with individual frames of the plurality of timestamped video frames according to a common timeframe,
classifying the plurality of timestamped video frames using an image recognition to obtain classified video frames,
identifying a start video frame and end video frame in the classified video frames, the start video frame representing a start of the event and the end video frame representing an end of the event; and
select a subset of video frames from the plurality of timestamped video frames based on a start timestamp associated with the start video frame and an end timestamp associated with the end video frame.

### Claim 11 (depends on 10)

The system of claim 10 wherein selecting the subset of video frames further comprises selecting a portion of each of the subset of video frames, the subset of video frames selected based on a determination of a mapping between at least one three-dimensional coordinate and a two-dimensional camera image pixel space.

### Claim 12 (depends on 10)

The system of claim 10 wherein the event processing system is further configured to perform the image recognition on the selected subset of video frames.

### Claim 13 (depends on 12)

The system of claim 12 wherein performing image recognition on the selected subset of video frames comprises:
training a machine learning algorithm using at least one video frame from the subset of video frames, the machine learning algorithm configured to automatically detect participants in a set of video frames;
receiving a second subset of video frames;
providing the second subset of frames to the machine learning algorithm; and
classifying one or more frames of the second subset of frames based on the detection of a participant within the one or more frames by the machine learning algorithm.

### Claim 14 (depends on 10)

The system of claim 10 wherein selecting the subset of video frames from the plurality of timestamped video frames based on the selected event comprises calculating a time period based on a time associated with the selected event and selecting one or more of the plurality of timestamped video frames occurring within the time period.

### Claim 15 (depends on 11)

The system of claim 11 wherein the identifying a time associated with the event further comprises determining a position and duration of the event based on the one or more sensor records.

### Claim 16 (depends on 10)

The system of claim 10 wherein the plurality of timestamped video frames is each associated with metadata wherein the metadata includes a user identifier representing a user present within the plurality of timestamped video frames.

### Claim 17 (depends on 10)

The system of claim 10 wherein the event processing system is further configured to calibrate one or more cameras after synchronizing the sensor records and the video frames, wherein the calibration comprises:
identifying one or more unknown parameters of the operational cameras, the unknown parameters relating to the field of view of the operational cameras;
determining a three-dimensional position of a participant appearing in the subset of video frames based on the sensor record data;
determining a two-dimensional position of one or more pixels containing the participant, and
utilizing camera calibration algorithm to generate the unknown parameters of the operational cameras based on the correspondence between the three-dimensional position and the two dimensional position.

### Claim 18 (depends on 10)

The system of claim 10 wherein the event processing system is further configured to:
calculate a three-dimensional position of the sensor recording device for each frame in the subset of video frames;
determine a set of image areas for each frame in the subset of video frames, the image areas framing a participant equipped with the sensor recording device; and
digitally zoom each frame in the subset of video frames based on the set of image areas.
