# US 2022/0036087 A1 · Computing system and a computer-implemented method for sensing events from geospatial data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036087 A1
- Document ID: US20220036087A1
- Status: Published application
- Kind: A1
- Application number: 17/388,510
- Filed: 2021-07-29
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Optima Sports Systems S.L.
- Applicants: Optima Sports Systems S.L.
- Inventors: Maurici A. LÓPEZ FELIP; Henry S. HARRISON
- Web page: https://patentgrep.com/patent/US20220036087A1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 20/42: {of sport video content}
- G06V 20/53: {Recognition of crowd images, e.g. recognition of crowd congestion}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q 10/10: Office automation; Time management
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20072: Graph-based image processing
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30241: Trajectory
- G06T 2210/62: Semi-transparency
- G06V 20/44: {Event detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer-implemented method and computing system for sensing events and optionally and preferably augmenting a video feed with overlay, comprising in some embodiments a data acquisition module, a sensor module, and optionally and preferably an overlay module. By describing the state of an activity with models that capture the semantics of the activity and comparing this description to a library of event patterns, occurrences of events are detected. Detected events are optionally processed by the overlay module to generate video feed augmented with overlay illustrating said events.

## Claims

### Claim 1

A computer-implemented method for sensing events during a dynamic activity, the method comprising: a data acquisition step and event sensing step, wherein:
a. the data acquisition step comprises the acquisition, by one or more of: video, position-measuring sensors, or digital transfer, a set of geospatial data including the positions of individuals during a time span thereof;
b. the event sensing step comprises a description step and an event detection step, wherein
i. the description step comprises
evaluation of a model graph, comprising a collection of models linked by input-output dependency relationships, with at least one model taking as input at least part of the geospatial data, and
storage by digital means of the model outputs, which together provide a high-level description of the activity; and
ii. the event detection step comprises the matching of the description output with patterns representing event types from a pattern library, outputting an event record whenever a match is found.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the event detection step further comprises:
a. for every timestep in a sample of the geospatial data,
i. and for every event in a predetermined library containing pattern definitions,
the model outputs at that timestep are compared to the criteria in the pattern definition using pattern matching criteria comprising one or more inequality relationships (e.g. greater than, less than) defined with reference to model outputs, and
in case a match is found, an event record is created including the timestep at which it matches, as the start time of the event, and
any matches that have been found in previous timesteps are re-evaluated, and
upon reaching the first timestep when the match conditions do not hold true, the previous timestep is added to the event record as the ending time of the event.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising, prior to the data acquisition step:
a. the acquisition by receiving or capture of video of the activity, of high-definition resolution (720p) or higher, capturing a subset or the entirety of the activity, and
b. a series of computer vision processing steps including
i. detection of individuals in some or all frames of the video;
ii. tracking the detected individuals across frames;
iii. estimating the homography relationship between the coordinate system of the video and that of the real-world space where the activity takes place;
iv. applying homography transformation, using matrix multiplication, at each frame with detected individuals, to transform the detections in video coordinates into real-world coordinates, thereby providing the positions of the individuals.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising: as an initial step of video capture of the activity with one or more cameras, together capturing a subset or the entirety of the activity.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein:
a. the collection of models is represented as a directed acyclic graph,
i. with each model being a node in the graph, and
ii. each edge in the graph representing a dependency relationship pointing from one model to another model whose output is required as an input to the first model, and
b. the models are evaluated in an order respecting their dependency relationships.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the order of evaluation of the models is determined by
a. traversing the directed acyclic graph of models in depth-first order,
b. reversing the resulting order such that the model traversed last is evaluated first, the model traversed first is evaluated last, and so on.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein:
a. the models are classified in two categories:
i. vector models configured to evaluate all timesteps of a sample of the geospatial data simultaneously, and
ii. spatial models that are executed timestep-by-timestep;
b. wherein the method further comprises the evaluation of a directed acyclic graph of the subset of the models that are vector models;
c. following which, at each timestep during the timestep-by-timestep evaluation of the pattern definitions,
i. a subset of spatial events is collected, by the following means: for each pattern definition,
an additional comparison included in the definition determines whether the event is potentially active or not during the current timestep, and
in case the event is potentially active, one or more spatial models defined in the pattern definition is added to the collection for the current timestep, and then
ii. a directed acyclic graph of models is constructed including all the spatial models collected in the previous step and all their dependencies,
iii. the spatial models in said graph are evaluated in an order respecting their dependency relationships, and then
iv. the output of the spatial models at each timestep together with the vector model outputs are used for the subsequent event matching steps.

### Claim 8 (depends on 3)

The computer-implemented method of claim 3, further comprising:
a. the pattern library includes, for each event,
i. rules for assigning roles to individuals during the event, using the same criteria for matching events (e.g., inequalities defined over model output values, in this case referring to models with different outputs per individuals), and when an event is matched, these rules are checked and thereby individual roles assignments are included in the resulting event record, and
ii. a mapping of roles to glyph specifications, wherein glyphs are visual elements;
b. overlay generation steps are performed subsequent to the event matching step, comprising
i. for each timestep from the start to the end time of the event,
a fully transparent image is created, and
for each individual role identified in the event record,
a. the position of the individual in that frame is retrieved from the positional data, and
b. the glyph associated with each individuals' role is drawn over the transparent image, and
the inverse of the homography transformation is applied over the resulting partially transparent image, using matrix multiplication, transforming it from real-world coordinates to video coordinates, and
the resulting partially transparent image is overlaid with the frame from the original video;
ii. all the frames are joined into a video.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein pattern definitions in the pattern library are pattern definitions relate to one of: crowd control, pandemic spread and a sporting event.

### Claim 10

A computing system for sensing events during a dynamic activity, the system comprising: a data acquisition module and a sensor module; wherein:
a. the data acquisition module comprises a video capture module and computer vision tracking system,
i. the video capture module is configured to receive or capture video footage from a dynamic activity, and
ii. the computer vision module is configured to digitally process the video footage to produce trajectories of individuals in the video and a homography mapping between video coordinates and real-world coordinates, and comprises a detection module, a tracking module, and a homography estimation module,
1. the detection module is configured to detect individuals in each frame of the video, and
2. the tracking module is configured to associate detected individuals from one frame to the next, and
3. the homography estimation module is configured to estimate a mapping of video coordinates to real-world coordinates;
b. the sensor module is configured to digitally process data from the data acquisition module and output event records, and comprises a description module and an event detector module,
i. the description module is configured to evaluate a collection of computer-implemented models, with each model taking as input as least one of the positional data determined by the data acquisition module, and the output of one or more other models from the collection,
ii. the event detector comprises a pattern library and a pattern matcher module,
1. the pattern library contains pattern definitions, each containing
a. one rule defining the conditions, in terms of inequality relationships defined over outputs from the model graph, during which the pattern should be considered matched, and
b. zero or more additional rules, each mapped from a role, in terms of outputs from the model graph, determining when an individual is considered as fulfilling that rule, and
c. a mapping of roles to glyph specifications, where glyphs are visual elements, and
2. the pattern matcher module is configured to evaluate rules from the pattern library in the form of inequalities over the model outputs, and
iii. the event detector is configured to produce event records that include start and end times of the matched event determined by the first to the last frame at which the event's pattern in the pattern library was matched, and one or more individual role assignments, determined by applying the rule associated with each role as specified by the entry in the pattern library.

### Claim 11 (depends on 10)

The computing system of claim 10, further comprising an overlay module configured to augment the original video using the positions of the individual in each role of the event record, looking up the glyph associated with each role, applying to each glyph the reverse of the homography transformation estimated by the data acquisition module, and drawing all the glyphs of the event record over the relevant frames of the original video, relevant frames being from the start to the end timestep of each event record.

### Claim 12 (depends on 10)

The computing system of claim 10, wherein the pattern definitions relate to one of: crowd control, pandemic spread and a sporting event.

### Claim 13

A computing system, comprising: at least one processor, and at least one memory; the at least one memory comprising: a collection of models associated with a dynamic activity, a pattern library with pattern definitions associated with the dynamic activity, and instructions which, when executed by the at least one processor, cause the computing system to at least perform the following:
digitally process video footage of the dynamic activity such that the digital processing:
registers coordinates of the video footage and estimate coordinates of a real-world space where the activity takes place as appearing in the video footage, and thereby provide a homography mapping between both coordinates; and
detects individuals in each frame of the video footage, associates the detected individuals between adjacent frames, and use both the associated detection of the individuals and the homography mapping to output a dataset with at least one of positional data of the individuals and trajectories of the individuals;
digitally process the collection of models such that the digital processing:
constructs a vector model graph with the models in the collection;
inputs at least part of the dataset into the vector model graph for evaluation thereof;
constructs, based on the evaluated vector model graph, a subset of spatial models into a graph for each group of N frames of the video footage, with the video footage at least comprising N frames; and
inputs at least part of the dataset into the graph of the subset of spatial models for evaluation thereof, thereby providing spatial model evaluation outputs;
digitally process the pattern definitions such that the digital processing:
compares each pattern definition with each spatial model evaluation output; and
outputs an event record for each spatial model evaluation output that fulfills the pattern definition, the event record including data at least indicative of an event of the pattern definition, and start and end times that the pattern definition is fulfilled according to respective start and end frames of the video footage;
wherein each pattern definition of the pattern library includes data at least indicative of a rule defining conditions in terms of inequality relationships defined over outputs from a model graph.

### Claim 14 (depends on 13)

The computing system of claim 13, wherein the data of each pattern definition is at least further indicative of:
one or more rules for assigning a role to individuals during the event or activity, each rule defining at least one condition in terms of outputs from the model graph for determining fulfillment of the rule by the individual; and
a mapping of roles to glyph specifications, where glyphs are visual elements in the video footage;
wherein the data of the event record is at least further indicative of one or more individual role assignments according to the fulfillment of the one or more rules of the pattern definition.

### Claim 15 (depends on 14)

The computing system of claim 14, wherein the video footage is a first video footage, and wherein the instructions further cause the computing system to at least perform: digitally process the first video footage such that the digital processing outputs second video footage comprising the first video footage with glyphs digitally added thereto, wherein the glyphs are at least added in the video footage according to one or more pattern definitions that are fulfilled, the glyphs being added between respective start and end frames of the pattern definition that is fulfilled and on the individual or individuals in the individual role assignment or assignments indicated on the event record, wherein the glyphs are determined by the mapping of roles to glyph specifications, and wherein the digital processing adds the glyphs in the video footage by applying a reverse of the homography mapping.

### Claim 16 (depends on 13)

The computing system of claim 13, further comprising at least one of: a video camera configured to capture the video footage, and a communications module configured to receive the video footage from a computing device or a video camera.

### Claim 17 (depends on 13)

The computing system of claim 13, wherein digitally processing the video footage from the dynamic activity to provide the homography mapping and the dataset comprises at least one of:
digitally processing two or more video footages of the same activity captured with different perspectives, the two or more video footages comprising the video footage; and
digitally processing the video footage and measurements of one or more position measuring sensors when arranged on at least some individuals participating in the activity;
wherein the at least one of the two or more video footages and the measurements are digitally processed to provide the homography mapping and the dataset.

### Claim 18 (depends on 17)

The computing system of claim 17, further comprising at least one of: a plurality of video cameras adapted to capture the two or more video footages, and the one or more positions measuring sensors.

### Claim 19 (depends on 13)

The computing system of claim 13, wherein the models of the collection are linked by input-output dependency relationships, with at least one model of the collection taking as input positional data of players of the sporting event.

### Claim 20 (depends on 13)

The computing system of claim 13, wherein the pattern definitions relate to one of: crowd control, pandemic spread and a sporting event.
