# US 2025/0157259 A1 · Activity recognition systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157259 A1
- Document ID: US20250157259A1
- Status: Published application
- Kind: A1
- Application number: 19/026,300
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2014-06-17
- Assignees: Nant Holdings IP, LLC
- Applicants: Nant Holdings IP, LLC
- Inventors: Kamil Wnuk; Nicholas J. Witchey
- Web page: https://patentgrep.com/patent/US20250157259A1

## CPC

- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- B25J 9/1697: {Vision controlled systems}
- G06F 16/2228: {Indexing structures}
- G06F 16/23: Updating
- G06F 16/24578: {using ranking}
- G06F 16/285: {Clustering or classification}
- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 11/26: Drawing of charts or graphs
- G06V 10/426: Graphical representations
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An activity recognition system is disclosed. A plurality of temporal features is generated from a digital representation of an observed activity using a feature detection algorithm. An observed activity graph comprising one or more clusters of temporal features generated from the digital representation is established, wherein each one of the one or more clusters of temporal features defines a node of the observed activity graph. At least one contextually relevant scoring technique is selected from similarity scoring techniques for known activity graphs, the at least one contextually relevant scoring technique being associated with activity ingestion metadata that satisfies device context criteria defined based on device contextual attributes of the digital representation, and a similarity activity score is calculated for the observed activity graph as a function of the at least one contextually relevant scoring technique, the similarity activity score being relative to at least one known activity graph.

## Claims

### Claim at position 1

1-28. (canceled)

### Claim 29

An activity recognition method using at least one processor and at least one memory, the method comprising:
receiving a digital representation comprising at least kinematic data of an observable sports or driving activity of at least one object;
generating from the digital representation, using at least one feature detection algorithm, features related to the observable sports or driving activity;
establishing an observed activity data object based on the features;
determining a similarity for the observed activity data object relative to at least one known activity data object based on a context relevant to the features, wherein each known activity data object represents at least one known activity having features based on a broadcast or recorded event;
accessing an activity recognition results set based on the similarity, the activity recognition results set comprising at least a comparison of choreographed, broadcast, or recorded activity with kinematic data; and
triggering an action based on the activity recognition results set, wherein the action includes at least one training action.

### Claim 30 (depends on 29)

The method of claim 29, wherein the at least one known activity includes at least two temporal features based on the broadcast or recorded event.

### Claim 31 (depends on 30)

The method of claim 30, wherein the at least one known activity includes a motion and a tracked location based on the broadcast or recorded event.

### Claim 32 (depends on 29)

The method of claim 29, wherein the at least one known activity further includes a verbal command.

### Claim 33 (depends on 29)

The method of claim 29, wherein the at least one known activity further includes at least one of a body movement, an interaction, a hand movement, and a facial movement or expression.

### Claim 34 (depends on 29)

The method of claim 29, wherein the at least one processor comprises an activity recognition device.

### Claim 35 (depends on 29)

The method of claim 29, wherein the context comprises a virtual machine environment.

### Claim 36 (depends on 35)

The method of claim 35, wherein the virtual machine environment comprises a virtual sports or driving environment.

### Claim 37 (depends on 36)

The method of claim 36, wherein the at least one object comprises actions or interactions among multiple objects.

### Claim 38 (depends on 37)

The method of claim 37, wherein the multiple objects are spaced apart from one another.

### Claim 39 (depends on 29)

The method of claim 29, wherein the at least one known activity data object further comprises at least a part of a template for interactions.

### Claim 40 (depends on 39)

The method of claim 39, wherein the at least a part of a template for interactions is configured for an online interaction involving at least one sports or driving activity.

### Claim 41 (depends on 29)

The method of claim 29, wherein the step of triggering an action comprises triggering a virtual action in a sports or driving environment.

### Claim 42 (depends on 29)

The method of claim 29, wherein the context comprises location data.

### Claim 43 (depends on 29)

The method of claim 29, wherein the digital representation further comprises at least one of image data, still image data, accelerometer data, tactile data, kinesthetic data, temperature data, 3D registration data, and radio signal or wireless data.

### Claim 44 (depends on 43)

The method of claim 43, wherein the image data comprises at least one of ultrasound, infrared, and visible spectrum data.

### Claim 45 (depends on 29)

The method of claim 29, wherein the at least one feature detection algorithm includes one of the following: a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG), and Transform Invariant Low-rank Textures (TILT) feature detection algorithm.

### Claim 46 (depends on 29)

The method of claim 29, wherein the digital representation comprises video and audio data captured over a time period or within a time frame.

### Claim 47 (depends on 29)

The method of claim 29, wherein at least some of the features describe a temporal or spatial relationship among comparable events in time.

### Claim 48 (depends on 29)

The method of claim 29, further comprising:
converting aspects of the digital representation to an observed activity graph; and
comparing the observed activity graph to known activity graphs.

### Claim 49 (depends on 29)

The method of claim 29, further comprising determining contextual relevance based on ingestion metadata.

### Claim 50 (depends on 49)

The method of claim 49, further comprising selecting the ingestion metadata used to determine contextual relevance based on one or more domain-specific attributes.

### Claim 51 (depends on 49)

The method of claim 49, wherein the ingestion metadata conforms to a defined attribute namespace.

### Claim 52 (depends on 29)

The method of claim 29, further comprising:
recognizing one or more objects in the digital representation using at least some of the plurality of features; and
retrieving object information related to the one or more recognized objects.

### Claim 53 (depends on 52)

The method of claim 52, further comprising using the object information to determine contextual relevance.

### Claim 54 (depends on 29)

The method of claim 29, wherein the similarity includes at least one of a Euclidean distance, linear kernel, polynomial kernel, Chi-squared kernel, Cauchy kernel, histogram intersection kernel, Hellinger's kernel, Jensen-Shannon kernel, hyperbolic tangent (sigmoid) kernel, rational quadratic kernel, multiquadratic kernel, inverse multiquadratic kernel, circular kernel, spherical kernel, wave kernel, power kernel, log kernel, spline kernel, Bessel kernel, generalized T-Student kernel, Bayesian kernel, wavelet kernel, radial basis function (RBF), exponential kernel, Laplacian kernel, ANOVA kernel and B-spline kernel function.

### Claim 55 (depends on 29)

The method of claim 29, further comprising selecting the similarity according to a data modality.

### Claim 56 (depends on 29)

The method of claim 29, wherein the similarity reflects a relative confidence of data from each of a plurality of sensing modalities.

### Claim 57 (depends on 29)

The method of claim 29, wherein the activity recognition results set comprises at least one of an activity identifier, a search result, a classification, a recommendation, an anomaly, a warning, a segmentation, a command, a ranking, context relevant information, content information, and an action prediction.

### Claim 58 (depends on 57)

The method of claim 57, wherein the action prediction is based on variations of known activities.

### Claim 59 (depends on 29)

The method of claim 29, wherein triggering the action comprises executing a command.

### Claim 60 (depends on 29)

The method of claim 29, wherein triggering the action comprises generating an alert.

### Claim 61 (depends on 29)

The method of claim 29, wherein the kinematic data relates to a volumetric space.

### Claim 62

An activity recognition device having a processor, wherein, upon execution of software instructions stored on a non-transitory computer readable medium, the processor is configured to:
receive a digital representation comprising at least kinematic data of an observable sports or driving activity of at least one object;
generate from the digital representation, using at least one feature detection algorithm, features related to the observable sports or driving activity;
establish an observed activity data object based on the features;
determine a similarity for the observed activity data object relative to at least one known activity data object based on a context relevant to the features, wherein each known activity data object represents at least one known activity having features based on a broadcast or recorded event;
access an activity recognition results set based on the similarity, the activity recognition results set comprising at least a comparison of choreographed, broadcast, or recorded activity with kinematic data; and
trigger an action based on the activity recognition results set, wherein the action includes at least one training action.

### Claim 63

A non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to:
receive a digital representation comprising at least kinematic data of an observable sports or driving activity of at least one object;
generate from the digital representation, using at least one feature detection algorithm, features related to the observable sports or driving activity;
establish an observed activity data object based on the features;
determine a similarity for the observed activity data object relative to at least one known activity data object based on a context relevant to the features, wherein each known activity data object represents at least one known activity having features based on a broadcast or recorded event;
access an activity recognition results set based on the similarity, the activity recognition results set comprising at least a comparison of choreographed, broadcast, or recorded activity with kinematic data; and
trigger an action based on the activity recognition results set, wherein the action includes at least one training action.
