# US 2017/0255831 A1 · System and method for relevance estimation in summarization of videos of multi-step activities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255831 A1
- Document ID: US20170255831A1
- Status: Published application
- Kind: A1
- Application number: 15/061,463
- Filed: 2016-03-04
- Published: 2017-09-07
- Earliest claimed: 2016-03-04
- Assignees: Xerox Corporation
- Applicants: Xerox Corporation
- Inventors: Edgar A. Bernal; Qun Li; Yun Zhang; Jayant Kumar; Raja Bala
- Web page: https://patentgrep.com/patent/US20170255831A1

## CPC

- H04L 67/04: specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/47: {Detecting features for summarising video content}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- H04L 65/612: for unicast
- H04L 65/765: {intermediate}
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- H04L 69/16: Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and system for identifying content relevance comprises acquiring video data, mapping the acquired video data to a feature space to obtain a feature representation of the video data, assigning the acquired video data to at least one action class based on the feature representation of the video data, and determining a relevance of the acquired video data.

## Claims

### Claim 1

A method for identifying content relevance in a video stream, said method comprising:
acquiring video data;
mapping said acquired video data to a feature space to obtain a feature representation of said video data;
assigning said acquired video data, via the use of a classifier, to at least one action class based on said feature representation of said video data; and
determining a relevance of said acquired video data based on said classifier output.

### Claim 2 (depends on 1)

The method of claim 1 wherein said video data comprises one of:
video acquired with an egocentric or wearable device;
video acquired with a vehicle-mounted device; and
surveillance or third-person view video.

### Claim 3 (depends on 1)

The method of claim 1 wherein determining a relevance of said acquired video data based on said classifier output further comprises:
enforcing a temporal smoothness requirement on at least one relevance score.

### Claim 4 (depends on 1)

The method of claim 1 wherein said extracted features comprise at least one of:
deep features; and
hand-engineered features.

### Claim 5 (depends on 1)

The method of claim 1 wherein said classifier comprises a support vector machine described by parameters w and b, and wherein a magnitude of a classification score |w·xj+b| for an input xj is used to estimate said relevance of said acquired video data.

### Claim 6 (depends on 1)

The method of claim 1 wherein said classifier comprises a neural network wherein estimating said relevance of said acquired video data further comprises estimating a relevance of an input sample xj based on outputs zk where 1≦k≦K, and K is a number of classes.

### Claim 7 (depends on 1)

The method of claim 1 further comprising an offline training stage comprising training said classifier to optimally discriminate between a plurality of different action classes according to their corresponding feature representations.

### Claim 8 (depends on 1)

The method of claim 1 wherein said classifier comprises at least one of:
a support vector machine;
a neural network;
a decision tree;
an expectation-maximization algorithm; and
a k-nearest neighbor clustering algorithm.

### Claim 9 (depends on 1)

The method of claim 1 wherein determining a relevance of said acquired video data based on said classifier output further comprises:
assigning said acquired data a classification confidence score; and
converting said classification confidence score to a relevance score.

### Claim 10

A system for identifying content relevance, said system comprising:
a video acquisition module for acquiring video data;
a processor;
a data bus coupled to said processor; and
a computer-usable medium embodying computer program code, said computer-usable medium being coupled to said data bus, said computer program code comprising instructions executable by said processor and configured for:
mapping said acquired video data to a feature space to obtain a feature representation of said video data;
assigning said acquired video data, via the use of a classifier, to at least one action class based on said feature representation of said video data; and
determining a relevance of said acquired video data based on said classifier output.

### Claim 11 (depends on 10)

The system of claim 10 wherein said video data comprises at least one frame of:
video acquired with an egocentric or wearable device;
video acquired with a vehicle-mounted device; and
surveillance or third-person view video.

### Claim 12 (depends on 10)

The system of claim 10 wherein determining a relevance of said acquired video data based on said classifier output further comprises enforcing a temporal smoothness requirement on at least one relevance score.

### Claim 13 (depends on 10)

The system of claim 10 wherein said extracted features comprises at least one of:
deep features; and
hand-engineered features.

### Claim 14 (depends on 10)

The system of claim 10 wherein said classifier comprises a support vector machine described by parameters w and b, and wherein a magnitude of a classification score |w·xj+b| for an input xj is used to estimate said relevance of said acquired video data.

### Claim 15 (depends on 10)

The system of claim 10 wherein said classifier comprises a neural network wherein estimating said relevance of said acquired video data further comprises estimating a relevance of an input sample xj based on outputs zk where 1≦k≦K, and K is a number of classes.

### Claim 16 (depends on 10)

The system of claim 10 further comprising an offline training stage comprising training said classifier to optimally discriminate between a plurality of different action classes according to their corresponding feature representations.

### Claim 17 (depends on 16)

The system of claim 16 wherein said classifier comprises at least one of:
a support vector machine;
a neural network;
a decision tree;
an expectation-maximization algorithm; and
a k-nearest neighbor clustering algorithm.

### Claim 18 (depends on 8)

The system of claim 8 wherein determining a relevance of said acquired video data based on said classifier output further comprises:
assigning said acquired data a classification confidence score; and
converting said classification confidence score to a relevance score.

### Claim 19

A processor-readable medium storing computer code representing instructions to cause a process for identifying content relevance, said computer code comprising code to:
train a classifier to optimally discriminate between a plurality of different action classes according to said feature representations; and
in an online stage:
acquire video data said video data comprising one of video acquired with an egocentric or wearable device; video acquired with a vehicle-mounted device; and surveillance or third-person view video;
segment said video data into at least one of a series of single frames and a series of groups of frames;
map said acquired video data to a feature space to obtain a feature representation of said video data;
assign said acquired video data, via the use of a classifier, to at least one action class based on said feature representation of said video data; and
assign said acquired data a classification confidence score and convert said classification confidence score to a relevance score to determine a relevance of said acquired video data based on the classifier output.

### Claim 20 (depends on 19)

The processor-readable medium of claim 19 wherein said extracted features comprise at least one of:
deep features, wherein said deep features are learned via the use of at least one of:
a long-short term memory network;
a convolutional network;
an autoencoder; and
a deep Boltzmann machine; and
hand-engineered features, wherein said hand-engineered features comprise at least one of:
scale-invariant features;
interest point and descriptors thereof;
dense trajectories;
histogram of oriented gradients; and
local binary patterns.
