# US 2022/0046292 A1 · Networked system for real-time computer-aided augmentation of live input video stream

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0046292 A1
- Document ID: US20220046292A1
- Status: Published application
- Kind: A1
- Application number: 17/394,102
- Filed: 2021-08-04
- Published: 2022-02-10
- Earliest claimed: 2020-08-05
- Assignees: Avesha, Inc.
- Applicants: Avesha, Inc.
- Inventors: Raj Nair; Prabhudev Navali; Sudhir Halbhavi; Chin-Cheng Wu
- Web page: https://patentgrep.com/patent/US20220046292A1

## CPC

- H04L 67/1001: for accessing one among a plurality of replicated servers
- H04L 67/1008: based on parameters of servers, e.g. available memory or workload (monitoring of computer activity G06F11/30)
- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- H04L 41/0816: the condition being an adaptation, e.g. in response to network events
- H04L 47/20: Traffic policing
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/101: based on network conditions
- H04L 67/1023: based on a hash applied to IP addresses or costs
- H04L 67/1029: using data related to the state of servers by a load balancer
- H04L 67/34: {involving the movement of software or configuration parameters (network booting or remote initial program loading [RIPL] G06F9/4416)}
- H04L 67/60: Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70)
- H04N 21/2187: Live feed
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations}
- H04N 21/4334: {Recording operations (recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27)}
- H04N 21/466: Learning process for intelligent management, e.g. learning user preferences for recommending movies {(services using the results of monitoring in broadcast systems H04H60/61)}
- H04N 21/47217: {for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks (specific graphical features in visual interfaces H04N21/4312)}
- G06N 3/08: Learning methods
- G06N 20/20: Ensemble learning
- G08B 13/19671: {Addition of non-video data, i.e. metadata, to video stream}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

A networked system for real-time computer-aided augmentation of a live input video stream includes an endpoint system configured and operative to handle the live input video stream and a live augmented video stream and to provide service access to a video augmentation service for the computer-aided augmentation of the live input video stream. The system further includes a backend system providing service orchestration to orchestrate use and operation of the video augmentation service, and an edge inference system coupled to the endpoint system and to the backend system, the edge inference system being configured and co-operative with the endpoint system to provide the video augmentation service on the live input video stream and thereby generate the live augmented video stream, based on the service orchestration by the backend system.

## Claims

### Claim 1

A networked system for real-time computer-aided augmentation of a live input video stream, comprising:
an endpoint system configured and operative to handle the live input video stream and a live augmented video stream and to provide service access to a video augmentation service for the computer-aided augmentation of the live input video stream;
a backend system providing service orchestration to orchestrate use and operation of the video augmentation service; and
an edge inference system coupled to the endpoint system and to the backend system, the edge inference system being configured and co-operative with the endpoint system to provide the video augmentation service on the live input video stream and thereby generate the live augmented video stream, based on the service orchestration by the backend system.

### Claim 2 (depends on 1)

The networked system of claim 1, wherein the video augmentation service is provided in a remote-live manner in which the edge inference system is operative to generate the live augmented video stream and provide it to the endpoint system, and the endpoint system is operative to receive the live augmented video stream from the edge inference system and make local use of the live augmented video stream within the endpoint system.

### Claim 3 (depends on 1)

The networked system of claim 1, wherein the video augmentation service is provided in a local-live manner in which the edge inference system is operative to generate augmentation metadata and provide it to the endpoint system, and the endpoint system is operative to receive the augmentation metadata from the edge inference system and add corresponding features to the live input video stream to generate the live augmented video stream.

### Claim 4 (depends on 1)

The networked system of claim 1, wherein the endpoint system and edge inference system are further configured and co-operative to provide a live review operating mode including playback control by which a user navigates the live augmented video stream for review purposes.

### Claim 5 (depends on 4)

The networked system of claim 4, wherein the live review mode further includes additional controls by which a user adds, reviews and revises annotations in the live augmented video stream.

### Claim 6 (depends on 1)

The networked system of claim 1, wherein the service access is via a client node of the endpoint system, and wherein the endpoint system and edge inference system are further configured and co-operative to provide a live join function by which a user joins an ongoing live session from a node other than the client node.

### Claim 7 (depends on 1)

The networked system of claim 1, wherein the endpoint system and edge inference system are further configured and co-operative to (1) record the live augmented video stream to create a recorded session, and (2) provide an offline review mode by which a user later reviews the recorded session.

### Claim 8 (depends on 1)

The networked system of claim 1, wherein the edge inference system is operative to perform inferencing on the live input video stream to detect an object and to generate corresponding metadata describing features to be added to the live input video stream to identify the detected object in the live augmented video stream, and the endpoint system and edge inference system are further configured and co-operative to use the metadata to generate the live augmented video stream including the features.

### Claim 9 (depends on 8)

The networked system of claim 8, wherein the features include a bounding graphic overlaid at the location of the object and substantially bounding the object in the augmented live video stream.

### Claim 10 (depends on 1)

The networked system of claim 1, wherein the edge inference system employs machine learning model-based inferencing the generate augmentation metadata describing features to be added to the live input video stream to generate the live augmented video stream.

### Claim 11 (depends on 10)

The networked system of claim 10, wherein the machine learning model-based inferencing employs multiple models along with corresponding pre-inferencing and post-inferencing, the models being trained on corresponding distinct training sets, the post-inferencing being operative to ensemble respective results of the models and thereby reduce an effect of over-training of any individual model.

### Claim 12 (depends on 11)

The networked system of claim 11, wherein the models are trained with datasets being labeled for object detection, and a training pipeline is used to train the models for different classes or objects encountered in the live input video stream during operation.

### Claim 13 (depends on 12)

The networked system of claim 12, wherein for a given use there are multiple objects of interests and multiple classes for a given object, and respective models are used for the corresponding classes.

### Claim 14 (depends on 13)

The networked system of claim 13, wherein training produces frozen graphs used during subsequent model-based object detection inference.

### Claim 15 (depends on 10)

The networked system of claim 10, wherein the edge inference system is further configured and operative to receive user voice commands as part of a voice user interface, and to employ natural language processing to identify the user voice commands and take corresponding automated action in providing the live augmented video stream.

### Claim 16 (depends on 1)

The networked system of claim 1, wherein the endpoint system, edge inference system and backend system and configured and co-operative to collect procedure data and metadata including metadata describing objects detected in the live input video stream, and to use the procedure data and metadata to generate corresponding documentation for later offline use.

### Claim 17 (depends on 1)

The networked system of claim 1, wherein the endpoint system includes:
an input device configured and operative to generate the live input video stream; and
a display configured and operative to display the live augmented video stream.

### Claim 18 (depends on 1)

The networked system of claim 1, wherein endpoint system is coupled to the edge inference system by a first network having lower latency than a second network that couples the backend system to the endpoint system and edge inference system.

### Claim 19 (depends on 1)

The networked system of claim 1, wherein the edge inference system is configured to provide the video augmentation service using a collection of microservices distributed among different computing nodes of the edge inference system, the computing nodes including respective containers or virtual machines hosting the respective microservices.

### Claim 20 (depends on 1)

The networked system of claim 1, wherein the video augmentation service is a medical imaging service for automated detection of medically relevant objects in the live input video stream and overlaying object identification features identifying the relevant objects in the live augmented video stream.