# US 2021/0279475 A1 · Computer vision systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279475 A1
- Document ID: US20210279475A1
- Status: Published application
- Kind: A1
- Application number: 16/321,500
- Filed: 2017-07-31
- Published: 2021-09-09
- Earliest claimed: 2016-07-29
- Assignees: Unifai Holdings Limited
- Applicants: Unifai Holdings Limited
- Inventors: Michael TUSCH; Andrew JACKSON; John DUNNE
- Web page: https://patentgrep.com/patent/US20210279475A1

## CPC

- G06V 20/53: {Recognition of crowd images, e.g. recognition of crowd congestion}
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- G06Q 30/0252: {based on events or environment, e.g. weather or festivals}
- G06Q 30/0261: {based on user location}
- G06Q 30/0271: {Personalized advertisement}
- G06T 7/55: from multiple images
- G06T 7/73: using feature-based methods
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/172: {Classification, e.g. identification}
- H04L 63/0861: {using biometrical features, e.g. fingerprint, retina-scan (cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231)}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer-vision system or engine that (a) generates from a pixel stream a digital representation of a person and (b) determines attributes or characteristics of the person from that digital representation and (c) based on those attributes or characteristics, outputs data to a cloud-based analytics system that enables that analytics system to identify and also to authenticate the person. The attributes or characteristics of the person include their pose, and the system or engine analyses that pose to extract a facial image from a video stream that is the best facial image for use by the cloud-based analytics system to identify and authenticate the person. The computer-vision system or engine outputs the facial image to the cloud-based analytics system, but does not output the full-frame real-time video to the cloud-based analytics system.

## Claims

### Claim 1

A computer-vision system or engine that (a) generates from a pixel stream a digital representation of a person and (b) determines attributes or characteristics of the person from that digital representation and (c) based on those attributes or characteristics, outputs data to a cloud-based analytics system that enables that analytics system to identify and also to authenticate the person.

### Claim 2 (depends on 1)

The computer-vision system or engine of claim 1 in which the attributes or characteristics of the person include their pose, and the system or engine analyses that pose to extract a facial image from the pixel stream that is the best facial image for use by the cloud-based analytics system to identify and authenticate the person.

### Claim 3 (depends on 2)

The computer-vision system or engine of claim 2 in which the computer-vision system or engine outputs the facial image to the cloud-based analytics system, but does not output the full-frame real-time video to the cloud-based analytics system.

### Claim 4 (depends on 1)

The computer-vision system or engine of claim 1, (i) in which the data output from the engine is indexed in real-time by the cloud-based analytics system with the identity of persons; or (ii) in which the computer-vision system or engine is implanted in an ASIC or SoC located in a camera, such as a security camera; or (iii) in which the computer-vision system or engine operates in real-time, directly processing raw image sensor data; or (iv) in which the cloud-based analytics system controls a digital advertising system or digital signage to provide one-to-one real time marketing to individuals whom it has recognised.

### Claim at position 5

5-8. (canceled)

### Claim 9 (depends on 1)

The computer-vision system or engine of claim 1, (i) in which the computer-vision system or engine dynamically preserves resolution of selective areas (or region of interest) within each frame; or (ii) in which the computer-vision system or engine dynamically adjusts the effective frame rate based on content and manages streaming level based on user control; or (iii) that outputs a real-time stream of metadata from the pixel stream, the metadata describing the instantaneous attributes or characteristics of each object in the scene it has been trained to search for; or (iv) that outputs a real-time stream of metadata from the pixel stream, the metadata describing the instantaneous attributes or characteristics of each object in the scene it has been trained to search for, and that sends or uses those attributes or characteristics to enable one or more networked devices or sensors to be controlled.

### Claim at position 7

10-12. (canceled)

### Claim 13 (depends on 1)

The computer-vision system or engine of claim 1, (i) that can detect multiple people in a scene and continuously track or detect one or more of their: trajectory, pose, gesture, identity: or (ii) that can infer or describe a person's behaviour or intent by analysing one or more of the trajectory, pose, gesture, identity of that person; or (iii) that performs real-time virtualisation of a scene, extracting objects from the scene and grouping their virtualised representations together; or (iv) that applies feature extraction and classification to find objects of known characteristics in each video frame or applies a convolutional or recurrent neural network or another object detection algorithm to do so; or (v) that detects people by extracting independent characteristics including one or more of the following: the head, head & shoulders, hands and full body, each in different orientations, to enable an individual's head orientation, shoulder orientation and full body orientation to be independently evaluated for reliable people tracking; or (vi) that continuously monitors the motion of individuals in the scene and predicts their next location to enable reliable tracking even when the subject is temporarily lost or passes behind another object; or (vii) that contextualizes individual local representations to construct a global representation of each person as they move through an environment of multiple sensors in multiple locations; or (viii) uses data from multiple sensors, each capturing different parts of an environment, to track and show an object moving through that environment and to form a global representation that is not limited to the object when imaged from a single sensor; or (ix) where the approximate location of an object in 3D is reconstructed using depth/distance estimation to assist accuracy of tracking and construction of the global representation from multiple sensors.

### Claim at position 9

14-21. (canceled)

### Claim 22 (depends on 1)

The computer-vision system or engine of claim 1, (i) that operates as an interface to enable control of multiple networked computer-enabled sensors and devices in the smart home or office; or (ii) where the digital representation conforms to an API; or (iii) where the digital representation includes feature vectors that define the appearance of a generalized person; or (iv) where the digital representation is used to display a person as a standardised shape.

### Claim at position 11

23-25. (canceled)

### Claim 26 (depends on 1)

The computer-vision system or engine of claim 1 where the digital representation is used to display a person as a graphical symbolic or simplified representation of a person.

### Claim 27 (depends on 26)

The computer-vision system or engine of claim 26, (i) where the symbolic or simplified representation is a flat or 2-dimensional shape including head, body, arms and legs; or (ii) where the symbolic or simplified representations of different people are distinguished using different colours; or (iii) where the symbolic or simplified representation is an avatar.

### Claim at position 14

28-29. (canceled)

### Claim 30 (depends on 1)

The computer-vision system or engine of claim 1, (i) where the digital representation includes feature vectors that define the appearance of a specific person; or (ii) where the digital representation of a person is used to analyse, or enable the analysis of one or more of trajectory, pose, gesture and identity of that person and smart home devices can respond to and predict the person's intent and/or needs based on that analysis; or (iii) where the digital representation is not an image and does not enable an image of a person to be created from which that person can be recognised; or (iv) that does not output continuous or streaming video but instead metadata that defines various attributes of individual persons.

### Claim at position 16

31-33. (canceled)

### Claim 34 (depends on 1)

The computer-vision system or engine of claim 1, that outputs continuous or streaming video and also metadata that defines various attributes of individual persons.

### Claim 35 (depends on 34)

The computer-vision system or engine of claim 34, (i) where the characteristics or attributes include one or more of trajectory, pose, gesture, identity; or (ii) where the characteristics or attributes include each of trajectory, pose, gesture, and identity.

### Claim 36

(canceled)

### Claim 37 (depends on 1)

The computer-vision system or engine of claim 1, (i) that works with standard images sensors working with chip-level systems that generate real-time data that enables a digital representation of people or other objects to be created; or (ii) that works with IP cameras to form a real-time metadata stream to accompany the output video stream providing an index of video content frame by frame; or (iii) that works with smart sensors that use visual information, but never form imagery or video at a hardware level; or (iv) that builds a virtualized digital representation of each individual in the home, comprising each individual's: Trajectory around the home, including for example the actions of standing and sitting; Pose, for example in which direction the person is facing, and/or in which direction they are looking; Gesture, for example motions made by the person's hands; and Identity, namely the ability to differentiate between people and assign a unique identity (e.g. name) to each person; or (v) that is programmed to understand a wide range of behaviours from the set: counting the number of people in the room, understanding people's pose, identifying persons using facial recognition data, determining where people are moving from/to, extracting specific gestures by an identified individual.

### Claim at position 21

38-41. (canceled)

### Claim 42 (depends on 1)

The computer-vision system or engine of claim 1, where the data rate of the data sent from the computer-vision system or engine is throttled up or based on event-triggering.

### Claim 43 (depends on 1)

The computer-vision system or engine of claim 1, that where multiple computer-vision systems or engines send their data to a hub that stores and analyses that data and enables a digital representation of a person to be constructed from computer-vision systems with both shared and differing fields of view, tracking that person and also recognizing that person.

### Claim 44 (depends on 43)

The computer-vision system or engine of claim 43 where the hub exposes an open, person-level digital representation API, enabling various appliances to use and to be controlled in dependence on the data encoded in the API.

### Claim 45 (depends on 1)

The computer-vision system or engine of claim 1, (i) where the digital representation is created locally at a computer-vision system, or at a hub, or in the cloud, or distributed across computer-vision systems and one or more hubs and the cloud; or (ii) where the digital representation is included in a ‘track record’ that uses the reformatting of real-time metadata into a per-object (e.g. per-person) record of one or more of their trajectory, pose and identity of that object.

### Claim at position 26

46-47. (canceled)

### Claim 48 (depends on 1)

The computer-vision system or engine of claim 1, (i) where the digital representation includes an estimate or measurement of depth or distance from the sensor of a person or object or part of the environment; or (ii) where the digital representation includes an estimate or measurement of depth or distance from the sensor of a person or object or part of the environment, and where depth sensing uses a calibration object of approximately known size, or stereoscopic cameras or structured light; or (iii) where the digital representation includes facial recognition data; or (iv) where sensor metadata is fed into a hub, gateway or controller that pushes events to smart devices in a network as specific commands, and differentiates the events created on a per service basis to allow each service to receive different data that is relevant to their service from the group of sensors as a single intelligent sensor; or (v) where event streams are sent to cloud analytics apps such as for example cloud-based data monitoring, data gathering or learning service; or (vi) where event streams are sent to cloud analytics apps such as, for example, cloud-based data monitoring, data gathering or learning service, and where an event subscription service, to which a system controller subscribes, receives event notifications and data from the devices or sensors.

### Claim at position 28

49-63. (canceled)

### Claim 64 (depends on 1)

The computer-vision system or engine of claim 1, that is localised in one or more of the following: (a) an edge layer that processes raw sensor data; (b) an aggregation layer that provides high level analytics by aggregating and processing data from the edge layer in the temporal and spatial domains; (c) a service layer that handles all connectivity to one or more system controllers and to the end customers for configuration of their home systems and the collection and analysis of the data produced.

### Claim 65

(canceled)

### Claim 66

An appliance that includes a sensor that in turn includes an embedded computer-vision engine that (a) generates from a pixel stream a digital representation of a person and (b) determines attributes or characteristics of the person from that digital representation and (c) based on those attributes or characteristics, outputs data to a cloud-based analytics system that enables that analytics system to identify and also to authenticate the person.

### Claim at position 32

67-68. (canceled)

### Claim 69

Chip-level firmware that provides a computer vision engine that (a) generates from a pixel stream a digital representation of a person and (b) determines attributes or characteristics of the person from that digital representation and (c) based on those attributes or characteristics, outputs data to a cloud-based analytics system that enables that analytics system to identify and also to authenticate the person.

### Claim at position 34

70-95. (canceled)
