# US 2019/0244386 A1 · Directional impression analysis using deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244386 A1
- Document ID: US20190244386A1
- Status: Published application
- Kind: A1
- Application number: 16/388,772
- Filed: 2019-04-18
- Published: 2019-08-08
- Earliest claimed: 2017-08-07
- Assignees: Standard Cognition, Corp.
- Applicants: Standard Cognition, Corp
- Inventors: JORDAN E. FISHER; Michael S. Suswal; Warren Green
- Web page: https://patentgrep.com/patent/US20190244386A1

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06F 16/53: Querying
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/197: {Matching; Classification}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30201: Face
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and techniques are provided for detecting gaze direction of subjects in an area of real space. The system receives a plurality of sequences of frames of corresponding fields of view in the real space. The system uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the subject in the area of real space over time. The system includes logic having access to a database identifying locations of items in the area of real space. The system identifies items in the area of real space matching the identified gaze directions of the identified subject.

## Claims

### Claim 1

A system for detecting gaze direction of subjects in an area of real space, comprising:
a processing system receiving a plurality of sequences of frames of corresponding fields of view in the real space, including:
logic that uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the subject in the area of real space over time, and
logic having access to a database identifying locations of items in the area of real space, that identifies items in the area of real space matching the identified gaze directions of the identified subject.

### Claim 2 (depends on 1)

The system of claim 1, further including logic that calculates distances of the identified subject from items having locations matching the identified gaze directions and said logic that identifies times utilizes the calculated distances.

### Claim 3 (depends on 1)

The system of claim 1, further including logic that determine lengths of time for which the subject maintains respective gaze directions and stores the determined lengths of times.

### Claim 4 (depends on 1)

The system of claim 1, further including logic that stores information including subject identifiers and item identifiers for the identified gaze directions.

### Claim 5 (depends on 1)

The system of claim 1, wherein the logic that uses sequences of frames in a plurality of sequences of frames to identify locations of the identified subject and gaze directions includes:
image recognition engines, which process the sequences of frames to generate corresponding arrays of joint data structures and to identify sets of joints as subjects in the real space; and
logic, using joints in the sets of joints, to determine the gaze directions of the subject.

### Claim 6 (depends on 1)

The system of claim 1, wherein the logic that uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the identified subject includes:
logic that defines the gaze directions as planes orthogonal to a floor in the area of real space, the plane including a vector corresponding to the gaze direction of the identified subject; and
the logic that identifies items in the area of real space matching the identified gaze directions of the subject identifies items mapped to inventory locations intersected by the plane.

### Claim 7 (depends on 1)

The system of claim 1, further including inventory display structures in the area of real space, the inventory display structures comprising inventory locations matched with cells in the area of real space in the database, and a database identifying inventory locations of items in the area of real space with cells in the area of real space.

### Claim 8 (depends on 3)

The system of claim 3, processing system including logic to accumulate a plurality of data sets each including locations of the identified subject, gaze directions of the subject, items in area of real space matching the identified gaze directions, distances of identified subjects from items, the determined lengths of times and store the accumulated data sets in the database configured for use to analyze the data to correlate a particular element of a plurality of data sets with other elements in the plurality of data sets related to the particular element.

### Claim 9 (depends on 8)

The system of claim 8, further including, logic to correlate a particular item of a plurality of data sets with identified subjects in the plurality of data sets related to the particular item and display on a user interface a graphical construct indicating correlations of the particular item to the plurality of identified subjects in multiple locations in the area of real space.

### Claim 10 (depends on 9)

The system of claim 9, wherein the logic that correlates the particular item of the plurality of data sets with identified subjects in the plurality of data sets related to the particular item includes logic that determines a count of the identified subjects related to the particular item.

### Claim 11 (depends on 8)

The system of claim 8, further including logic to correlate a particular identified subject of the plurality of data sets with items in the plurality of data sets related to the particular subject and display on a user interface a graphical construct indicating the lengths time for which items are in the gaze directions of the particular identified subject.

### Claim 12

A method of detecting gaze direction of subjects in an area of real space, the method including:
receiving a plurality of sequences of frames of corresponding fields of view in the real space;
identifying locations of an identified subject and gaze directions of the subject in the area of real space over time using sequences of frames in the plurality of sequences of frames, and
identifying items in the area of real space matching the identified gaze directions of the identified subject.

### Claim 13 (depends on 12)

The method of claim 12, the method further including:
calculating distances of the identified subject from items having locations matching the identified gaze directions and storing the calculated distances.

### Claim 14 (depends on 12)

The method of claim 12, the method further including:
determining lengths of time for which the subject maintains respective gaze directions and stores the determined lengths of times.

### Claim 15 (depends on 12)

The method of claim 12, the method further including:
storing information including subject identifiers and item identifiers for the identified gaze directions.

### Claim 16 (depends on 12)

The method of claim 12, wherein identifying locations of an identified subject and gaze directions of the subject in the area of real space over time using sequences of frames in the plurality of sequences of frames further includes:
defining the gaze directions as planes orthogonal to a floor in the area of real space, the plane including a vector corresponding to the gaze direction of the identified subject; and
the identifying items in the area of real space matching the identified gaze directions of the identified subject includes identifying items mapped to inventory locations intersected by the plane.

### Claim 17

A non-transitory computer readable storage medium impressed with computer program instructions to detect gaze direction of subjects in an area of real space, the instructions, when executed on a processor, implement a method comprising:
receiving a plurality of sequences of frames of corresponding fields of view in the real space;
identifying locations of an identified subject and gaze directions of the subject in the area of real space over time using sequences of frames in the plurality of sequences of frames, and
identifying items in the area of real space matching the identified gaze directions of the identified subject.

### Claim 18

A system for detecting gaze direction of subjects in an area of real space, comprising:
a processing system receiving a plurality of sequences of frames of corresponding fields of view in the real space, including:
logic that uses sequences of frames in the plurality of sequences of frames to identify locations of an identified subject includes logic that processes the sequences of frames to generate corresponding arrays of joint data structures and to identify sets of joints as subjects in the real space;
logic using joints in the sets of joints, to identify gaze directions of the subject; and
logic having access to a database identifying locations of items in the area of real space, that identifies items in the area of real space matching the identified gaze directions of the identified subject.

### Claim 19 (depends on 18)

The system of claim 18, wherein the logic, that uses joints in the sets of joints, to determine the gaze directions of the subject, includes logic to use neck and shoulder joints of the identified subject to determine the gaze directions of the subject.

### Claim 20 (depends on 18)

The system of claim 18, wherein the logic, that uses joints in the sets of joints, to determine the gaze directions of the subject, includes logic that defines a vector corresponding to the gaze direction of the identified subject as normal to a plane formed by neck and shoulder joints of the identified subject.

### Claim 21 (depends on 20)

The system of claim 20, further comprising:
logic that defines gaze directions as planes orthogonal to a floor in the area of real space, the plane including the vector corresponding to the gaze direction of the identified subject; and
the logic that identifies items in the area of real space matching the identified gaze directions of the subject identifies items mapped to inventory locations intersected by the plane.
