# US 10,769,854 B2 · Holographic technology implemented security solution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,854 B2
- Document ID: US10769854B2
- Status: Granted
- Kind: B2
- Application number: 15/379,657
- Filed: 2016-12-15
- Published: 2018-01-18
- Granted: 2020-09-08
- Earliest claimed: 2016-07-12
- Assignees: Tyco Fire & Security GmbH
- Applicants: Tyco Fire & Security GmbH
- Inventors: Robert B. Locke
- Web page: https://patentgrep.com/patent/US10769854B2

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G02B 27/017: {Head mounted}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/1446: {display composed of modules, e.g. video walls}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/205: Parsing
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 5/043: Distributed expert systems; Blackboards
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0269: {based on user profile or attribute}
- G06Q 30/0281: {Customer communication at a business location, e.g. providing product or service information, consulting}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/172: {Classification, e.g. identification}
- G07C 9/257: {electronically (G07C9/26 takes precedence)}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N 7/183: {for receiving images from a single remote source}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 13/344: with head-mounted left-right displays
- H04N 13/398: Synchronisation thereof; Control thereof
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2219/004: Annotating, labelling
- G08B 13/19608: {Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position}
- G08B 13/19656: {Network used to communicate with a camera, e.g. WAN, LAN, Internet}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

Disclosed are techniques that use mixed reality, e.g., augmented reality and virtual reality technologies to improve analysis of security situations as well as retail processes and activity in retail stores. For security these techniques merge the physical world embodied in security systems with the virtual world of policies and analytics. In the retail aspect, these techniques merge the physical world of retail items, displays, and spaces with the virtual world of policies and analytics.

## Claims

### Claim 1

A system, comprising:
server computer systems that receive video feeds from plural fixed video cameras in a facility, the server computer systems including a storage device that stores a program of computing instructions for execution by server computer system, the program comprising instructions configured to cause the server computer system to:
construct a virtual, visual 3D model of the facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility; and
control a mixed reality system comprising a processor device and a memory in communication with the processor device, and a head mounted display device including a stereoscopic 3D display; with the mixed reality system configured to:
render using the head mounted display device the virtual, visual 3D model of the facility;
receive, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;
determine a plurality of commands by translating the images to identify the gestures;
receive a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;
send requests to the server computer system for the video feeds associated with the specified area of the facility;
receive the video feeds;
receive a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and
render using the head mounted display device, the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

### Claim 2 (depends on 1)

The system of claim 1 wherein included with the video feeds are data including location data that specifies a location and an identification data that identifies a surveilling camera.

### Claim 3 (depends on 2)

The system of claim 2 wherein the server computer systems process the data using convention processing analytics for determining either potential of alarm conditions or verification of asserted alarm conditions.

### Claim 4 (depends on 1)

The system of claim 1 wherein personnel access the video feeds in an organized manner by the mixed reality system being configured to:
execute a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.

### Claim 5 (depends on 4)

The system of claim 4 wherein the video feeds are rendered in the stereoscopic 3D display of the corresponding mixed reality system as being affixed to the virtual wall at a fixed location in the display generated, by the corresponding mixed reality system, in response to a user pin command received from the user.

### Claim 6 (depends on 4)

The system of claim 4 further configured to:
receive from a user a selection of the virtual wall of one of the video monitors for closer examination.

### Claim 7 (depends on 6)

The system of claim 6 further configured to:
cause the mixed reality system to enter a mode by which the camera of the head mounted display device captures user gestures that signal the mixed reality system to select the one of the virtual monitors.

### Claim 8 (depends on 7)

The system of claim 7 further configured to:
clear the stereoscopic 3D display upon receiving a selection of the virtual monitor and place the selected virtual monitor in the stereoscopic 3D display.

### Claim 9 (depends on 1)

The system of claim 1 further comprising:
a session manager that controls interactions with the mixed reality system over the Internet using a “session portal” that includes a web service API or a dedicated socket with a transfer protocol.

### Claim 10

A method performed by a mixed reality system comprising a head mounted display device including a stereoscopic 3D display, comprising:
rendering using the head mounted display device a virtual, visual 3D model of a facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility;
receiving, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;
determining a plurality of commands by translating the images to identify the gestures;
receiving a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;
sending requests to a server system for the video feeds associated with the specified area of the facility;
receiving the video feeds;
receiving a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and
using the head mounted display device to render the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

### Claim 11 (depends on 10)

The method of claim 10, wherein the video feeds are data including location data that specifies a location and an identification data that identifies a surveilling camera.

### Claim 12 (depends on 11)

The method of claim 11, wherein the server computer system process the data using convention processing analytics for determining either potential of alarm conditions or verification of asserted alarm conditions.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
executing a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.

### Claim 14 (depends on 13)

The method of claim 13, wherein the video feeds are rendered in the stereoscopic 3D display of the corresponding mixed reality system as being affixed to the virtual wall at a fixed location in the display generated, by the corresponding mixed reality system, in response to a user pin command received from the user.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
receiving from a user a selection of the virtual wall of one of the video monitors for closer examination.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
entering a mode by which the camera of the head mounted display device captures user gestures that signal the mixed reality system to select the one of the virtual monitors.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
clearing the stereoscopic 3D display upon receiving a selection of the virtual monitor and placing the selected virtual monitor in the stereoscopic 3D display.

### Claim 18

A mixed reality system, comprising:
an interface coupled to at least one server computer that receives video feeds from video cameras in a facility;
a storage device that stores a program of computing instructions for execution a processor, the program comprising instructions configured to a head mounted display device including a stereoscopic 3D display to:
render using the head mounted display device a virtual, visual 3D model of the facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility;
receive, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;
determine a plurality of commands by translating the images to identify the gestures;
receive a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;
send request to the server for the video feeds associated with the specified area of the facility;
receive the video feeds;
receive a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and
render using the head mounted display device, the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

### Claim 19 (depends on 18)

The system of claim 18, wherein the video feeds comprise data comprising location data that specifies a location and an identification data that identifies a surveilling camera.

### Claim 20 (depends on 18)

The system of claim 18, wherein the mixed reality system is configured to:
execute a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.