# US 10,768,704 B2 · System and method for modulating a peripheral device based on an unscripted feed using computer vision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,768,704 B2
- Document ID: US10768704B2
- Status: Granted
- Kind: B2
- Application number: 16/196,254
- Filed: 2018-11-20
- Published: 2019-05-30
- Granted: 2020-09-08
- Earliest claimed: 2015-03-17
- Applicants: Whirlwind VR, Inc.
- Inventors: Timothy Chia-Chieh Sun; Matthew Ryan Gilg
- Web page: https://patentgrep.com/patent/US10768704B2

## CPC

- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G05B 15/02: electric
- G06F 1/1605: {Multimedia displays, e.g. with integrated or attached speakers, cameras, microphones}
- G06F 1/181: {Enclosures (for portable computers G06F1/1613)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/90: Determination of colour characteristics
- G06V 20/20: in augmented reality scenes
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G10L 25/21: the extracted parameters being power information
- G10L 25/51: for comparison or discrimination
- A47C 7/72: Adaptations for incorporating lamps, radio sets, bars, telephones, ventilation, heating or cooling arrangements or the like
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 20/44: {Event detection}
- G10L 25/48: specially adapted for particular use
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- A47C: CHAIRS; SOFAS; BEDS

## Abstract

A system for processing at least one of an audio or video input for non-scripted modulation of at least one peripheral device, comprising: the at least one peripheral device in physical contact with at least one user or free from the at least one user and in communication with at least a first device playing at least one of an original programming feed or live feed unscripted with modulation triggers; a processor; a memory element coupled to the processor; a program executable by the processor to: recognize at least one of the audio or video input from the at least one of the original programming feed or live feed, and determine for at least one tagged event, at least one of a pixel color score, a pixel velocity score, an event proximity score or an audio score; and convert the at least one scored event into at least one of an output command that triggers or controls a modulation effect of the at least one peripheral device in physical contact or free from the user in communication with the at least the first device playing the at least one of the original programming feed or live feed, thereby enabling modulation of the at least one peripheral device based on any programming comprising at least one of an audio or video input and not requiring scripted modulation triggers.

## Claims

### Claim 1

A system for processing at least one of an audio or video input for non-scripted modulation of at least one peripheral device, said system comprising:
the at least one peripheral device in physical contact with at least one user or free from the at least one user and in communication with at least a first device playing at least one of an original programming feed or live feed unscripted with modulation triggers;
a processor;
a memory element coupled to the processor; a program executable by the processor to:
recognize at least one of the audio or video input from the at least one of the original programming feed or live feed, and determine for at least one tagged event, at least one of a pixel color score, a pixel velocity score, or an audio score, wherein the pixel color score is derived from a calculated average of at least one of hue, color value, luminescence, or saturation from a captured screen buffer, wherein the pixel velocity score is derived from a calculated coefficient of a per-frame and per-range delta for at least one of a hue, color value, luminescence, or saturation for a captured screen buffer, or wherein the audio score is derived from a calculated coefficient related to changes in a frequency spectrum from a captured audio buffer; and
convert the at least one score into at least one of an output command that triggers or controls a modulation effect of the at least one peripheral device in physical contact or free from the user in communication with the at least the first device playing the at least one of the original programming feed or live feed, thereby enabling modulation of the at least one peripheral device based on any programming comprising at least one of an audio or video input and not requiring scripted modulation triggers.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor further determines an event proximity score of the tagged event by determining a distance from any one a selected target zone comprising the event and a selected destination zone comprising or representing a user, within a matrix of zones occupying an entire field of at least one of a view and sound.

### Claim 3 (depends on 1)

The system of claim 1, wherein the at least first device is coupled to a display screen for user viewing and said at least first device is in communication to the at least one peripheral device in physical contact with the at least one user or free from the at least one user for triggering at least one of an actuation or haptic effect.

### Claim 4 (depends on 3)

The system of claim 3, wherein the at least one peripheral device is a device for controlling viewing operation of the at least one of the original programming feed or live feed displayed on the screen coupled to the at least first device.

### Claim 5 (depends on 1)

The system of claim 1, wherein at least one of the audio or video input is from at least the first device that outputs an audio and, or video signal during operation and is in its original programming feed form for triggering or controlling at least one of an actuation or haptic effect from a peripheral device in physical contact with the at least one user or free from the at least one user, wherein the at least first device and peripheral device are the same device.

### Claim 6 (depends on 3)

The system of claim 3, further comprising a plurality of peripheral devices with at least one in physical contact with the at least one user or free from the at least one user and in communication to the same original programming feed or live feed from the at least first device.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor tags at least one event for scoring by at least one of, motion, shape, color or sound.

### Claim 8 (depends on 7)

The system of claim 7, wherein the processor further comprises an a/v recognition block that tags at least one event for scoring by at least one of, motion, shape, color or sound.

### Claim 9 (depends on 1)

The system of claim 1, wherein the processor applies a scoring rule, wherein any of a tagged and scored event is a threshold-grade scored event, and said threshold-grade scored event is converted into an output command.

### Claim 10 (depends on 1)

The system of claim 1, further comprising at least one of a feed-forward or back-propagated neural network trained to trigger a modulation effect based on any one of, or combination of, a stored data input, stored tagged event, stored coefficient value, stored event proximity score value, stored pixel color score value, stored pixel velocity score value, stored audio score value, or output command.

### Claim 11 (depends on 10)

The system of claim 10, wherein at least one of the feed-forward or back-propagated neural network uses a series of externally captured buffers containing known audio-visual sources to aid in real-time recognition of the audio or video input by using a probabilistic approach to determine presence in the captured buffer.

### Claim 12 (depends on 1)

The system of claim 1, wherein the at least first device is playing a programming feed comprising audio signals with a frequency imperceptible to a human ear (sub-audio), whereby the sub-audio signal triggers or controls the modulation effect from the at least one peripheral device in physical contact with the user or free from the user.

### Claim 13 (depends on 1)

The system of claim 1, wherein the at least first device is at least one of a computing, gaming, streaming, television, or audio or video playback device playing an original programming feed and the at least one peripheral device in physical contact with the at least one user is at least one of a haptic triggering glove, thimble, vest, jacket, wearable, watch, mobile phone, tablet, joystick, toy, erotic toy, game controller, interactive seat, head phones, or head gear.

### Claim 14 (depends on 1)

The system of claim 1, wherein the at least first device is at least one of a computing, gaming, streaming, television, or audio or video playback device playing an original programming feed and the peripheral device is free from the user and is at least one of a stand-alone haptic tower, heat lamp, fan, light source, light fixture, house alarm, or IoT hub.

### Claim 15 (depends on 1)

The system of claim 1, wherein the at least first device is at least one of a camera, microphone, sensor, or audio or video capture for playing a live feed and the peripheral device in physical contact with the user is at least one of a mobile phone, biomedical tool, erotic toy, steering wheel, or automobile pedal.

### Claim 16 (depends on 1)

The system of claim 1, wherein the at least first device is at least one of a camera, microphone, sensor, or audio or video capture for playing a live feed and the peripheral device free from the user is at least one of an automobile alarm, house alarm, stand-alone haptic tower, heat lamp, fan, light source, light fixture, thermostat, or IoT hub.