# US 2021/0334066 A1 · Devices with enhanced audio

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334066 A1
- Document ID: US20210334066A1
- Status: Published application
- Kind: A1
- Application number: 17/369,374
- Filed: 2021-07-07
- Published: 2021-10-28
- Earliest claimed: 2011-07-28
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Aleksandar Pance; Brett Bilbrey; Darby E. Hadley; Martin E. Johnson; Ronald Nadim Isaac
- Web page: https://patentgrep.com/patent/US20210334066A1

## CPC

- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 17/00: Speaker identification or verification techniques
- H04N 7/15: Conference systems
- H04N 21/4143: {embedded in a} Personal Computer [PC]
- H04N 21/42201: {biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015)}
- H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/439: Processing of audio elementary streams
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04R 3/04: for correcting frequency response
- H04R 29/001: {for loudspeakers (H04R29/007 takes precedence)}
- H04S 7/303: {Tracking of listener position or orientation}
- H04S 7/305: {Electronic adaptation of stereophonic audio signals to reverberation of the listening space (H04S7/301 takes precedence)}
- H04N 5/60: for the sound signals
- H04N 7/142: {Constructional details of the terminal equipment, e.g. arrangements of the camera and the display}
- H04N 21/42222: {Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/4852: {for modifying audio parameters, e.g. switching between mono and stereo}
- H04R 2430/01: Aspects of volume control, not necessarily automatic, in sound systems
- H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A system for enhancing audio including a plurality of sensors, an output device, and a processor in communication with the plurality of sensors and the output device. The processor is configured to process data captured by the plurality of sensors, and based on that, modify an output of the output device. The processor also is configured to determine whether there are a plurality of users associated with a video conferencing session, determine which user of the plurality of users is speaking, and enhance the audio or video output of the speaking user on the output device.

## Claims

### Claim 1

A computing device, comprising:
a processor that is configured to communicate with an audio output device and a display to provide a video conferencing session having video and sound output, the processor to
produce, during the video conferencing session, i) video on the display that includes simultaneous images of a plurality of persons, and ii) sound output by the audio output device that includes audio of the plurality of persons,
determine that there are multiple persons in the video conferencing session, and
determine when at least one of the multiple persons is speaking, and in response enhance audio and video of said at least one of the multiple persons, relative to audio and video of remaining one or more of the multiple persons.

### Claim 2 (depends on 1)

The computing device of claim 1 wherein the processor is configured to enhance video by minimizing or hiding the image of the remaining one or more of the multiple persons.

### Claim 3 (depends on 2)

The computing device of claim 2 wherein each person of the multiple persons is in a separate chat window, and the chat window of a speaking person is enhanced.

### Claim 4 (depends on 3)

The computing device of claim 3 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

### Claim 5 (depends on 4)

The computing device of claim 4 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the video, or by performing voice recognition on the sound output.

### Claim 6 (depends on 1)

The computing device of claim 1 wherein each person of the multiple persons is in a separate chat window, and the chat window of a speaking person is enhanced.

### Claim 7 (depends on 6)

The computing device of claim 6 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

### Claim 8 (depends on 7)

The computing device of claim 7 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the video, or by performing voice recognition on the sound output.

### Claim 9 (depends on 1)

The computing device of claim 1 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the video, or by performing voice recognition on the sound output.

### Claim 10 (depends on 1)

The computing device of claim 1 wherein the processor selects one of the multiple persons in the video conference session by detecting a viewing user's tap on an image of the selected person, and focuses on the selected person by transmitting a signal to a location of the selected user to process the audio of the selected person to better capture speech of the selected person.

### Claim 11

A method for providing a video conference session having video and sound output, the method comprising:
producing, during the video conferencing session, i) video on a display that includes simultaneous images of a plurality of persons, and ii) sound output by an audio output device that includes audio of the plurality of persons;
determining that there are multiple persons in the video conferencing session; and
determining when at least one of the multiple persons is speaking, and in response enhancing audio and video of the at least one of the multiple persons who is speaking, relative to remaining one or more of the multiple person.

### Claim 12 (depends on 11)

The method of claim 11 wherein enhancing video of the at least one of the multiple persons who is speaking comprises
minimizing or hiding the image of only the remaining one or more of the multiple persons.

### Claim 13 (depends on 12)

The method of claim 12 wherein each person of the multiple persons is in a separate chat window, and the chat window of the at least of the multiple persons who is speaking is enhanced.

### Claim 14 (depends on 13)

The method of claim 13 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

### Claim 15 (depends on 14)

The method of claim 14 wherein determining there are multiple persons in the video conferencing session comprises
performing facial recognition on the video, or performing voice recognition on the sound output.

### Claim 16 (depends on 11)

The method of claim 11 wherein each person of the multiple persons is in a separate chat window, and the chat window of a speaking person is enhanced.

### Claim 17 (depends on 16)

The method of claim 16 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

### Claim 18 (depends on 17)

The method of claim 17 wherein determining there are multiple persons in the video conferencing session comprises
performing facial recognition on the video, or performing voice recognition on the sound output.

### Claim 19 (depends on 11)

The method of claim 11 wherein determining there are multiple persons in the video conferencing session comprise performing facial recognition on the video, or performing voice recognition on the sound output.

### Claim 20 (depends on 11)

The method of claim 11 further comprising selecting one of the multiple persons in the video conference session by detecting a viewing user's tap on an image of the selected person, and focusing on the selected person by transmitting a signal to a location of the selected user to process the audio of the selected person to better capture speech of the selected person.
