# US 2020/0019184 A1 · Robotic systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0019184 A1
- Document ID: US20200019184A1
- Status: Published application
- Kind: A1
- Application number: 16/508,089
- Filed: 2019-07-10
- Published: 2020-01-16
- Earliest claimed: 2018-07-10
- Assignees: Emotech Ltd.
- Applicants: Emotech Ltd.
- Inventors: Ondrej MIKSIK; Pawel SWIETOJANSKI; Srikanth REDDY; Raymond W.M. NG
- Web page: https://patentgrep.com/patent/US20200019184A1

## CPC

- G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 19/023: {including video camera means}
- B25J 19/026: {Acoustical sensing devices}
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0255: {using acoustic signals, e.g. ultra-sonic singals (sonar systems designed for anti-collision purposes G01S15/93)}
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06F 2218/22: Source localisation; Inverse modelling
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A robotic system is controlled. Audiovisual data representing an environment in which at least part of the robotic system is located is received via at least one camera and at least one microphone. The audiovisual data comprises a visual data component representing a visible part of the environment and an audio data component representing an audible part of the environment. A location of a sound source that emits sound that is represented in the audio data component of the audiovisual data is identified based on the audio data component of the audiovisual data. The sound source is outside the visible part of the environment and is not represented in the visual data component of the audiovisual data. Operation of a controllable element located in the environment is controlled based on the identified location of the sound source.

## Claims

### Claim 1

A method of controlling a robotic system, the method comprising:
receiving, via at least one camera and at least one microphone, audiovisual data representing an environment in which at least part of the robotic system is located, the audiovisual data comprising a visual data component representing a visible part of the environment and an audio data component representing an audible part of the environment;
identifying, based on the audio data component of the audiovisual data, a location of a sound source that emits sound that is represented in the audio data component of the audiovisual data, wherein the sound source is outside the visible part of the environment and is not represented in the visual data component of the audiovisual data; and
controlling operation of a controllable element located in the environment based on the identified location of the sound source.

### Claim 2 (depends on 1)

A method according to claim 1, wherein the at least one microphone comprises an array of microphones comprising at least first and second microphones, wherein the sound emitted by the sound source is received at both the first and second microphones in the microphone array, and wherein the location of the sound source is identified based at least in part on differences in phase and/or intensity of the sound as received at the first and second microphones in the microphone array.

### Claim 3 (depends on 1)

A method according to claim 1, wherein the identifying of the location of the sound source comprises determining a direction and/or a distance of the sound source with respect to the at least one microphone.

### Claim 4 (depends on 1)

A method according to claim 1, wherein the identifying of the location of the sound source comprises using a spatial model of the environment.

### Claim 5 (depends on 4)

A method according to claim 4, wherein the spatial model of the environment represents at least part of the audible part of the environment, wherein the at least part of the audible part of the environment is not also part of the visible part of the environment.

### Claim 6 (depends on 4)

A method according to claim 4, comprising generating the spatial model using data received via the at least one camera and/or via the at least one microphone.

### Claim 7 (depends on 1)

A method according to claim 1, comprising identifying an activity type of the audible activity.

### Claim 8 (depends on 7)

A method according to claim 7, wherein the identifying of the location of the sound source is based at least in part on the identified activity type of the audible activity.

### Claim 9 (depends on 1)

A method according to claim 1, wherein the controllable element is comprised in a self-propelling device and wherein the controlling of the operation of the controllable element comprises causing the self-propelling device to avoid the location of the sound source.

### Claim 10 (depends on 1)

A method according to claim 1, wherein the controllable element is comprised in a self-propelling device and wherein the controlling of the operation of the controllable element comprises causing the self-propelling device to move towards the location of the sound source.

### Claim 11 (depends on 1)

A method according to claim 1, comprising determining a location of at least part of the robotic system using the location of the sound source as a reference location.

### Claim 12 (depends on 1)

A method according to claim 1, comprising, in response to determining that the audible activity is outside an initial field of view of the at least one camera, causing the at least one camera to be configured to have a further, different field of view.

### Claim 13 (depends on 1)

A method according to claim 1, wherein controlling the operation of the controllable element comprises transmitting a control signal to the controllable element.

### Claim 14 (depends on 1)

A method according to claim 1, comprising analysing the visual data component of the audiovisual data for visible activity corresponding to the audible activity.

### Claim 15 (depends on 1)

A method according to claim 1, wherein the audible activity is caused by activity of a person located in the environment.

### Claim 16 (depends on 15)

A method robotic system according to claim 15, comprising identifying the person.

### Claim 17 (depends on 1)

A method according to claim 1, wherein the at least one camera, the at least one microphone and the at least one controllable element are all comprised in the same electronic device as each other.

### Claim 18 (depends on 1)

A method according to claim 1, wherein the at least one camera, the at least one microphone and the at least one controllable element are distributed across at least two different electronic devices.

### Claim 19 (depends on 1)

A method according to claim 1, wherein the at least one camera, the at least one microphone and/or the at least one controllable element is comprised in:
a vacuum cleaner;
a lawnmower;
a smart home controller;
a robotic pet;
an appliance;
a vehicle; and/or
a robotic assistive device.

### Claim 20 (depends on 1)

A method according to claim 1, wherein the controllable element is configured to control at least one environmental parameter of the environment.

### Claim 21 (depends on 1)

A method according to claim 1, wherein the identified location of the sound source is a first location of the sound source, and wherein method comprises:
identifying, based on at least the audio data component of the audiovisual data, a second location of the sound source; and
tracking movement of the sound source based on at least the first and second locations of the sound source.

### Claim 22 (depends on 21)

A method according to claim 21, wherein the second location of the sound source is within the visible part of the environment.

### Claim 23 (depends on 1)

A method according to claim 1, comprising identifying, based on the audiovisual data, at least one attribute of further activity, wherein the further activity involves emission of sound by a further sound source that is within the visible part of the environment and that is represented in the visual data component of the audiovisual data.

### Claim 24 (depends on 1)

A robotic system comprising a controller configured to perform a method according to claim 1.

### Claim 25 (depends on 1)

A computer program arranged to perform a method according to claim 1.
