# US 2021/0342479 A1 · Privacy protection in mobile robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342479 A1
- Document ID: US20210342479A1
- Status: Published application
- Kind: A1
- Application number: 16/861,451
- Filed: 2020-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Cobalt Robotics Inc.
- Applicants: Cobalt Robotics Inc.
- Inventors: Erik Schluntz; Travis J. Deyle; Michael O'Kelley; Ashu Gupta
- Web page: https://patentgrep.com/patent/US20210342479A1

## CPC

- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- B25J 5/007: {mounted on wheels}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 13/006: {by means of a wireless system for controlling one or several manipulators}
- B25J 19/023: {including video camera means}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G06T 5/70: Denoising; Smoothing
- G06T 7/74: {involving reference images or patches}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06V 40/172: {Classification, e.g. identification}
- H04L 9/0643: {Hash functions, e.g. MD5, SHA, HMAC or f9 MAC}
- H04L 63/0421: {Anonymous communication, i.e. the party's identifiers are hidden from the other party or parties, e.g. using an anonymizer}
- H04L 63/107: {wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals}
- G06F 2221/2111: Location-sensitive, e.g. geographical location, GPS
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30176: Document
- G06T 2207/30201: Face
- G06T 2210/12: Bounding box
- H04L 2209/42: Anonymization, e.g. involving pseudonyms
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A mobile robot is configured for operation in a commercial or industrial setting, such as an office building or retail store. The mobile robot may include cameras for capturing images and videos and include microphones for capturing audio of its surroundings. To improve privacy by preventing confidential information from being transmitted, the mobile robot may detect text in images and modify the images to make the text illegible before transmitting the images. The mobile robot may also detect human voice in audio and modify audio to make the human voice unintelligible before transmitting the audio.

## Claims

### Claim 1

A mobile robot comprising:
a motorized base configured to move the mobile robot;
a wireless transceiver configured to communicatively couple to a server;
a housing body storing a camera configured to capture images of an environment surrounding the mobile robot and a microphone configured to capture an audio signal from the environment; and
a controller configured to:
receive images of the environment from the camera;
detect text in the images;
responsive to detecting the text, determine whether to modify the images based in part on a semantic map that stores information describing the environment;
responsive to determining to modify the images, generate a bounding box around the text in the images;
modifying the images within the bounding box to make the text illegible; and
transmit the modified images to the server.

### Claim 2 (depends on 1)

The mobile robot of claim 1, wherein determining whether to modify the images comprises:
determining a location of the mobile robot;
querying the semantic map associated with the environment to determine a privacy setting associated with the location; and
determining to modify the images based on the privacy setting associated with the location.

### Claim 3 (depends on 1)

The mobile robot of claim 1, wherein determining whether to modify the images comprises:
performing image recognition to determine an object type associated with an object including the text in the images;
querying the semantic map associated with the environment to determine a privacy setting associated with the object type; and
determining to modify a region of the images including the object based the privacy setting associated with the object type.

### Claim 4 (depends on 1)

The mobile robot of claim 1, wherein detecting text further comprises:
for each of a plurality of regions within the image:
identifying a number of edges in the region;
comparing the number of edges to a predetermined threshold; and
responsive to the number of edges exceeding the predetermined threshold, determining that there is text in the region.

### Claim 5 (depends on 1)

The mobile robot of claim 1, wherein the controller is further configured to:
detect a face of a person in the images;
apply a bounding box around the face in the images;
modify the images within the bounding box to make the face unidentifiable from the images; and
transmit the modified images to the server.

### Claim 6 (depends on 6)

The mobile robot of claim 6, wherein the controller is further configured to:
generate a hash value based on the detected face; and
transmit the hash value to the server in association with the modified images.

### Claim 7 (depends on 1)

The mobile robot of claim 1, wherein the controller is further configured to:
receive the audio signal of the environment from the microphone;
detect human voice in the audio signal;
responsive to detecting the human voice, modify a portion of the audio signal within a frequency range of human voice to be unintelligible; and
transmit the modified audio signal to the server.

### Claim 8

A method comprising:
capturing, by a mobile robot, images of an environment surrounding the mobile robot;
detecting, by the mobile robot, text in the images;
responsive to detecting the text, determining, by the mobile robot, whether to modify the images based in part on a semantic map that stores information describing the environment;
responsive to determining to modify the images, generating, by the mobile robot, a bounding box around the text in the images;
modifying, by the mobile robot, the images within the bounding box to make the text illegible, and
transmitting, by the mobile robot, the modified images to a server.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining whether to modify the images comprises:
determining a location of the mobile robot;
querying the semantic map associated with the environment to determine a privacy setting associated with the location; and
determining to modify the images based on the privacy settings associated with the location.

### Claim 10 (depends on 8)

The method of claim 8, wherein determining whether to modify the images comprises:
performing image recognition to determine an object type associated with an object including the text in the images;
querying the semantic map associated with the environment to determine a privacy setting associated with the object type; and
determining to modify a region of the images including the object based the privacy setting associated with the object type.

### Claim 11 (depends on 8)

The method of claim 8, wherein detecting text, further comprises:
for each of a plurality of regions within the image:
identifying a number of edges in the region;
comparing the number of edges to a predetermined threshold; and
responsive to the number of edges exceeding the predetermined threshold, determining that there is text in the region.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
detecting a face of a person in the images;
applying a bounding box around the face in the images;
modifying the images within the bounding box to make the face unidentifiable from the images; and
transmitting the modified images to the server.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
generate a hash value based on the detected face; and
transmit the hash value to the server in association with the modified images.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
receiving an audio signal of the environment surrounding the mobile robot;
detecting human voice in the audio signal;
responsive to detecting the human voice, modifying a portion of the audio signal within a frequency range of human voice to be unintelligible; and
transmitting the modified audio signal to the server.

### Claim 15

A non-transitory computer readable storage medium storing computer program instructions, the instructions when executed by a processor causing the processor to:
capture images of an environment surrounding a mobile robot;
detect text in the images;
responsive to detecting the text, determine whether to modify the images based in part on a semantic map that stores information describing the environment;
responsive to determining to modify the images, generate a bounding box around the text in the images;
modify the images within the bounding box to make the text illegible, and
transmit the modified images to a server.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein determining whether to modify the images comprises:
determining a location of the mobile robot;
querying the semantic map associated with the environment to determine a privacy setting associated with the location; and
determining to modify the images based on the privacy settings associated with the location.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein determining whether to modify the images comprises:
performing image recognition to determine an object type associated with an object including the text in the images;
querying the semantic map associated with the environment to determine a privacy setting associated with the object type; and
determining to modify a region of the images including the object based the privacy setting associated with the object type.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein detecting text, further comprises:
for each of a plurality of regions within the image:
identifying a number of edges in the region;
comparing the number of edges to a predetermined threshold; and
responsive to the number of edges exceeding the predetermined threshold, determining that there is text in the region.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the processors are further configured to:
detect a face of a person in the images;
apply a bounding box around the face in the images;
modify the images within the bounding box to make the face unidentifiable from the images; and
transmit the modified images to the server.

### Claim 20 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the processors are further configured to:
determine an identity of the person;
generate a hash value associated with the identity; and
transmit the hash value to the server in association with the modified images.
