# US 2019/0244019 A1 · Information processing device and information processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244019 A1
- Document ID: US20190244019A1
- Status: Published application
- Kind: A1
- Application number: 16/390,485
- Filed: 2019-04-22
- Published: 2019-08-08
- Earliest claimed: 2012-03-27
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: RYO FUKAZAWA; SHUNICHI KASAHARA; OSAMU SHIGETA; SEIJI SUZUKI; MAKI MORI
- Web page: https://patentgrep.com/patent/US20190244019A1

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/63: {by static guides}
- H04N 23/611: where the recognised objects include parts of the human body
- G07C 11/00: Arrangements, systems or apparatus for checking, e.g. the occurrence of a condition, not provided for elsewhere
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE

## Abstract

According to an illustrative embodiment, an information processing device is provided. The information processing device includes an image acquisition unit configured to receive an image; a recognition unit configured to acquire a recognition result of a user based on the received image, wherein the recognition result includes a position of the user, the user being associated with a display terminal; an image determination unit configured to determine an object based on the recognition result; and a display control unit configured to control display of the object on the display terminal.

## Claims

### Claim 1

An information processing system, comprising:
a portable terminal including a display;
a mobile object including
an imaging device configured to capture an image, and
an actuator to move the mobile object; and
circuitry configured to:
determine a position of a specific subject related to the portable terminal based on the captured image;
send the captured image and information on the determined position of the specific subject to the portable terminal;
determine a first virtual object based on the determined position of the specific subject; and
control the display to display the first virtual object over the received image to indicate the determined position of the specific subject.

### Claim 2 (depends on 1)

The information processing system according to claim 1,
wherein the portable terminal includes a touch panel configured to receive a touch operation and is configured to send, to the circuitry, data of a position of the touch operation in the captured image, and
wherein the circuitry is configured to determine the position of the subject related to the portable terminal based on the position of the touch operation in the captured image.

### Claim 3 (depends on 1)

The information processing system according to claim 1, wherein the circuitry is configured to:
determine whether there is a first subject performing a larger movement than a movement of a second subject in the captured image; and
determine the position of the specific subject based on a position of the first subject, when it is determined that there is the first subject performing the larger movement in the captured image.

### Claim 4 (depends on 1)

The information processing system according to claim 1, wherein the circuitry is configured to:
determine a travelling direction of the specific subject based on the captured image; and
change a composition of the captured image based on the determined position of the specific subject and the determined travelling direction of the specific subject.

### Claim 5 (depends on 4)

The information processing system according to claim 4,
wherein the imaging device is a movable camera configured to change a capturing direction in at least one of a pan direction or a tilt direction,
wherein the circuitry is configured to actuate the movable camera to change the composition of the captured image, based on the determined position of the specific subject and the determined travelling direction of the specific subject.

### Claim 6 (depends on 4)

The information processing system according to claim 4, wherein the circuitry is configured to change the composition of the captured image such that a first displayed region of the captured image corresponding to the travelling direction becomes larger than a second displayed region of the captured image corresponding to a direction opposite to the traveling direction.

### Claim 7 (depends on 1)

The information processing system according to claim 1, wherein the circuitry is configured to:
determine whether the specific subject is hidden behind at least one subject different from the specific subject in the captured image, and
control the display to display a second virtual object different from the first virtual object to emphasize the position of the specific subject in the captured image.

### Claim 8 (depends on 1)

The information processing system according to claim 1, wherein the circuitry is configured to control the imaging device to zoom-out the position of the specific subject after the position of the specific subject is zoomed-in when the position of the specific subject is determined.

### Claim 9 (depends on 1)

The information processing system according to claim 1, wherein the first virtual object is a message object including instruction to move the specific subject.

### Claim 10

A method for controlling an information processing system including a portable terminal and a mobile object, the method comprising:
connecting the portable terminal with the mobile object wirelessly;
determining a position of a specific subject related to the portable terminal based on an image captured by the mobile object; and
sending the captured image and information on the determined position of the specific subject to the portable terminal;
determining a virtual object based on the determined position of the specific subject;
controlling a display of the portable terminal to display the virtual object over the received image to indicate the determined position of the specific subject.

### Claim 11

At least one non-transitory computer-readable medium encoded with instructions which, when executed by at least one of a portable terminal, a mobile object, and a network server, causes a combination of the portable terminal and the mobile object to:
connect the portable terminal with the mobile object wirelessly;
determine a position of a specific subject related to the portable terminal based on an image captured by the mobile object; and
send the captured image and information on the determined position of the specific subject to the portable terminal;
determine a virtual object based on the determined position of the specific subject;
control a display of the portable terminal to display the virtual object over the received image to indicate the determined position of the specific subject.
