# US 2021/0278904 A1 · Information processing device, information processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278904 A1
- Document ID: US20210278904A1
- Status: Published application
- Kind: A1
- Application number: 16/336,730
- Filed: 2017-08-21
- Published: 2021-09-09
- Earliest claimed: 2016-11-08
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: CHAO MA; KAZUHIRA OKADA; MAKOTO ENDO
- Web page: https://patentgrep.com/patent/US20210278904A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- A61B 34/25: {User interfaces for surgical systems}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0485: Scrolling or panning
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Provided is a user interface suitable for, for example, a non-contact operation or the like. First, an operation direction and an operation distance are calculated from acquired operation information. In addition, an operation space is set in which regions of items to be selected are arranged around an origin position. Then, in a case where a positional relationship between a region of an item and an operation position specified by the operation direction and the operation distance from the origin position as a starting point is changed to a specific state, a process of determining that the item has been selected is performed. In addition, display data corresponding to the determination result is generated such that the user recognizes an operation situation.

## Claims

### Claim 1

An information processing device comprising:
an operation detection unit that calculates an operation direction and an operation distance from acquired operation information;
a control unit that, in a case where a positional relationship between a region of an item to be selected and an operation position specified by the operation direction and the operation distance calculated by the operation detection unit using an origin position as a starting point is changed to a specific state in a state in which the regions of the items to be selected are arranged around the origin position, performs a process of determining that the item has been selected; and
a display data generation unit that generates display data corresponding to a determination result of the control unit.

### Claim 2 (depends on 1)

The information processing device according to claim 1,
wherein the operation detection unit detects non-contact operation information as the operation information, and
the control unit sets the origin position substantially at a center of an operation region in which the non-contact operation information is effective.

### Claim 3 (depends on 1)

The information processing device according to claim 1,
wherein the display data generation unit generates display data for displaying the operation direction and the operation distance calculated by the operation detection unit on an operation plane.

### Claim 4 (depends on 1)

The information processing device according to claim 1,
wherein the control unit sets an operation plane in which a plurality of items to be selected are arranged around the origin position so as not to overlap each other as viewed from the origin position.

### Claim 5 (depends on 4)

The information processing device according to claim 4,
wherein, in a case where the control unit recognizes that a certain item has been selected by an operation position on the operation plane and a lower-layer item is present in the selected item, the control unit resets the operation plane such that the selected item is arranged at the origin position and the lower-layer items are arranged around the origin position so as not to overlap each other as viewed from the origin position, and
in a case where the operation position indicates a certain lower-layer item on the reset operation plane, the control unit performs a process of recognizing that the lower-layer item has been selected.

### Claim 6 (depends on 5)

The information processing device according to claim 5,
wherein, in a case where the operation plane is set or reset, the control unit arranges a plurality of items around the origin position along a virtual ring of a true circle having the origin position as a center.

### Claim 7 (depends on 5)

The information processing device according to claim 5,
wherein, in a case where the operation plane is reset, the control unit controls the display data generation unit such that display presenting movement of the selected item to the origin position is performed.

### Claim 8 (depends on 5)

The information processing device according to claim 5,
wherein, in a case where the operation plane is reset, the control unit arranges the lower-layer items and an item for a layer return operation around the origin position so as not to overlap each other as viewed from the origin position.

### Claim 9 (depends on 8)

The information processing device according to claim 8,
wherein, in a case where the item for the layer return operation is selected, the control unit controls the display data generation unit such that display presenting movement of the item for the layer return operation to the origin position is performed, and
the control unit resets the operation plane such that upper-layer items are arranged around the origin position so as not to overlap each other as viewed from the origin position.

### Claim 10 (depends on 1)

The information processing device according to claim 1,
wherein the control unit sets, as an operation plane for performing an operation involving a plurality of stages for selection or designation of an item, an operation plane in which a plurality of items to be selected in an X-th stage are arranged on an X-th virtual ring-shaped line from the origin position so as not to overlap each other as viewed from the origin position, and
in a case where the control unit recognizes that a certain item among the plurality of items in the X-th stage has been selected by an operation position, the control unit sets an operation plane in which one or a plurality of items to be designated or selected according to the item selected in the X-th stage are arranged on an (X+1) -th virtual ring-shaped line from the origin position (where Xis a natural number equal to or greater than 1).

### Claim 11 (depends on 10)

The information processing device according to claim 10,
wherein, in a case where the control unit recognizes that a certain item among the plurality of items in the X-th stage has been selected by the operation position, the control unit sets an operation plane in which a plurality of lower-layer items to be selected in a lower layer of the item selected in the X-th stage are arranged on the (X+1)-th virtual ring-shaped line from the origin position, and
in a case where the operation position indicates a lower-layer item, the control unit performs a process of recognizing that the lower-layer item has been selected.

### Claim 12 (depends on 10)

The information processing device according to claim 10,
wherein, in a case where the control unit recognizes that a certain item among the plurality of items in the X-th stage has been selected by the operation position, the control unit sets an operation plane in which an item for an enter operation for the item selected in the X-th stage is arranged on the (X+1)-th virtual ring-shaped line from the origin position, and
in a case where the operation position designates the item for the enter operation, the control unit performs a process of recognizing that the enter operation for the item selected in the X-th stage has been performed.

### Claim 13 (depends on 12)

The information processing device according to claim 12,
wherein the control unit arranges the item for the enter operation at a position that is on the (X+1)-th virtual ring-shaped line and is adjacent to the item selected in the X-th stage.

### Claim 14 (depends on 10)

The information processing device according to claim 10,
wherein the control unit sets the plurality of items to be selected in the X-th stage as items for a scroll operation, and
in a case where an item for the scroll operation is selected, the control unit performs a process of recognizing the scroll operation.

### Claim 15 (depends on 14)

The information processing device according to claim 14,
wherein the control unit performs a process of recognizing a direction of the scroll operation according to an order in which a plurality of items for the scroll operation are selected.

### Claim 16 (depends on 15)

The information processing device according to claim 15,
wherein the control unit arranges an item for an enter operation after scrolling on the (X+1)-th virtual ring-shaped line in the same angular range as an item that is being selected among the items for the scroll operation.

### Claim 17 (depends on 10)

The information processing device according to claim 10,
wherein, in a case where the control unit recognizes that a certain item among the plurality of items in the X-th stage has been selected by the operation position, the control unit instructs the display data generation unit such that a display aspect of items which have not been selected among the plurality of items in the X-th stage is changed.

### Claim 18 (depends on 10)

The information processing device according to claim 10,
wherein the control unit sets an operation plane in which one or a plurality of items to be designated or selected in an (X+1)-th stage according to the item selected in the X-th stage are arranged, and
in a case where selection of the selected item is canceled, the control unit sets the operation plane such that the items in the (X+1)-th stage are canceled.

### Claim 19

An information processing method performed by an information processing device, the method comprising:
an operation detection procedure of calculating an operation direction and an operation distance from acquired operation information;
a determination procedure of, in a case where a positional relationship between a region of an item to be selected and an operation position specified by the operation direction and the operation distance calculated in the operation detection procedure using an origin position as a starting point is changed to a specific state in a state in which the regions of the items to be selected are arranged around the origin position, determining that the item has been selected; and
a display data generation procedure of generating display data corresponding to a determination result in the determination procedure.

### Claim 20

A program that causes an information processing device to perform:
an operation detection step of calculating an operation direction and an operation distance from acquired operation information;
a determination step of, in a case where a positional relationship between a region of an item to be selected and an operation position specified by the operation direction and the operation distance calculated in the operation detection step using an origin position as a starting point is changed to a specific state in a state in which the regions of the items to be selected are arranged around the origin position, determining that the item has been selected; and
a display data generation step of generating display data corresponding to a determination result in the determination step.
