# US 12,217,504 B2 · Method and apparatus for recognizing target device, for augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,504 B2
- Document ID: US12217504B2
- Status: Granted
- Kind: B2
- Application number: 18/112,726
- Filed: 2023-02-22
- Published: 2023-06-22
- Granted: 2025-02-04
- Earliest claimed: 2020-08-25
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Junsu Choi; Hyunkee Min; Jihoon Sung; Sunkee Lee; Junghun Lee
- Web page: https://patentgrep.com/patent/US12217504B2

## CPC

- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06V 20/20: in augmented reality scenes
- G01S 5/0284: {Relative positioning}
- G01S 13/72: for two-dimensional [2D] tracking, e.g. combination of angle and range tracking, track-while-scan radar
- G01S 13/89: for mapping or imaging
- G06F 1/1698: {the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone (details of antennas disposed inside a computer H01Q1/2266)}
- G06F 1/3206: Monitoring of events, devices or parameters that trigger a change in power modality
- G06F 1/3278: {Power saving in modem or I/O interface}
- G06F 1/3287: by switching off individual functional units in the computer system
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- H04N 21/41: Structure of client; Structure of client peripherals
- H04N 21/436: Interfacing a local distribution network, e.g. communicating with another STB {or one or more peripheral devices} inside the home
- H04N 21/4363: Adapting the video stream to a specific local network, e.g. a Bluetooth® network
- H04N 21/81: Monomedia components thereof
- G01S 3/74: Multi-channel systems specially adapted for direction-finding, i.e. having a single antenna system capable of giving simultaneous indications of the directions of different signals (systems in which the directions of different signals are determined sequentially and displayed simultaneously G01S3/04, G01S3/14)
- G01S 3/78: using electromagnetic waves other than radio waves
- G01S 5/0218: {Multipath in signal reception}
- G01S 5/12: by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic device may include a display, a camera, a communication circuitry, and a processor, wherein the processor may be configured to: control to transmit an image obtained through the camera to an external device through the communication circuitry, receive an AR image including at least one object from the external device through the communication circuitry and display same through the display, recognize a target object from among the at least one object, and map one of peripheral devices found on a data communication link through the communication circuitry to a target device corresponding to the recognized target object. Various embodiments are possible.

## Claims

### Claim 1

An electronic device comprising:
a display;
a camera;
a communication circuitry; and
at least one processor comprising processing circuitry; and
memory storing instructions that, when executed by the at least one processor, cause the electronic device to:
control to transmit an image obtained through the camera to an external device through the communication circuitry;
receive an augmented reality (AR) image including at least one object related to at least one peripheral device that exists around the electronic device from the external device through at least the communication circuitry;
control to display the AR image including the at least one object through the display;
recognize a target object from among the at least one object;
determine, based on signals transmitted by a plurality of peripheral devices on a data communication link, location information of the plurality of peripheral devices on the data communication link;
determine location information of the target object in the AR image; and
map, based on the determined location information of the target object in the AR image and the determined location information of the plurality of peripheral devices on the data communication link a peripheral device among the plurality of peripheral devices found on the data communication link to a target device corresponding to the recognized target object in the AR image.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the instructions, when executed by the at least one processor, cause electronic device to:
estimate a distance and direction to the target object on the AR image,
identify whether a signal transmitted from the target device on the data communication link is received from the estimated distance and direction, and
map the target object and the target device when the signal transmitted from the target device is received from the estimated distance and direction.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the instructions, when executed by the at least one processor, cause electronic device to:
perform a first operation for an angle of arrival (AoA) estimation for the target device by applying a designated first estimation algorithm, and
perform a second operation for the angle of arrival estimation at least by applying a second estimation algorithm designated based on an estimation result according to the performing of the first operation.

### Claim 4 (depends on 3)

The electronic device of claim 3, wherein the instructions, when executed by the at least one processor, cause electronic device to perform a channel impulse response (CIR) tab selection based on the signal from the target device.

### Claim 5 (depends on 3)

The electronic device of claim 3, wherein the instructions, when executed by the at least one processor, cause electronic device to select a tab of interest to be used for the angle of arrival estimation from among a plurality of tabs sampled at a predetermined interval for the signal from the target device,
wherein the tab of interest comprises a tab which corresponds to a value obtained at least by dividing the corresponding distance by a speed of an electromagnetic wave based on the distance estimated on the AR image.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the instructions, when executed by the at least one processor, cause electronic device to:
generate a plurality of outputs at least by applying the first estimation algorithm to the tab of interest, and
apply the second estimation algorithm to the plurality of outputs,
wherein the first estimation algorithm comprises an algorithm for estimating a signal in a specific direction by beamforming, and the second estimation algorithm comprises an algorithm for separating a designated signal and noise.

### Claim 7

An electronic device comprising:
a camera;
a display;
a communication circuitry; and
at least one processor comprising processing circuitry; and
memory storing instructions that, when executed by the at least one processor, cause the electronic device to:
control to transmit an image obtained through the camera to an external device through the communication circuitry;
receive an augmented reality (AR) image based on the transmitted image from the external device;
control to display the received AR image through the display;
estimate a distance and direction to a target object on the AR image displayed through the display based on sensing information using at least one sensor;
perform a tab selection among a plurality of channel impulse response tabs sampled at a predetermined interval based on a signal transmitted by the target object and the distance and direction estimated to the target object on the AR image;
apply a first estimation algorithm to the selected tab to obtain, for each antenna subset of a plurality of antennas of the communication circuitry, an output by using at least the estimated distance and direction; and
estimate an angle of arrival (AoA) at least by applying a second estimation algorithm to the outputs obtained by applying the first estimation algorithm to the antenna subsets.

### Claim 8 (depends on 7)

The electronic device of claim 7, wherein the first estimation algorithm comprises an algorithm for estimating a signal in a specific direction by beamforming, and the second estimation algorithm comprises an algorithm for separating a designated signal and noise.

### Claim 9 (depends on 7)

The electronic device of claim 7, wherein the instructions, when executed by the at least one processor, cause electronic device to:
recognize the target object on the image obtained through the camera and displayed through the display,
estimate a distance and direction to the recognized target object,
select a tab corresponding to the estimated distance and direction on the image as a tab of interest to be used for the angle of arrival estimation among a plurality of tabs sampled at a predetermined interval, and
provide the selected tab of interest as an input for applying the first estimation algorithm.

### Claim 10 (depends on 7)

The electronic device of claim 7, wherein the instructions, when executed by the at least one processor, cause electronic device to generate the antenna subset including a designated number of antennas based on the plurality of antennas.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the instructions, when executed by the at least one processor, cause electronic device to generate the antenna subset by overlapping at least one antenna in each of the antenna subsets to have a maximum and/or large number of antenna subsets.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the instructions, when executed by the at least one processor, cause electronic device to:
generate a plurality of outputs corresponding to the number of antenna subsets at least by applying the first estimation algorithm to the antenna subsets, and
apply the second estimation algorithm to the plurality of outputs.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the instructions, when executed by the at least one processor, cause the electronic device to:
separate eigenvectors constituting a signal subspace and eigenvectors constituting a noise subspace for the plurality of outputs, and
complete the angle of arrival estimation based on the signal subspace.

### Claim 14

A method of operating an electronic device, the method comprising:
obtaining an image through a camera of the electronic device;
transmitting at least a part of the obtained image to an external device;
receiving an augmented reality (AR) image including at least one object related to at least one peripheral device that exist around the electronic device based on the transmitted image, from the external device;
displaying the AR image including the at least one object via a display of the electronic device;
recognizing a target object from among the at least one object;
determining, based on signals transmitted by a plurality of peripheral devices on a data communication link, location information of the plurality of peripheral devices on the data communication link;
determining location information of the target object in the AR image; and
mapping, based on the determined location information of the target object in the AR image and the determined location information of the plurality of peripheral devices on the data communication link, a peripheral device among the plurality of peripheral devices found on the data communication link to a target device corresponding to the recognized target object in the AR image.

### Claim 15 (depends on 14)

The method of claim 14, wherein the mapping of one of peripheral devices comprises:
estimating a distance and direction to the target object on the AR image;
identifying whether a signal transmitted from the target device on the data communication link is received from the estimated distance and direction; and
mapping the target object and the target device in case that the signal transmitted from the target device is received from the estimated distance and direction.

### Claim 16 (depends on 7)

The electronic device of claim 7, wherein the antenna subsets of a plurality of antennas have at least one antenna which is included in an antenna subset adjacent to each other.

### Claim 17 (depends on 7)

The electronic device of claim 7, wherein the selected tab corresponds to a value obtained by dividing the estimated distance of the target object by a speed of an electromagnetic wave transmitted by the communication circuit.

### Claim 18

A non-transitory computer-readable recording medium storing a program which, when executed, causes an electronic device to perform operations comprising:
obtaining an image through a camera of the electronic device,
transmitting at least a part of the obtained image to an external device,
receiving an augmented reality (AR) image including at least one object related to at least one peripheral device that exist around the electronic device based on the transmitted image, from the external device,
displaying the AR image including the at least one object via a display of the electronic device,
recognizing a target object from among the at least one object,
determining, based on signals transmitted by a plurality of peripheral devices on a data communication link, location information of the plurality of peripheral devices on the data communication link;
determining location information of the target object in the AR image; and
mapping, based on the determined location information of the target object in the AR image and the determined location information of the plurality of peripheral devices on the data communication link, a peripheral device among the plurality of peripheral devices found on the data communication link to a target device corresponding to the recognized target object in the AR image.

### Claim 19 (depends on 1)

The electronic device of claim 1, wherein the location information of the plurality of peripheral devices on the data communication link includes distance and direction information of the plurality of peripheral devices.