# US 2021/0342590 A1 · Mobile apparatus and control method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342590 A1
- Document ID: US20210342590A1
- Status: Published application
- Kind: A1
- Application number: 17/284,566
- Filed: 2019-02-19
- Published: 2021-11-04
- Earliest claimed: 2018-10-12
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Eun Young CHOI; Min Sun PARK; Sung Hyun JANG; Ji-Youn HAN; Soo Hyun HWANG
- Web page: https://patentgrep.com/patent/US20210342590A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/20: in augmented reality scenes
- H04L 12/2812: {describing content present in a home automation network, e.g. audio video content (retrieval from the Internet G06F16/95)}
- H04N 21/41265: {having a remote control device for bidirectional communication between the remote control device and client device (unidirectional remote control device H04N21/42204)}
- H04N 21/42222: {Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 2203/04803: Split screen, i.e. subdividing the display area or the window area into separate subareas
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06V 2201/02: Recognising information on displays, dials, clocks
- G09G 3/003: {to produce spatial visual effects}
- G09G 2340/0464: Positioning
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G 2354/00: Aspects of interface with display user
- H04L 2012/2849: {Audio/video appliances}
- H04W 88/02: Terminal devices
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

In order to provide information related to content played on a target device within a visual field or a field of view (FOV) of the camera's, and to control the target device, a mobile apparatus according to an embodiment of present disclosure includes: a display; a camera configured to photograph a target device; a user interface configured to receive user's input; and a controller configured to control the display to display the image of the target device photographed by the camera and information related to an object included in the content being played by the target device in response to the user's input.

## Claims

### Claim 1

A mobile apparatus comprising:
a display;
a camera configured to photograph a target device;
a user interface configured to receive user's input; and
a controller configured to control the display to display the image of the target device photographed by the camera and information related to an object included in the content being played by the target device in response to the user's input.

### Claim 2 (depends on 1)

The mobile apparatus according to claim 1, wherein the controller is configured to control the display to display at least one virtual image including information related to the object around the image of the target device within the field of view (FOV) of the camera, and control the target device in response to the user's input for at least one of the image of the target device and the at least one virtual image.

### Claim 3 (depends on 2)

The mobile apparatus according to claim 2, wherein the controller is configured to control the display to enlarge and display the virtual image located in the moving direction of the center line of the FOV of the camera.

### Claim 4 (depends on 2)

The mobile apparatus according to claim 2, wherein the controller is configured to control the target device so that the target device outputs first content corresponding to the selected virtual image in response to the user's input for selecting and moving any one of the at least one virtual image.

### Claim 5 (depends on 4)

The mobile apparatus according to claim 4, wherein the controller is configured to divide the output area of the target device into a first area and a second area in response to the user's input for additionally selecting and moving the virtual image, and control the target device to output the first content and the second content corresponding to the additionally selected virtual image in the first area and the second area, respectively.

### Claim 6 (depends on 5)

The mobile apparatus according to claim 5, wherein the controller is configured to control the target device to adjust the size of the first area and the size of the second area in response to the zoom input for the first area or the second area of the target device.

### Claim 7 (depends on 1)

The mobile apparatus according to claim 1, wherein the controller is configured to control the display to display a screen providing detailed contents of information related to the object in response to the user's input tilting the mobile apparatus.

### Claim 8 (depends on 2)

The mobile apparatus according to claim 2, wherein the controller is configured to control the display to display a screen providing information on contents corresponding to each of the at least one virtual image as a list in response to the user's input tilting the mobile apparatus.

### Claim 9

A control method of a mobile apparatus, the method comprising:
photographing a target device using a camera;
identifying an object included in the content being played by the photographed target device in response to a user's input; and
displaying the photographed image of the target device and information related to the identified object on a display.

### Claim 10 (depends on 9)

The control method according to claim 9, wherein the displaying on the display comprises:
displaying at least one virtual image including information related to the object around the image of the target device on the display; and
controlling the target device in response to the user's input for at least one of the image of the target device and the at least one virtual image.

### Claim 11 (depends on 10)

The control method according to claim 10, wherein the displaying at least one virtual image on the display comprises:
enlarging and displaying the virtual image located in the moving direction of the center line of the FOV of the camera.

### Claim 12 (depends on 10)

The control method according to claim 10, wherein the controlling the target device comprises:
controlling the target device so that the target device outputs first content corresponding to the selected virtual image in response to the user's input for selecting and moving any one of the at least one virtual image.

### Claim 13 (depends on 12)

The control method according to claim 12, wherein the controlling the target device comprises:
dividing the output area of the target device into a first area and a second area in response to the user's input for additionally selecting and moving the virtual image, and controlling the target device to output the first content and the second content corresponding to the additionally selected virtual image in the first area and the second area, respectively.

### Claim 14 (depends on 9)

The control method according to claim 9, further comprising:
displaying a screen providing detailed contents of information related to the object on the display in response to the user's input tilting the mobile apparatus.

### Claim 15 (depends on 10)

The control method according to claim 10, further comprising:
displaying a screen providing information on contents corresponding to each of the at least one virtual image as a list on the display in response to the user's input tilting the mobile apparatus.
