# US 10,504,584 B2 · Display method and display device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,584 B2
- Document ID: US10504584B2
- Status: Granted
- Kind: B2
- Application number: 15/838,791
- Filed: 2017-12-12
- Published: 2018-04-26
- Granted: 2019-12-10
- Earliest claimed: 2015-12-17
- Assignees: Panasonic Intellectual Property Corporation of America
- Applicants: Panasonic Intellectual Property Corporation of America
- Inventors: Hideki Aoyama; Mitsuaki Oshima
- Web page: https://patentgrep.com/patent/US10504584B2

## CPC

- H04B 10/116: Visible light communication
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 3/147: using display panels
- G06Q 30/0241: Advertisements
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/20: in augmented reality scenes
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- G09G 3/346: {based on modulation of the reflection angle, e.g. micromirrors (micromirrors devices per se G02B26/0833)}
- G09G 5/14: Display of multiple viewports
- G11C 11/418: Address circuits
- G11C 11/419: Read-write [R-W] circuits
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 21/2393: {involving handling client requests (scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4722: for requesting additional data associated with the content
- G02B 27/646: {compensating for small deviations, e.g. due to vibration or shake (movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68)}
- G09G 2360/144: the light being ambient light
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- G09G 2370/04: Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
- G09G 2370/042: for monitor identification
- G09G 2370/06: Consumer Electronics Control, i.e. control of another device by a display or vice versa
- G09G 2370/10: Use of a protocol of communication by packets in interfaces along the display data pipeline
- G09G 2370/16: Use of wireless transmission of display information
- G09G 2370/18: Use of optical transmission of display information
- G09G 2380/10: Automotive applications
- H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
- H10K 50/11: characterised by the electroluminescent [EL] layers
- H04B: TRANSMISSION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G11C: STATIC STORES (semiconductor memory devices H10B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- H10K: ORGANIC ELECTRIC SOLID-STATE DEVICES

## Abstract

A display method capable of displaying an image valuable to a user is provided. The display method for a display device to display an image includes: obtaining a captured display image and an image for decoding by an image sensor capturing a subject; decoding the image for decoding to obtain light identification information (light ID); transmitting the light ID to a server; obtaining an augmented reality (AR) image and recognition information corresponding to the light ID from the server; out of the captured display image, recognizing a region corresponding to the recognition information as a target region; and displaying the captured display image with the AR image superimposed on the target region.

## Claims

### Claim 1

A display method for displaying an image on a display device, the display method comprising:
obtaining a first image and a second image which includes light identification information by capturing a subject using an image sensor, the subject transmitting the light identification information by change in luminance;
decoding the second image to obtain the light identification information;
transmitting the light identification information to a server;
a plurality of sets of (i) an augmented reality image, (ii) recognition information, and (iii) identification information being stored in the server, out of the plurality of sets, obtaining, from the server, the augmented reality image and the recognition information included in each of the sets including the identification information that corresponds to the light identification information;
recognizing a target region in the first image using the obtained recognition information; and
displaying the first display image with the obtained augmented reality image superimposed on the recognized target region, wherein:
the recognition information includes:
reference information for specifying a reference region in the first image; and
target information indicating a relative position of the target region with respect to the reference region, and
the recognizing of the target region includes:
specifying the reference region from the first image by matching the reference information and the first image; and
recognizing the target region in the first image at the relative position from the reference region using the target information.

### Claim 2 (depends on 1)

The display method according to claim 1, wherein
the first image and the second image are obtained alternatively.

### Claim 3 (depends on 2)

The display method according to claim 2, wherein
a first exposure time of the image sensor to obtain the first image is longer than a second exposure time of the image sensor to obtain the second time in the obtaining of the first image and the second image,
a plurality of bright lines appear in the second image by exposure of a plurality of exposure lines included in the image sensor using the second exposure time,
the plurality of bright lines do not appear in the first image using the first exposure time.

### Claim 4 (depends on 3)

The display method according to claim 3, wherein
the reference information indicates that the position of the reference region in the first image is identical to a position of a bright line pattern region including the plurality of bright lines in the second image.

### Claim 5 (depends on 1)

The display method according to claim 1, wherein
the reference information indicates that the reference region in the first image is a region in which a display is shown in the first image.

### Claim 6 (depends on 5)

The display method according to claim 5, wherein
the reference region is an outer frame portion with predetermined color in an image displayed on the display.

### Claim 7 (depends on 1)

The display method according to claim 1, wherein
the displaying of the first image includes displaying a first augmented reality image, which is the augmented reality image, for a display period determined in advance while inhibiting display of a second augmented reality image different from the first augmented reality image.

### Claim 8 (depends on 7)

The display method according to claim 7, wherein
the displaying of the first image includes prohibiting decoding a new identification information in the display period.

### Claim 9 (depends on 8)

The display method according to claim 8, wherein
the displaying of the first image further includes:
measuring an acceleration level of the display device by an acceleration sensor which is included in the display device in the display period;
determining whether the measured acceleration level is equal to or greater than a threshold; and
when it is determined that the measured acceleration level is equal to or greater than the threshold, displaying the second augmented reality image instead of the first augmented reality image by canceling the inhibition of the display of the second augmented reality image.

### Claim 10 (depends on 1)

The display method according to claim 1, wherein
the displaying of the first image further includes:
determining whether a face of a user is approaching the display device using a face camera included in the display device; and
when it is determined that the face is approaching, displaying a first augmented reality image which is the augmented reality image while inhibiting display of a second augmented reality image different from the first augmented reality image.

### Claim 11 (depends on 1)

The display method according to claim 1, wherein
the displaying of the first image further includes:
determining whether a face of a user is approaching the display device with an acceleration level of the display device measured by an acceleration sensor which is included in the display device; and
when it is determined that the face is approaching, displaying a first augmented reality image which is the augmented reality image while inhibiting display of a second augmented reality image different from the first augmented reality image.

### Claim 12 (depends on 1)

The display method according to claim 1, wherein
the obtaining obtains the first image and the second image by capturing a plurality of displays that display images as a plurality of subjects,
the recognizing recognizes the target region in which one of the plurality of displays is shown, the one of the plurality of displays transmitting the light identification information, and
displaying the first image includes:
superimposing, on the target region, a first subtitle corresponding to an image displayed on the one of the plurality of displays as the augmented reality image; and
further superimposing a second subtitle that is a subtitle obtained by enlarging the first subtitle on a region larger than the target region in the first image.

### Claim 13 (depends on 12)

The display method according to claim 12, wherein
the displaying of the first image further includes:
determining whether sound information is included in information obtained from the server; and
when it is determined that the sound information is included, outputting sound indicated by the sound information with priority over the first and second subtitles.

### Claim 14

An apparatus comprising:
an image sensor that obtains a first image and a second image by capturing a subject, the subject transmitting light identification information by change in luminance;
a processor; and
a memory storing thereon a computer program, which when executed by the processor, causes the processor to perform operations including:
decoding the second image to obtain the light identification information;
transmitting the light identification information to a server;
a plurality of sets of (i) an augmented reality image, (ii) recognition information, and (iii) identification information being stored in the server, out of the plurality of sets, obtaining, from the server, the augmented reality image and the recognition information included in each of the sets including the identification information that corresponds to the light identification information;
recognizing a target region using the recognition information; and
displaying the first image with the augmented reality image superimposed on the target region, wherein:
the recognition information includes:
reference information for specifying a reference region in the first image; and
target information indicating a relative position of the target region with respect to the reference region, and
the recognizing of the target region includes:
specifying the reference region from the first image by matching the reference information and the first image; and
recognizing the target region in the first image at the relative position from the reference region using the target information.

### Claim 15

A non-transitory recording medium storing thereon a computer program, which when executed by a processor, causes the processor to perform operations including:
obtaining a first image and a second image by capturing a subject using an image sensor, the subject transmitting light identification information by change in luminance;
decoding the second image to obtain light identification information;
transmitting the light identification information to a server;
a plurality of sets of (i) an augmented reality image, (ii) recognition information, and (iii) identification information being stored in the server, out of the plurality of sets, obtaining, from the server, the augmented reality image and the recognition information included in each of the sets including the identification information that corresponds to the light identification information;
recognizing a target region using the recognition information; and
displaying, on a display, the first image with the augmented reality image superimposed on the target region, wherein:
the recognition information includes:
reference information for specifying a reference region in the first image; and
target information indicating a relative position of the target region with respect to the reference region, and
the recognizing of the target region includes:
specifying the reference region from the first image by matching the reference information and the first image; and
recognizing the target region in the first image at the relative position from the reference region using the target information.