# US 10,319,110 B2 · Display control method and system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,110 B2 - Document ID: US10319110B2 - Status: Granted - Kind: B2 - Application number: 14/643,039 - Filed: 2015-03-10 - Published: 2015-10-22 - Granted: 2019-06-11 - Earliest claimed: 2014-04-16 - Assignees: Fujitsu Limited - Applicants: Fujitsu Limited - Inventors: Hidenori Ishikawa - Web page: https://patentgrep.com/patent/US10319110B2 ## CPC - G06T 7/74: {involving reference images or patches} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06K 19/00: Record carriers for use with machines and with at least a part designed to carry digital markings - G06V 10/225: {based on a marking or identifier characterising the area} - G06T 2207/30204: Marker - G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system includes circuitry configured to detect a first object and a second object from a first image captured, generate positional relationship information between the first object and the second object, and distortion information regarding a second shape of the second object in the first image with reference to a first shape of the first object in the first image, store, into a memory, the positional relationship information and the distortion information associated with the second object, acquire a second image, and control a display to display a content associated with the second object based on the positional relationship information and the distortion information when the first object is detected from the second image and when the second object is not detected from the second image, the content data being superimposed on the second image at a specific position with respect to the second object. ## Claims ### Claim 1 A system, comprising: circuitry configured to: detect a first object and a second object from a first image captured by an electronic device, wherein the first object is different from the second object and the first object has a different planar orientation from the second object; generate positional relationship information between the first object and the second object, and distortional relationship information regarding a second shape of the second object in the first image with reference to a first shape of the first object in the first image; store, into a memory, the positional relationship information and the distortional relationship information associated with a combination of the first object and the second object; detect the first object and a third object from a second image captured by the electronic device, wherein the second object is not detected from the second image; generate other positional relationship information between the first object and the third object, and other distortional relationship information regarding a third shape of the third object in the second image with reference to a fourth shape of the first object in the second image; store, into the memory, the other positional relationship information and the other distortional relationship information associated with a combination of the first object and the third object; acquire a third image captured by the electronic device; specify a position and a distortion of the third object in the third image when the third object is detected from the third image; and control a display to display a content associated with the second object based on the position, the distortion, the positional relationship information, the distortional relationship information, the other positional relationship information, and the other distortional relationship information when the first object and the second object are not detected from the third image, the content being superimposed on the third image. ### Claim 2 (depends on 1) The system according to claim 1, wherein the circuitry is configured to: estimate another position of the second object which is not detected from the third image based on the position, the distortion, the positional relationship information, the distortional relationship information, the other positional relationship information and the other distortional relationship information, and determine a specific position based on the another position, the content being superimposed on the third image at the specific position. ### Claim 3 (depends on 1) The system according to claim 1, wherein the circuitry is configured to control the display to display another content associated with the third object on the third image based on the position and the distortion, the another content being superimposed on the third image at another specific position with respect to the third object. ### Claim 4 (depends on 1) The system according to claim 1, wherein the other positional relationship information and the other distortional relationship information are deleted when a standard time has elapsed from storing of the other positional relationship information and the other distortional relationship information. ### Claim 5 (depends on 1) The system according to claim 1, wherein the distortional relationship information is generated based on a first difference between the first shape and an undistorted shape of the first object, and a second difference between the second shape and another undistorted shape of the second object. ### Claim 6 (depends on 1) The system according to claim 1, wherein the first object and the second object are markers having at least one of a specific shape or pattern. ### Claim 7 (depends on 1) The system according to claim 1, further comprising: the electronic device, wherein the electronic device includes an image pickup device configured to capture the first image and the second image; and a communication interface configured to send the first image and the second image to the system via a network and receive information from the system via the network. ### Claim 8 (depends on 1) The system according to claim 1, further comprising: the electronic device, wherein the electronic device includes the display configured to display the content. ### Claim 9 (depends on 1) The system according to claim 1, wherein the system is a server. ### Claim 10 (depends on 9) The system according to claim 9, wherein the server includes the circuitry; and a communication interface configured to receive the first image and the second image from the electronic device via a network and transmit the content to the electronic device via the network. ### Claim 11 (depends on 1) The system according to claim 1, wherein when the second object is not detected in the second image, the circuitry is further configured to adjust original content, associated with the second object, based on the position, the distortion, the positional relationship information, and the distortional relationship information, to generate the content, associate the content with the second object, and control the display to display the content. ### Claim 12 A display control method executed by a circuitry, the display control method comprising: detecting a first object and a second object from a first image captured by an electronic device, wherein the first object is different from the second object and the first object has a different planar orientation from the second object; generating positional relationship information between the first object and the second object, and distortional relationship information regarding a second shape of the second object in the first image with reference to a first shape of the first object in the first image; storing, into a memory, the positional relationship information and the distortional relationship information associated with a combination of the first object and the second object; detecting the first object and a third object from a second image captured by the electronic device, wherein the second object is not detected from the second image; generating other positional relationship information between the first object and the third object, and other distortional relationship information regarding a third shape of the third object in the second image with reference to a fourth shape of the first object in the second image; storing, into the memory, the other positional relationship information and the other distortional relationship information associated with a combination of the first object and the third object; acquiring a third image captured by the electronic device; specifying a position and a distortion of the third object in the third image when the third object is detected from the third image; and controlling a display to display a content associated with the second object based on the position, the distortion, the positional relationship information, the distortional relationship information, the other positional relationship information, and the other distortional relationship information when the first object and the second object are not detected from the third image, the content data being superimposed on the third image. ### Claim 13 (depends on 12) The display control method according to claim 12, further comprising: estimating another position of the second object which is not detected from the third image based on the position, the distortion, the positional relationship information, the distortional relationship information, the other positional relationship information and the other distortional relationship information; and determining a specific position based on the another position, the content being superimposed on the third image at the specific position. ### Claim 14 (depends on 12) The display control method according to claim 12, further comprising: controlling the display to display another content associated with the third object on the third image based on the position and the distortion, the another content being superimposed on the third image at another specific position with respect to the third object. ### Claim 15 (depends on 12) The display control method according to claim 12, wherein the other positional relationship information and the other distortional relationship information are deleted when a standard time has elapsed from storing of the other positional relationship information and the other distortional relationship information. ### Claim 16 (depends on 12) The display control method according to claim 12, wherein the distortional relationship information is generated based on a first difference between the first shape and an undistorted shape of the first object, and a second difference between the second shape and another undistorted shape of the second object. ### Claim 17 (depends on 12) The display control method according to claim 12, wherein the first object and the second object are markers having at least one of a specific shape or pattern. ### Claim 18 (depends on 12) The display control method according to claim 12, further comprising, when the second object is not detected in the second image: adjusting original content, associated with the second object, based on the position, the distortion, the positional relationship information, and the distortional relationship information, to generate the content; associating the content with the second object; and controlling the display to display the content. ### Claim 19 A non-transitory computer-readable storage medium storing a program which, when executed by a circuitry, causes the circuitry to: detect a first object and a second object from a first image captured by an electronic device, wherein the first object is different from the second object and the first object has a different planar orientation from the second object; generate positional relationship information between the first object and the second object, and distortional relationship information regarding a second shape of the second object in the first image with reference to a first shape of the first object in the first image; store, into a memory, the positional relationship information and the distortional relationship information associated with a combination of the first object and the second object; detect the first object and a third object from a second image captured by the electronic device, wherein the second object is not detected from the second image; generate other positional relationship information between the first object and the third object, and other distortional relationship information regarding a third shape of the third object in the second image with reference to a fourth shape of the first object in the second image; store, into the memory, the other positional relationship information and the other distortional relationship information associated with a combination of the first object and the third object; acquire a third image captured by the electronic device; specify a position and a distortion of the third object in the third image when the third object is detected from the third image; and control a display to display a content associated with the second object based on the position, the distortion, the positional relationship information, the distortional relationship information, the other positional relationship information, and the other distortional relationship information when the first object and the second object are not detected from the third image, the content data being superimposed on the third image. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium according to claim 19, wherein when the second object is not detected in the second image, the circuitry is further caused to adjust original content, associated with the second object, based on the position, the distortion, the positional relationship information, and the distortional relationship information, to generate the content, associate the content with the second object, and control the display to display the content.