# US 8,751,969 B2 · Information processor, processing method and program for displaying a virtual image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,751,969 B2 - Document ID: US8751969B2 - Status: Granted - Kind: B2 - Application number: 13/383,511 - Filed: 2010-06-30 - Published: 2012-05-17 - Granted: 2014-06-10 - Earliest claimed: 2009-07-21 - Assignees: Sony Corporation - Applicants: Kouichi Matsuda; Masaki Fukuchi - Inventors: Kouichi Matsuda; Masaki Fukuchi - Web page: https://patentgrep.com/patent/US8751969B2 ## CPC - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G02B 2027/014: {comprising information/image processing systems} - G06T 2219/2004: Aligning objects, relative positioning of parts - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention provides a configuration that allows for a cursor or other object that has moved outside the display section to be displayed as a virtual object and observed. A cursor or object lying in an area outside the area of the display section of a PC or other device is displayed as a virtual object. For example, the display of goggles worn by the user displays a display device such as a PC and the area outside the display device. The three-dimensional position of the cursor or object that has probably moved in response to user operation is calculated, after which the cursor or object is displayed as a virtual object at the calculated position. Further, object information for the object specified by the cursor is acquired and presented. ## Claims ### Claim 1 An information processor comprising: a coordinate processing module adapted to determine whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and output cursor position information to a virtual object management section if the cursor is located outside the area of the first display section; a camera adapted to capture an image made up of a real object including the first display section; a three-dimensional information analysis section adapted to analyze a three-dimensional position of the real object included in the camera-captured image; a second display section adapted to display the camera-captured image; a virtual object management section adapted to generate a virtual object different from the real object included in the camera-captured image and generate a composite image including the generated virtual object and the real object so as to display the composite image on the second display section; and an application execution section adapted to process a specified object specified by the cursor, wherein the virtual object management section calculates a three-dimensional position of the cursor based on the cursor position information supplied from the coordinate processing module so as to display, on the second display section, the composite image in which a virtual cursor is placed at a first calculated position; wherein the application execution section determines whether the specified object is moved by the cursor to locate in or outside the area of the first display section and outputs object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and wherein the virtual object management section calculates a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position. ### Claim 2 (depends on 1) The information processor of claim 1, wherein if the three-dimensional position of the specified object includes the display area of the first display section, the virtual object management section displays, on the second display section, a composite image with an object area image overlapping the display area of the first display section. ### Claim 3 (depends on 1) The information processor of claim 1 further comprising: an object information acquisition section adapted to acquire image data of a real object specified by the cursor placed as the virtual object and search data based on the acquired image data so as to acquire object information, wherein the object information acquisition section outputs the acquired object information to the first display section as display data. ### Claim 4 (depends on 3) The information processor of claim 3, wherein the object information acquisition section accesses a database or a server storing real object image data and object information associated with each other so as to acquire object information through a search based on the real object image data. ### Claim 5 (depends on 1) The information processor of claim 1, wherein the virtual object management section calculates a plane including the display surface of the first display section based on three-dimensional position information of components making up the first display section included in the camera-captured image and calculates the three-dimensional position of the cursor so that the cursor position is placed on the plane. ### Claim 6 (depends on 1) The information processor of claim 1, wherein the cursor is a mouse cursor that moves by mouse operation, and the coordinate processing module receives mouse cursor displacement information resulting from the mouse operation and determines whether the mouse cursor is located in or outside the area of the first display section. ### Claim 7 An information processing method performed by an information processor, the information processing method comprising: determining, by the information processor, whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and outputting cursor position information if the cursor is located outside the area of the first display section; analyzing a three-dimensional position of a real object in a camera-captured image, the camera-captured image including the first display section; generating a virtual object different from the real object included in the camera-captured image; generating a composite image including the generated virtual object and the real object so as to display the composite image on a second display section; calculating a three-dimensional position of the cursor based on the cursor position information so as to display, on the second display section, a composite image in which a virtual cursor is placed at a first calculated position; determining whether a specified object specified by the cursor is moved by the cursor to locate in or outside the area of the first display section and outputting object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and calculating a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position. ### Claim 8 A non-transitory computer-readable medium storing a program which, when executed by an information processor, causes the information processor to: determine whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and output cursor position information to a virtual object management section if the cursor is located outside the area of the first display section; analyze a three-dimensional position of the real object included in the camera-captured image, the camera-captured image including the first display section; generate a virtual object different from the real object included in the camera-captured image and generate a composite image including the generated virtual object and the real object so as to display the composite image on the second display section; calculate a three-dimensional position of the cursor based on the cursor position information supplied from the coordinate processing module so as to display, on the second display section, a composite image in which a virtual cursor is placed at the calculated position; determine whether a specified object specified by the cursor is moved by the cursor to locate in or outside the area of the first display section and output object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and calculate a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position.