# US 9,438,879 B2 · Combining 3D image and graphical data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,438,879 B2 - Document ID: US9438879B2 - Status: Granted - Kind: B2 - Application number: 13/201,736 - Filed: 2010-02-09 - Published: 2011-12-15 - Granted: 2016-09-06 - Earliest claimed: 2009-02-17 - Assignees: Koninklijke Philips N.V. - Applicants: Philip Steven Newton; Gerardus Wilhelmus Theodorus Van Der Heijden; Wiebe De Haan; Johan Cornelis Talstra; Wilhelmus Hendrikus Alfonsus Bruls; Georgios Parlantzas; Marc Helbing; Christian Benien; Vasanth Philomin; Christiaan Varekamp - Inventors: Philip Steven Newton; Gerardus Wilhelmus Theodorus Van Der Heijden; Wiebe De Haan; Johan Cornelis Talstra; Wilhelmus Hendrikus Alfonsus Bruls; Georgios Parlantzas; Marc Helbing; Christian Benien; Vasanth Philomin; Christiaan Varekamp - Web page: https://patentgrep.com/patent/US9438879B2 ## CPC - H04N 13/361: Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 13/183: On-screen display [OSD] information, e.g. subtitles or menus - H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Three dimensional [3D] image data and auxiliary graphical data are combined for rendering on a 3D display by detecting depth values occurring in the 3D image data and setting auxiliary depth values for the auxiliary graphical data adaptively in dependence of the detected depth values. The 3D image data and the auxiliary graphical data at the auxiliary depth value are combined based on the depth values of the 3D image data. First an area of attention in the 3D image data is detected. A depth pattern for the area of attention is determined, and the auxiliary depth values are set in dependence of the depth pattern. ## Claims ### Claim 1 A method of combining three dimensional (3D) image and auxiliary data in a display area of a stereoscopic screen, the method comprising acts of: obtaining information describing scaling and/or shifting for use with the 3D image; resizing the 3D image in accordance with the information by scaling and/or shifting in at least one of a horizontal direction and a vertical direction of a display area to create a spatial area fitting in a portion of the display area that is not occupied by the 3D image to be occupied by the auxiliary data; combining the scaled and/or shifted 3D image and the auxiliary data such that the auxiliary data is placed within the spatial area; creating a left border region and a right border region that extend into the portion of the display area that is occupied by the 3D image; and adjusting a depth position of the left and right border regions such that each border region is closer to a viewer than any of the portion of the 3D image in the display area that is overlapped by the left and right border regions. ### Claim 2 (depends on 1) The method of claim 1, wherein the auxiliary data is selected from at least one of: two dimensional subtitle information, two dimensional subpicture information, 3D subtitle information and 3D subpicture information. ### Claim 3 (depends on 1) The method of claim 1, wherein the information relates to scaling and/or shifting within the direction of the display area. ### Claim 4 (depends on 3) The method of claim 3, wherein the information does not relate to scaling and/or shifting within the depth direction perpendicular to the display area. ### Claim 5 (depends on 1) The method of claim 1, wherein the information is a scale factor. ### Claim 6 (depends on 5) The method of claim 5, wherein the scale factor applies to scaling in x and y directions of the display area. ### Claim 7 (depends on 1) The method of claim 1, wherein the information is an offset in at least one of: the horizontal direction of the display area and the vertical direction of the display area. ### Claim 8 (depends on 1) The method of claim 1, wherein the auxiliary data is located entirely within the spatial area defined by the 3D image but not occupied by the scaled and/or shifted 3D image. ### Claim 9 (depends on 8) The method of claim 8, wherein the spatial area is padded with black background information. ### Claim 10 A three dimensional (3D) display device for combining 3D image and auxiliary data in a display area of the device, the device comprising: a processor configured to: obtain information describing scaling and/or shifting for use with the 3D image, resize the 3D image in accordance with the information by scaling and/or shifting in at least one of a horizontal direction and a vertical direction of a display area to create a spatial area in a portion of the display area that is not occupied by the 3D image, combine the scaled and/or shifted 3D image and the auxiliary data such that the auxiliary data is placed within the spatial area, create a left border region and a right border region that extend into the portion of the display area that is occupied by the 3D image, and adjust a depth position of the left and right border regions such that each border region is closer to a viewer than any of the portion of the 3D image in the display area that is overlapped by the left and right border regions. ### Claim 11 (depends on 10) The display device of claim 10, wherein the depth position of the left and right borders cause the borders to appear closer to the viewer than the display device. ### Claim 12 (depends on 11) The display device of claim 11, further comprising: creating at least a bottom border region that extends into the portion of the display area that is occupied by the 3D image; and adjusting a depth position of the bottom border regions such that the bottom border region is closer to the viewer than any of the portion of the 3D image in the display area that is overlapped by the bottom border region, and closer to the viewer than the display device. ### Claim 13 (depends on 10) The display device of claim 10, wherein the left and right borders comprise a texture pattern that corresponds to an external surface of the display device. ### Claim 14 (depends on 10) The display device of claim 10, wherein the left and right borders comprise black stripes. ### Claim 15 A three dimensional (3D) source device for combining 3D image and auxiliary data in a display area, the device comprising: a processor configured to: obtain information describing scaling and/or shifting for use with the 3D image, resize the 3D image in accordance with the information by scaling and/or shifting in at least one of a horizontal direction and a vertical direction of a display area to create a spatial area in a portion of the display area that is not occupied by the 3D image, combine the scaled and/or shifted 3D image and the auxiliary data such that the auxiliary data is placed within the spatial area, create a left border region and a right border region that extend into the portion of the display area that is occupied by the 3D image, and adjust a depth position of the left and right border regions such that each border region is closer to a viewer than any of the portion of the 3D image in the display area that is overlapped by the left and right border regions. ### Claim 16 (depends on 10) The display device of claim 10, wherein the spatial area comprises a black background to the auxiliary data. ### Claim 17 A non-transitory computer readable information carrier medium comprising a three dimensional (3D) image and auxiliary data, the information carrier further comprising a program that, when executed on a processing system, causes the processing system to: obtain information describing scaling and/or shifting for use with the 3D image; resize the 3D image in accordance with the information by scaling and/or shifting in at least one of a any of horizontal direction and vertical direction of a display area to create a spatial area in a portion of the display area that is not occupied by the 3D image; and combine the scaled and/or shifted 3D image and the auxiliary data such that the auxiliary data is placed within the spatial area; create a left border region and a right border region that extend into the portion of the display area that is occupied by the 3D image, and adjust a depth position of the left and right border regions such that each border region is closer to a viewer than any of the portion of the 3D image in the display area that is overlapped by the left and right border regions. ### Claim 18 A non-transitory computer readable medium comprising a program that, when executed by a processor, causes the processor to: obtain information describing scaling and/or shifting for use with the 3D image; resize the 3D in accordance with the information image by scaling and/or shifting in at least one of a horizontal direction and a vertical direction of a display area of a display device to create a spatial area in a portion of the display area that is not occupied by the 3D image; combine the scaled and/or shifted 3D image and the auxiliary data such that the auxiliary data is placed within the spatial area; create a left border region and a right border region that extend into the portion of the display area that is occupied by the 3D image, and adjust a depth position of the left and right border regions such that each border region is closer to a viewer than any of the portion of the 3D image in the display area that is overlapped by the left and right border regions. ### Claim 19 (depends on 18) The median of claim 18, wherein the spatial area comprises a black background to the auxiliary data. ### Claim 20 (depends on 18) The median of claim 18, wherein the depth position of the left and right borders cause the borders to appear closer to the viewer than the display device. ### Claim 21 (depends on 20) The median of claim 20, further comprising: creating at least a bottom border region that extends into the portion of the display area that is occupied by the 3D image; and adjusting a depth position of the bottom border regions such that the bottom border region is closer to the viewer than any of the portion of the 3D image in the display area that is overlapped by the bottom border region, and closer to the viewer than the display device. ### Claim 22 (depends on 18) The median of claim 18, wherein the left and right borders comprise a texture pattern that corresponds to an external surface of the display device. ### Claim 23 (depends on 18) The median of claim 18, wherein the left and right borders comprise black stripes.