# US 10,482,670 B2 · Method for reproducing object in 3D scene and virtual reality head-mounted device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,670 B2 - Document ID: US10482670B2 - Status: Granted - Kind: B2 - Application number: 15/808,151 - Filed: 2017-11-09 - Published: 2018-03-15 - Granted: 2019-11-19 - Earliest claimed: 2014-12-30 - Assignees: Qingdao GoerTek Technology Co., Ltd. - Applicants: Qingdao GoerTek Technology Co., Ltd. - Inventors: Maoshan Jiang; Xiangjun Zhang; Hongwei Zhou - Web page: https://patentgrep.com/patent/US10482670B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/285: using a sequence of stereo image pairs - G06T 7/564: from contours - G06T 7/593: from stereo images - G06T 7/73: using feature-based methods - G06V 20/20: in augmented reality scenes - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F 18/25: Fusion techniques - G06T 2207/10016: Video; Image sequence - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/30196: Human being; Person - G06T 2207/30241: Trajectory - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure discloses a method for reproducing an object in a 3D scene and a virtual reality head-mounted device. The method comprises: simultaneously gathering at least two channels of video stream data in real time at different angles for an object to be displayed by using at least two image gatherers, wherein first field angles of the image gatherers at least partially overlap in space, and the first field angles are a horizontal field angle, a vertical field angle or a diagonal field angle; identifying out a shape of the object varying in real time from the at least two channels of video stream data; according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory; and processing the shape of the object varying in real time and the corresponding object motion trajectory into a 3D image and in real time superposition-displaying the 3D image into the 3D scene. The technical solutions of the present disclosure can reproduce the object in the 3D scene, and achieve the purpose of displaying the realistic object in the 3D scene. ## Claims ### Claim 1 A method for reproducing an object in a 3D scene, wherein the method comprises: simultaneously gathering at least two channels of video stream data in real time at different angles for an object to be displayed by using at least two image gatherers, wherein first field angles of the image gatherers at least partially overlap in space, and the first field angles are a horizontal field angle, a vertical field angle or a diagonal field angle; identifying out a shape of the object varying in real time from the at least two channels of video stream data; according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory; and processing the shape of the object varying in real time and the corresponding object motion trajectory into a 3D image and in real time superposition-displaying the 3D image into the 3D scene; wherein the identifying out a shape of the object varying in real time from the at least two channels of video stream data comprises: performing sampling processing of each of the at least two channels of video stream data, to obtain video image data that are sampled each time; judging whether the video image data contain an object, and if so, performing binarization processing to the video image data, to extract contour information of the object; identifying out a shape of the object that is corresponding to the contour information of the object from a preset object model database; and combining shapes of the object that are identified out in each sampling of respective channels of video stream data, to obtain the shape of the object varying in real time; wherein the according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory comprises: obtaining relative spatial position information of the shape of the object varying in real time: according to contacts that are determined on the shape of the object varying in real time, obtaining variation information of the contacts on the shape of the object varying in real time, wherein the contacts are characteristic key points for identifying the object; and according to the relative spatial position information and the variation information of the contacts, obtaining a corresponding object motion trajectory from a preset motion trajectory database; and wherein the obtaining relative spatial position information of the shape of the object varying in real time comprises: obtaining angle information with the shape of the object varied, from video image information of the at least two channels of video stream data; obtaining distance information of the object according to the angle information with the shape of the object varied; or sensing distance information of the object in real time by a distance sensor; and according to the angle information with the shape of the object varied and the distance information of the object, obtaining the relative spatial position information of the object. ### Claim 2 (depends on 1) The method according to claim 1, wherein second field angles of the image gatherers at least partially overlap in space, and the second field angles comprise at least one of field angles other than the first field angles. ### Claim 3 (depends on 1) The method according to claim 1, wherein the obtaining relative spatial position information of the shape of the object varying in real time comprises: gathering video raw data of the object, and acquiring nine-axis sensor data; and processing the video raw data and the nine-axis sensor data, to obtain relative spatial position information of the shape of the object varying in real time. ### Claim 4 (depends on 1) The method according to claim 1, wherein the object is a hand of the user; and correspondingly, the simultaneously gathering at least two channels of video stream data in real time at different angles for an object to be displayed by using at least two image gatherers comprises: gathering at least two channels of video stream data in real time at different angles for an hand gesture of the user by using at least two image gatherers; the identifying out a shape of the object varying in real time from the at least two channels of video stream data comprises: identifying out a hand gesture shape varying in real time from the at least two channels of video stream data; the according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory comprises: resolving the hand gesture shape varying in real time, to obtain the corresponding hand gesture motion; and the processing the shape of the object varying in real time and the corresponding object motion trajectory into a 3D image and in real time superposition-displaying the 3D image into the 3D scene comprises: processing the hand gesture motion into a 3D image and in real time displaying the 3D image into the 3D scene. ### Claim 5 (depends on 4) The method according to claim 4, wherein the method further comprises: obtaining a hand gesture meaning of the hand gesture motion and an operation instruction that is corresponding to the hand gesture meaning from a preset meaning database; and executing the operation instruction, to control the 3D scene to conduct the operation of the hand gesture meaning. ### Claim 6 (depends on 4) The method according to claim 4, wherein the method further comprises: acquiring spatial information of a preset range by using an infrared sensor; combining the at least two channels of video stream data, and displaying in the 3D scene the video stream data that are generated after the combining; and according to the hand gesture motion that is obtained by the resolving and the spatial information of the preset range, controlling a target in the 3D scene. ### Claim 7 A virtual reality head-mounted device, wherein the virtual reality head-mounted device comprises at least two image gatherers, a memory storing an instruction and at least one processor, wherein the at least one processor executes the following operations according to the instruction stored on the memory: acquiring at least two channels of video stream data that are simultaneously gathered in real time at different angles for an object to be displayed from an output of the at least two image gatherers, wherein first field angles of the image gatherers at least partially overlap in space, and the first field angles are a horizontal field angle, a vertical field angle or a diagonal field angle; identifying out a shape of the object varying in real time from the at least two channels of video stream data; according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory; and processing the shape of the object varying in real time and the corresponding object motion trajectory into a 3D image and in real time superposition-displaying the 3D image into the 3D scene; wherein the identifying out a shape of the object varying in real time from the at least two channels of video stream data comprises: performing sampling processing of each of the at least two channels of video stream data, to obtain video image data that are sampled each time; judging whether the video image data contain the object, and if so, performing binarization processing to the video image data, to extract contour information of the object; identifying out a shape of the object that is corresponding to the contour information of the object from a preset object model database; and combining shapes of the object that are identified out in each sampling of respective channels of video stream data, to obtain the shape of the object varying in real time; wherein the according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory comprises: obtaining relative spatial position information of the shape of the object varying in real time; according to contacts that are determined on the shape of the object varying in real time, obtaining variation information of the contacts on the shape of the object varying in real time, wherein the contacts are characteristic key points for identifying the object; and according to the relative spatial position information and the variation information of the contacts, obtaining a corresponding object motion trajectory from a preset motion trajectory database; and wherein the obtaining relative spatial position information of the shape of the object varying in real time comprises: obtaining angle information with the shape of the object varied, from video image information of the at least two channels of video stream data; obtaining distance information of the object according to the angle information with the shape of the object varied; or sensing distance information of the object in real time by a distance sensor; and according to the angle information with the shape of the object varied and the distance information of the object, obtaining the relative spatial position information of the object. ### Claim 8 (depends on 7) The device according to claim 7, wherein second field angles of the image gatherers at least partially overlap in space, and the second field angles comprise at least one of field angles other than the first field angles. ### Claim 9 The device according to 7, wherein the device further comprises a nine-axis sensor; and the obtaining relative spatial position information of the shape of the object varying in real time comprises: gathering video raw data of the object, and acquiring nine-axis sensor data; and processing the video raw data and the nine-axis sensor data, to obtain relative spatial position information of the shape of the object varying in real time. ### Claim 10 (depends on 7) The device according to claim 7, wherein the object is a hand of the user; and the operations comprise: acquiring at least two channels of video stream data that are simultaneously gathered in real time at different angles for a hand gesture of the user from outputs of the at least two image gatherers; identifying out a hand gesture shape varying in real time from the at least two channels of video stream data; resolving the hand gesture shape varying in real time, to obtain the corresponding hand gesture motion; and processing the hand gesture motion into a 3D image and in real time displaying the 3D image into the 3D scene. ### Claim 11 (depends on 10) The device according to claim 10, wherein the operations further comprise: obtaining a hand gesture meaning of the hand gesture motion and an operation instruction that is corresponding to the hand gesture meaning from a preset meaning database; and executing the operation instruction, to control the 3D scene to conduct the operation of the hand gesture meaning. ### Claim 12 (depends on 10) The device according to claim 10, wherein the device further comprises: an infrared sensor; and the operations further comprise: acquiring peripheral spatial position information of the hand of the user by using the infrared sensor; combining the at least two channels of video stream data, and displaying in the 3D scene the video stream data that are generated after the combining; and according to the hand gesture motion that is obtained by the resolving and the peripheral spatial position information of the hand of the user, controlling a target in the 3D scene. ### Claim 13 A non-transitory computer readable storage medium, having a program instruction stored thereon, wherein the program instruction when executed by a controlling unit implements the following steps: simultaneously gathering at least two channels of video stream data in real time at different angles for an object to be displayed by using at least two image gatherers, wherein first field angles of the image gatherers at least partially overlap in space, and the first field angles are a horizontal field angle, a vertical field angle or a diagonal field angle; identifying out a shape of the object varying in real time from the at least two channels of video stream data; according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory; and processing the shape of the object varying in real time and the corresponding object motion trajectory into a 3D image and in real time superposition-displaying the 3D image into the 3D scene; wherein the identifying out a shape of the object varying in real time from the at least two channels of video stream data comprises: performing sampling processing of each of the at least two channels of video stream data, to obtain video image data that are sampled each time; judging whether the video image data contain an object, and if so, performing binarization processing to the video image data, to extract contour information of the object; identifying out a shape of the object that is corresponding to the contour information of the object from a preset object model database; and combining shapes of the object that are identified out in each sampling of respective channels of video stream data, to obtain the shape of the object varying in real time; wherein the according to the shape of the object varying in real time, obtaining a corresponding object motion trajectory comprises: obtaining relative spatial position information of the shape of the object varying in real time; according to contacts that are determined on the shape of the object varying in real time, obtaining variation information of the contacts on the shape of the object varying in real time, wherein the contacts are characteristic key points for identifying the object; and according to the relative spatial position information and the variation information of the contacts, obtaining a corresponding object motion trajectory from a preset motion trajectory database; and wherein the obtaining relative spatial position information of the shape of the object varying in real time comprises: obtaining angle information with the shape of the object varied, from video image information of the at least two channels of video stream data; obtaining distance information of the object according to the angle information with the shape of the object varied; or sensing distance information of the object in real time by a distance sensor; and according to the angle information with the shape of the object varied and the distance information of the object, obtaining the relative spatial position information of the object. ### Claim 14 (depends on 13) The non-transitory computer readable storage medium according to claim 13, wherein second field angles of the image gatherers at least partially overlap in space, and the second field angles comprise at least one of field angles other than the first field angles.