# US 10,319,100 B2 · Methods, devices, and systems for identifying and tracking an object with multiple cameras > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,100 B2 - Document ID: US10319100B2 - Status: Granted - Kind: B2 - Application number: 15/669,822 - Filed: 2017-08-04 - Published: 2019-01-10 - Granted: 2019-06-11 - Earliest claimed: 2017-07-07 - Assignees: Guangdong Virtual Reality Technology Co., Ltd. - Applicants: Guangdong Virtual Reality Technology Co., Ltd. - Inventors: Jingwen Dai; Yongtao Hu; Jie He - Web page: https://patentgrep.com/patent/US10319100B2 ## CPC - G06T 7/292: Multi-camera tracking - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/248: {involving reference images or patches} - G06T 7/285: using a sequence of stereo image pairs - G06T 7/74: {involving reference images or patches} - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H10F 39/8053: {Colour filters} - G06T 2200/28: involving image processing hardware - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H10F: INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION ## Abstract An object identifying and tracking device is provided. The device includes a first image capturer to capture a first raw image of an object, a second image capturer to capture a second raw image of the object, a first processor, a second processor and a post processor. The object includes a first light source emitting light of a first wavelength and a second light source emitting light of a second wavelength. The first processor and the second processor respectively receive the first and the second raw images and respectively extract a first group of information associated with the light of the first wavelength and a second group of information associated with the light of the second wavelength emitted from the object. The post processor processes the first group of information and the second group of information to obtain a third group of information. ## Claims ### Claim 1 An object identifying and tracking device, comprising: a first image capturer configured to capture an invisible image of a trackable object, wherein the trackable object comprises at least two light elements of different wavelength combined together to form a light source, the at least two light elements comprises a first light element emitting invisible light and a second light element emitting visible light, the image of the invisible light is captured in the invisible image; a second image capturer configured to capture a visible image of the trackable object, the image of the visible light is captured in the visible image, wherein the first image capturer and the second image capturer are disposed at two ends of an imaging device; a first processor, coupled to the first image capturer, configured to receive the invisible image, determine a first candidate pixel region associated with pixels of the invisible light in the invisible image, and generate a candidate area corresponding to light sources of the visible image based on the first candidate pixel region; a second processor, coupled to the second image capturer, configured to receive the visible image and generate a second candidate pixel region corresponding to light sources from the visible image based on the candidate area; and a post processor, coupled to the first processor and the second processor, configured to calculate 3D positions of light sources in real world based on the first candidate pixel region and the second candidate pixel region. ### Claim 2 (depends on 1) The object identifying and tracking device of claim 1, wherein: the first image capturer comprises a first band-pass filter to allow the invisible light to pass through; and the second image capturer comprises a second band-pass filter to allow the visible light to pass through. ### Claim 3 (depends on 2) The object identifying and tracking device of claim 2, wherein a bandwidth of the first band-pass filter and a bandwidth of the second band-pass filter do not overlap with each other. ### Claim 4 (depends on 3) The object identifying and tracking device of claim 3, wherein a wavelength of the invisible light is in a range from 760nm to 1 mm, and a wavelength of the visible light is in a range from 390 nm to 760 nm. ### Claim 5 (depends on 2) The object identifying and tracking device of claim 2, wherein the second band-pass filter comprises a color filter array. ### Claim 6 (depends on 5) The object identifying and tracking device of claim 5, wherein the color filter array is a Bayer RGB filter pattern. ### Claim 7 (depends on 1) The object identifying and tracking device of claim 1, wherein the first processor further configured to extract: a pre-processed first image based on the invisible image; a first coordinate of a center of the trackable object in the invisible image. ### Claim 8 (depends on 7) The object identifying and tracking device of claim 7, wherein the pre-processed first image is a binarization of the invisible image. ### Claim 9 (depends on 7) The object identifying and tracking device of claim 7, wherein the first coordinate of the center of the trackable object in the invisible image are determined by both said invisible image and the pre-processed image. ### Claim 10 (depends on 7) The object identifying and tracking device of claim 7, wherein the first coordinate coordinates of the center of the trackable object are calculated as a shape center of the first candidate pixel region associated with the trackable object in the invisible image. ### Claim 11 (depends on 7) The object identifying and tracking device of claim 7, wherein the first coordinate of the center of the trackable object are calculated as a weighted mean center of the pixels in the first candidate pixel region associated with the trackable object imaging in the invisible image. ### Claim 12 (depends on 7) The object identifying and tracking device of claim 7, wherein the candidate pixel area of the trackable object in the visible image is obtained by means of epipolar geometry and calculated using a fundamental matrix of the device. ### Claim 13 (depends on 12) The object identifying and tracking device of claim 12, wherein the fundamental matrix of the device is predetermined by calibrating the device before starting up the device, or by On the Fly Calibration after starting up the device. ### Claim 14 (depends on 5) The object identifying and tracking device of claim 5, wherein the second processor further configured to extract at least one of: a pre-processed second image based on said visible image; a second coordinate of a center of the pixels in the second candidate pixel region associated with the trackable object in the visible image; or an identity of the trackable object. ### Claim 15 (depends on 14) The object identifying and tracking device of claim 14, wherein the pre-processed second image is a demosaic image generated from the visible image based on a pattern of the color filter array of the second band-pass filter. ### Claim 16 (depends on 14) The object identifying and tracking device of claim 14, wherein the second coordinate of the center of the trackable object are calculated as a shape center of the pixels in the second candidate pixel region associated with the trackable object in the visible image. ### Claim 17 (depends on 14) The object identifying and tracking device of claim 14, wherein the post processor further configured to extract at least one of: a 3D coordinate of the center of the trackable object in space; an identity of the trackable object; an orientation of the trackable object in space; a trajectory of the trackable object in space; or a velocity and/or an acceleration of the trackable object in space. ### Claim 18 An object identifying and tracking system, comprising: at least one trackable object, comprising at least two light elements of different wavelength combined together to form a light source, wherein the at least two light elements comprises a first light element emitting invisible light and a second light element emitting visible light; a first image capturer, disposed at one end of an imaging device, comprising a first band-pass filter allowing the invisible light to pass through and a first camera capturing light passing through the first band-pass filter; a second image capturer, disposed at the other end of the imaging device, comprising a second band-pass filter allowing the visible light to pass through and a second camera capturing light passing through the second band-pass filter; a first processor, coupled to the first image capturer, configured to instruct the first image capturer to capture an invisible image including the at least one trackable object and image of the invisible light, determine a first candidate pixel region associated with pixels of the invisible light in the invisible image, and generate a candidate area corresponding to light sources of the visible image based on the first candidate pixel region; a second processor, coupled to the second image capturer, configured to instruct the second image capturer to capture a visible image including the at least one trackable object and image of the visible light, and generate a second candidate pixel region corresponding to light sources from the visible image based on the candidate area; a post processor, coupled to the first processor and the second processor, configured to calculate 3D positions of light sources in real world based on the first candidate pixel region and the second candidate pixel region. ### Claim 19 An object identifying and tracking method, comprising: combining at least two light elements of different wavelength together to form a light source inside a trackable object, wherein the at least two light elements comprises a first light element emitting invisible light, and a second light element emitting visible light; disposing the light source onto a place in real world whose location is interested by a virtual reality (VR) system; constructing an image capture system using at least two image sensors disposed at two ends of an imaging device and two band-pass filters, wherein the first band-pass filter allows invisible light to pass through, and the second band-pass filter allows visible light to pass through; capturing a pair of raw images, through the image capture system, by obtaining an invisible image through the first band-pass filter and a visible image through the second band-pass filter, wherein the image of the invisible light emitted by the first light element is captured in the invisible image, the image of the visible light emitted by the second light element is captured in the visible image; processing the invisible image to determine a first candidate pixel region associated with pixels of the invisible light emitted by the first light element in the invisible image and generate a candidate area corresponding to light sources of the visible image based on the first candidate pixel region; processing the visible image to generate a second candidate pixel region corresponding to light sources from the visible image based on the candidate area; and processing the first candidate pixel region and the second candidate pixel region to calculate 3D positions of light sources in real world.