# US 10,481,679 B2 · Method and system for optical-inertial tracking of a moving object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,679 B2 - Document ID: US10481679B2 - Status: Granted - Kind: B2 - Application number: 15/844,967 - Filed: 2017-12-18 - Published: 2019-06-20 - Granted: 2019-11-19 - Earliest claimed: 2017-12-18 - Assignees: Alt LLC - Applicants: Alt LLC - Inventors: Petr Vyacheslavovich Sevostianov - Web page: https://patentgrep.com/patent/US10481679B2 ## CPC - G06F 3/012: {Head tracking input arrangements} - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/235: using a wireless connection, e.g. infrared or piconet - A63F 13/25: Output arrangements for video game devices - A63F 13/44: involving timing of operations, e.g. performing an action within a time slot - A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character - A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G09B 9/00: Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131}) - A63F 2300/8082: Virtual reality - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Method for inside-out optical-inertial tracking of a movable object includes providing IR light by a plurality of markers; reading a picture from an optical sensor and detecting pixels satisfying predefined conditions by a first data processing device; detecting marker blobs, based on the pixels, and determining parameters of the blobs by the first data processing device; reading data from an inertial sensor; generating a consolidated data flow comprising the parameters of the marker blobs and the data of the inertial sensor by the first data processing device, and transmitting the data flow to a second communication device; determining a tracker positioning data in a coordinate system of a motion area of the object, based on the consolidated data flow, by a second data processing device; validating a model of a tracker motion path, based on tracker positioning data; extrapolating tracker positioning data onto an extrapolation horizon, based on the model. ## Claims ### Claim 1 A method of optical-inertial tracking a movable object, comprising the steps of: (a) providing emission of IR light by a plurality of markers fixedly mounted so as to be visible from the movable object in order to perform inside-out tracking; (b) reading a picture, by a first data processing device, from an optical sensor that is mounted on the movable object and detecting pixels satisfying predefined conditions in the read picture; (c) detecting marker blobs, based on the detected pixels, and determining parameters of the marker blobs by the first data processing device; (d) reading data, by the first data processing device, from at least one inertial sensor that is mounted on the movable object; (e) generating, by the first data processing device, a consolidated data flow comprising the parameters of the marker blobs and the data from the at least one inertial sensor, and transmitting the consolidated data flow by a first communication device to a second communication device; (f) receiving the consolidated data flow by the second communication device and determining, by a second data processing device, a tracker positioning data in a coordinate system of a motion area of the movable object, based on the consolidated data flow; (g) validating, by the second data processing device, a mathematical model of a tracker motion path, based on the tracker positioning data; and (h) extrapolating, by the second data processing device, the tracker positioning data onto an extrapolation horizon, based on the validated mathematical model of the tracker motion path. ### Claim 2 (depends on 1) The method of claim 1, wherein the picture from the optical sensor in step (b) is read form a portion of the optical sensor, wherein a size of the portion is smaller than an entire frame size of the optical sensor. ### Claim 3 (depends on 2) The method of claim 2, wherein the portion to be read is a tracing window. ### Claim 4 (depends on 1) The method of claim 1, wherein the predefined conditions in step (b) include threshold values of brightness of the pixels. ### Claim 5 (depends on 4) The method of claim 4, wherein the predefined conditions include a requirement related to aggregation of the pixels. ### Claim 6 (depends on 1) The method of claim 1, wherein detecting the pixels satisfying the predefined conditions in step (b) is started before an end of reading of the picture from the optical sensor. ### Claim 7 (depends on 1) The method of claim 1, wherein step (b) is performed by an FPGA. ### Claim 8 (depends on 1) The method of claim 1, wherein the parameters of each of the marker blobs in step (c) include coordinates of its center in a coordinate system of a frame of the optical sensor, an effective radius and a weighted brightness. ### Claim 9 (depends on 8) The method of claim 8, wherein the coordinates of the center are determined with a subpixel accuracy. ### Claim 10 (depends on 8) The method of claim 8, wherein the parameters of the marker blob include a timestamp indicating time when a frame was captured. ### Claim 11 (depends on 1) The method of claim 1, wherein step (c) is performed by a microcontroller. ### Claim 12 (depends on 1) The method of claim 1, wherein the inertial sensor in step (d) is a gyroscope. ### Claim 13 (depends on 1) The method of claim 1, wherein the inertial sensor in step (d) is an accelerometer. ### Claim 14 (depends on 1) The method of claim 1, wherein the data read from the inertial sensor in step (d) includes a timestamp indicating a time when the data was read. ### Claim 15 (depends on 1) The method of claim 1, wherein the consolidated data flow generated in step (e) includes data of multiple trackers. ### Claim 16 (depends on 15) The method of claim 15, wherein the consolidated data flow in step (f) is divided into multiple consolidated data flows, whose number corresponds to a number of the trackers. ### Claim 17 (depends on 1) The method of claim 1, wherein the first data processing device and the first communication device are located in the tracker, and the second data processing device and the second communication device are located in a host. ### Claim 18 (depends on 17) The method of claim 17, wherein the tracker is combined with the host. ### Claim 19 (depends on 1) The method of claim 1, wherein the markers are identified based on the parameters of the marker blobs, so as to determine the tracker positioning data in step (f). ### Claim 20 (depends on 19) The method of claim 19, wherein the identification of the markers is performed using a predefined optical pattern. ### Claim 21 (depends on 20) The method of claim 20, wherein the optical pattern is a linear optical pattern. ### Claim 22 (depends on 21) The method of claim 21, wherein the linear optical pattern includes blobs corresponding to a group of three markers aligned in a line, wherein a distance between a first marker and a second marker is twice as much as a distance between the second marker and a third marker. ### Claim 23 (depends on 1) The method of claim 1, wherein a set of historic tracking data is generated, based on the tracker positioning data determined in step (f) for different time points. ### Claim 24 (depends on 23) The method of claim 23, wherein the validation of the mathematical model of the tracker motion path in step (g) is performed using the set of historic tracking data. ### Claim 25 (depends on 24) The method of claim 24, wherein the set of historic tracking data includes a set of data S comprising a multi-dimensional vector that characterizes position and orientation of the tracker, and a set of data L comprising data of one or more inertial sensors. ### Claim 26 (depends on 25) The method of claim 25, wherein the set of historic tracking data includes a set of data OF comprising information on rays in a local coordinate system of the tracker, the rays directed to the markers, whose blobs were detected in step (c). ### Claim 27 (depends on 26) The method of claim 26, wherein the validation of the mathematical model of the tracker motion path in step (g) includes calculating the data S in a backward direction through an entire depth of the historic tracking data, optimizing the data S in an earliest time point of the history, and then calculating an optimized data S in a forward direction through the entire depth of the historic tracking data. ### Claim 28 (depends on 27) The method of claim 27, wherein a vector E of the tracker positioning error is calculated by comparison of the marker coordinates and based on the data OF and predetermined actual marker coordinates, in a backward direction through entire depth of the historic tracking data, and the optimization of the data S in the earliest time point of the history is performed, using the vector E of the tracker positioning error. ### Claim 29 (depends on 28) The method of claim 28, wherein the marker coordinates that are calculated based on the data OF are polar coordinates. ### Claim 30 (depends on 28) The method of claim 28, wherein the optimization of the data S in the earliest point of the history is performed using a least squares method. ### Claim 31 (depends on 30) The method of claim 30, wherein the optimization of the data S in the earliest point of the history is performed, using a Levenberg-Marquardt algorithm. ### Claim 32 (depends on 28) The method of claim 28, wherein the vector E of the tracker positioning error is calculated in a forward direction through entire depth of the historic tracking data, and the validation of the mathematical model of the tracker motion path is performed by comparing the vectors E of the tracker positioning error calculated in the forward direction and the vectors E of the tracker positioning error calculated in the backward direction, and by further calculating a target function used for judging relevance of the mathematical model of the tracker motion path. ### Claim 33 (depends on 32) The method of claim 32, wherein the target function is a weighting function. ### Claim 34 (depends on 32) The method of claim 32, wherein the validation of the mathematical model of the tracker motion path results in the optimized data S, which ensures minimal error of the tracker positioning. ### Claim 35 (depends on 1) The method of claim 1, wherein the determination of the tracker positioning data in step (f) is performed each time new data of the at least one inertial sensor is received in the consolidated data flow. ### Claim 36 (depends on 1) The method of claim 1, wherein the validation of the mathematical model of the tracker motion path in step (g) is performed each time new parameters of the marker blobs are received in the consolidated data flow. ### Claim 37 (depends on 1) The method of claim 1, wherein the extrapolation of the tracker positioning data in step (h) is performed by defining an extrapolation horizon and an extrapolation base, and by further extrapolating the tracker motion path onto the extrapolation horizon. ### Claim 38 (depends on 37) The method of claim 37, wherein the extrapolation horizon is defined based on data received from a 3D-engine that renders a 3D-scene. ### Claim 39 (depends on 1) The method of claim 1, further comprising rendering a 3D-scene based on the extrapolated tracker positioning data, and outputting the 3D-scene into a head-mounted display of a user in a VR/AR system. ### Claim 40 A system for optical-inertial tracking a movable object, the system comprising: at least one plurality of markers fixedly mounted so as to be visible from the movable object in order to perform inside-out tracking, including IR light emitting markers; at least one tracker mounted on the movable object, and including an optical sensor, at least one inertial sensor, a first data processing device and a first communication device, wherein the tracker is configured to read, by the first data processing device, a picture from the optical sensor, detect, by the first data processing device, pixels satisfying predefined conditions in the read picture, detect, by the first data processing device, marker blobs based on the detected pixels, to determine, by the first data processing device, parameters of the marker blobs, read, by the first data processing device, data from the at least one inertial sensor, generate, by the first data processing device, a consolidated data flow including the parameters of the marker blobs and the data of the at least one inertial sensor, and transmit, by the first communication device, the consolidated data flow to a host; at least one host comprising a second communication device and a second data processing device, wherein the host is configured to receive, by the second communication device, the consolidated data flow, determine, by the second data processing device, a tracker positioning data in a coordinate system of a motion area of the movable object, based on the consolidated data flow, validate, by the second data processing device, a mathematical model of a tracker motion path, based on the tracker positioning data, and extrapolate, by the second data processing device, the tracker positioning data onto an extrapolation horizon, based on the validated mathematical model of the tracker motion path. ### Claim 41 (depends on 40) The system of claim 40, wherein the first data processing device is configured to read the picture from a portion of the optical sensor, wherein the portion is smaller than an entire frame size of the optical sensor. ### Claim 42 (depends on 41) The system of claim 41, wherein the portion is a tracing window. ### Claim 43 (depends on 40) The system of claim 40, wherein the predefined conditions include a threshold value of brightness of the pixel. ### Claim 44 (depends on 43) The system of claim 43, wherein the predefined conditions include a requirement related to aggregation of the pixels. ### Claim 45 (depends on 40) The system of claim 40, wherein the first data processing device is configured to start detecting pixels satisfying the predefined conditions before reading the picture from the optical sensor is finished. ### Claim 46 (depends on 40) The system of claim 40, wherein the first data processing device includes an FPGA configured to read the picture from the optical sensor and detect pixels satisfying the predefined conditions in the read picture. ### Claim 47 (depends on 40) The system of claim 40, wherein the parameters of each of the marker blobs include coordinates of its center in a coordinate system of a frame of the optical sensor, an effective radius and a weighted brightness. ### Claim 48 (depends on 47) The system of claim 47, wherein the first data processing device is configured to determine the coordinates of the center with a subpixel accuracy. ### Claim 49 (depends on 47) The system of claim 47, wherein the parameters of the marker blob include a timestamp indicating a time when an appropriate frame was captured. ### Claim 50 (depends on 40) The system of claim 40, wherein the first data processing device includes a microcontroller configured to detect the marker blobs, based on the detected pixels, and to determine parameters of the marker blobs. ### Claim 51 (depends on 40) The system of claim 40, wherein the inertial sensor is a gyroscope. ### Claim 52 (depends on 40) The system of claim 40, wherein the inertial sensor is an accelerometer. ### Claim 53 (depends on 40) The system of claim 40, wherein the data read from the inertial sensor includes a timestamp indicating a time when the data was read. ### Claim 54 (depends on 40) The system of claim 40, wherein the consolidated data flow includes data of multiple trackers. ### Claim 55 (depends on 54) The system of claim 54, wherein the second data processing device is configured to divide the consolidated data flow into multiple consolidated data flows whose number corresponds to a number of the trackers. ### Claim 56 (depends on 40) The system of claim 40, wherein the tracker is combined with the host. ### Claim 57 (depends on 40) The system of claim 40, wherein the second data processing device is configured to identify the markers, based on the parameters of the marker blobs, so as to determine the tracker positioning data. ### Claim 58 (depends on 57) The system of claim 57, wherein the second data processing device is configured to identify the markers using a predefined optical pattern. ### Claim 59 (depends on 58) The system of claim 58, wherein the predefined optical pattern is a linear optical pattern. ### Claim 60 (depends on 59) The system of claim 59, wherein the linear optical pattern includes blobs corresponding to a group of three markers aligned in a line, wherein a distance between a first marker and a second marker is twice a distance between the second marker and a third marker. ### Claim 61 (depends on 40) The system of claim 40, wherein the second data processing device is configured to generate a set of historic tracking data based on the tracker positioning data determined for different time points. ### Claim 62 (depends on 61) The system of claim 61, wherein the second data processing device is configured to validate the mathematical model of the tracker motion path using the set of historic tracking data. ### Claim 63 (depends on 62) The system of claim 62, wherein the set of historic tracking data includes a set of data S comprising a multi-dimensional vector characterizing position and orientation of the tracker, and a set of data L including data of one or more inertial sensors, wherein the data S and the data L correspond to parameters of the mathematical model of the tracker motion path. ### Claim 64 (depends on 63) The system of claim 63, wherein the set of historical tracking data includes a set of data OF comprising information on rays in a local coordinate system of the tracker, the rays directed to the markers, which blobs were detected. ### Claim 65 (depends on 64) The system of claim 64, wherein the second data processing device is configured to calculate the data S in a backward direction through entire depth of the historic tracking data, optimize the data S in an earliest time point of the history, and then calculate an optimized data S in a forward direction through the entire depth of the historic tracking data during the validation of the mathematical model of the tracker motion path. ### Claim 66 (depends on 64) The system of claim 64, wherein the second data processing device is configured to calculate a vector E of the tracker positioning error, which is determined by comparison of the marker coordinates calculated, based on the data OF, and predetermined actual marker coordinates, in a backward direction through an entire depth of the historic tracking data, and to optimize the data S in an earliest time point of the history, using the vector E of the tracker positioning error. ### Claim 67 (depends on 66) The system of claim 66, wherein the marker coordinates calculated based on the data OF are polar coordinates. ### Claim 68 (depends on 66) The system of claim 66, wherein the optimization of the data S in the earliest time point of the history is performed, using a least squares method. ### Claim 69 (depends on 68) The system of claim 68, wherein the optimization of the data S in the earliest time point of the history is performed, using a Levenberg-Marquardt algorithm. ### Claim 70 (depends on 66) The system of claim 66, wherein the second data processing device is configured to calculate the vector E of the tracker positioning error in a forward direction through entire depth of the historical tracking data, and validate the mathematical model of the tracker motion path by comparing the vectors E of the tracker positioning error calculated in the forward direction and the vectors E of the tracker positioning error calculated in the backward direction, and by further calculating a target function for judging relevance of the mathematical model of the tracker motion path. ### Claim 71 (depends on 70) The system of claim 70, wherein the target function is a weighting function. ### Claim 72 (depends on 70) The system of claim 70, wherein the validation of the mathematical model of the tracker motion path results in the optimized data S, which ensures minimal error of the tracker positioning. ### Claim 73 (depends on 40) The system of claim 40, wherein the second data processing device is configured to determine the tracker positioning data each time, when new data of the at least one inertial sensor is received in the consolidated data flow. ### Claim 74 (depends on 40) The system of claim 40, wherein the second data processing device is configured to validate the mathematical model of the tracker motion path each time new parameters of the marker blobs are received in the consolidated data flow. ### Claim 75 (depends on 40) The system of claim 40, wherein the second data processing device is configured to define an extrapolation horizon and an extrapolation base, and to extrapolate the tracker motion path onto the extrapolation horizon. ### Claim 76 (depends on 75) The system of claim 75, wherein the extrapolation horizon may be defined, based on data received from a 3D-engine configured to render a 3D-scene. ### Claim 77 (depends on 40) The system of claim 40, further comprising at least one head-mounted display configured to present a 3D-scene to a VR/AR system user, wherein the extrapolation of the tracker positioning data is performed so as to further render the 3D-scene, based on the extrapolated tracker positioning data, and to output the 3D-scene into the head-mounted display of the VR/AR system user.