# US 2019/0244384 A1 · Method, device and system for estimating a pose of a camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0244384 A1 - Document ID: US20190244384A1 - Status: Published application - Kind: A1 - Application number: 16/266,079 - Filed: 2019-02-03 - Published: 2019-08-08 - Earliest claimed: 2018-02-05 - Applicants: Interdigital Ce Patent Holdings - Inventors: Tao LUO; Caroline BAILLARD; Fabien SERVANT - Web page: https://patentgrep.com/patent/US20190244384A1 ## CPC - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/32: using correlation-based methods - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - G06T 2207/10016: Video; Image sequence - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method and device are described for time aligning a current frame of a current video stream with respect to reference frames of a reference video stream, the current video stream corresponding to at least one part of the reference video stream. The method includes: obtaining at least two reference frames corresponding to at least part of at least some of multiple displayed frames displayed on a display; obtaining current frame from images of said displayed frames captured with a camera by shooting said display; and determining a time alignment of said current frame with respect to said at least two reference frames by correlating said current frame with a time interpolation of said at least two reference frames. The time interpolation being based on correlations between the current frame and respectively each of the at least two reference frames, and on at least one correlation between at least two of the selected reference frames together. ## Claims ### Claim 1 A method for time aligning a current frame of a current video stream with respect to reference frames of a reference video stream, the method comprising: obtaining two reference frames corresponding to frames of the reference video stream displayed on a display; obtaining the current frame from a camera shooting said display; time aligning said current frame as a function of a correlation between said current frame and each of said two reference frames, and of a correlation between the two reference frames. ### Claim 2 (depends on 1) The method according to claim 1, comprising determining a pose estimation associated with said current frame, from spatially correlating said current frame and said at least two reference frames based on said time aligning. ### Claim 3 (depends on 1) The method according to claim 1, wherein said time aligning is an interpolation based on the correlations S1 and S2 between said current frame and each of the two reference frames, and the correlation S3 between the two frames, said interpolation corresponding to a linear normalized time positioning weight α between the two reference frames, said time positioning weight α being given by: α = - - S 2  S 3 + S 1 S 1  S 3 + S 2  S 3 - S 1 - S 2 . ### Claim 4 (depends on 3) The method according to claim 3, wherein the time aligning of said current frame is possible with respect to at least two sets of a predetermined number of at least two reference frames, said method comprising determining said time aligning with respect to one of said sets of said at least two reference frames that provides a highest correlation of said current frame with the interpolation of said reference frames, among said at least two sets of said at least two reference frames. ### Claim 5 (depends on 1) The method according to claim 1, wherein said time aligning comprises selecting said two reference frames within a time window including a current time corresponding to said current frame and having a predetermined time length. ### Claim 6 (depends on 1) The method according to claim 1, wherein said correlations are respectively applied to centered versions of said current frame and said two reference frames, a centered version of a frame comprising pixels being given by a difference between said frame and an averaging of said frame over said pixels. ### Claim 7 (depends on 1) The method according to claim 1, wherein said two reference frames are consecutive in said reference video stream. ### Claim 8 (depends on 1) The method according to claim 1, wherein said current video stream is obtained in real time, determining said time alignment being performed in real time. ### Claim 9 (depends on 1) The method according to claim 1, wherein said current video stream corresponds to an adapted version of said reference video stream. ### Claim 10 (depends on 1) A computer program product comprising computer executable program code recorded on a computer readable non-transitory storage medium, the computer executable program code when executed, performing the method according to claim 1. ### Claim 11 A device for time aligning a current frame of a current video stream with respect to reference frames of a reference video stream, said device comprising a processor configured for: obtaining two reference frames corresponding to frames of the reference video stream displayed on a display; obtaining the current frame from a camera by shooting said display; time aligning said current frame as a function of a correlation between said current frame and each of said at least two reference frames, and of a correlation between the two reference frames. ### Claim 12 (depends on 11) The device according to claim 11, wherein said processor is configured for determining a pose estimation associated with said current frame, from spatially correlating said current frame and said at least two reference frames based on said time aligning. ### Claim 13 (depends on 11) The device according to claim 11, wherein said time aligning is an interpolation based on the correlations S1 and S2 between said current frame and each of the two reference frames, and the correlation S3 between the two frames, said interpolation corresponding to a linear normalized time positioning weight α between the two reference frames, said time positioning weight α being given by: α = - - S 2  S 3 + S 1 S 1  S 3 + S 2  S 3 - S 1 - S 2 . ### Claim 14 (depends on 13) The device according to claim 13, wherein the time aligning of said current frame is possible with respect to at least two sets of a predetermined number of at least two reference frames, said method comprising determining said time aligning with respect to one of said sets of said at least two reference frames that provides a highest correlation of said current frame with the interpolation of said reference frames, among said at least two sets of said at least two reference frames. ### Claim 15 (depends on 11) The device according to claim 11, wherein said time aligning comprises selecting said two reference frames within a time window including a current time corresponding to said current frame and having a predetermined time length. ### Claim 16 (depends on 11) The device according to claim 11, wherein said correlations are respectively applied to centered versions of said current frame and said two reference frames, a centered version of a frame comprising pixels being given by a difference between said frame and an averaging of said frame over said pixels. ### Claim 17 (depends on 11) The device according to claim 11, wherein said two reference frames are consecutive in said reference video stream. ### Claim 18 (depends on 11) The device according to claim 11, wherein said current video stream is obtained in real time, determining said time alignment being performed in real time. ### Claim 19 (depends on 11) The device according to claim 11, wherein said current video stream corresponds to an adapted version of said reference video stream. ### Claim 20 (depends on 11) A mobile apparatus, comprising a device according to claim 11.