# US 2018/0324449 A1 · Method and apparatus for communicating and recovering motion information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0324449 A1 - Document ID: US20180324449A1 - Status: Published application - Kind: A1 - Application number: 16/032,404 - Filed: 2018-07-11 - Published: 2018-11-08 - Earliest claimed: 2011-09-09 - Assignees: NewSouth Innovations Pty Limited - Applicants: Newsouth Innovations Pty Limited - Inventors: David Scott Taubman; Aous Thabit Naman - Web page: https://patentgrep.com/patent/US20180324449A1 ## CPC - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/53: Multi-resolution motion estimation; Hierarchical motion estimation - H04N 19/543: using regions - H04N 19/61: in combination with predictive coding - H04N 19/63: using sub-band based transform, e.g. wavelets - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This invention describes a method for communicating crude motion information using tracking metadata and recovering more accurate motion information from the received tracking metadata and partial video frame data; in particular, we use metadata to convey crude boundaries of objects in the scene and signal motion information for these objects. The proposed method leaves the task of identifying the exact boundaries of an object to the decoder/client. The proposed method is particularly appealing when metadata itself carries semantics that the client is interested in, such as tracking information in surveillance applications, because, in this case, metadata does not constitute an overhead. The proposed method involves motion descriptions that can be used to predict the appearance of an object in any one frame from its appearance in any other frame that contains the object. That is, the motion information itself allows locations within an object to be invertibly mapped to locations within the same object in any other relevant frame. This is a departure from conventional motion coding schemes, which tightly-couple motion information to the prediction strategy. This property makes the proposed method particularly suitable for applications which require flexible access to the content. ## Claims ### Claim 1 A method for recovering motion information within multiple frame media content, using video frame sample data for some frames, together with motion data that carries some information about the motion between frames, comprising the following steps: (a) selection of two or more reference frames to be used in predicting a further frame; (b) using the motion data to identify at least two spatial domains within a first reference frame, to group each of these with a corresponding spatial domain in each other reference frame, and to determine a parametric representation of motion between corresponding domains in different reference frames; (c) using the motion representations, domain correspondences and reference video frame sample values to determine validity information for each overlapping domain within the first reference frame; (d) using the parametric motion representations, domain correspondences and reference video frame sample values to determine validity information for each overlapping domain within each other reference frame; (e) using the parametric motion representations, validity information and reference frame sample values to form a prediction of said further frame. ### Claim 2 (depends on 1) The method of claim 1, comprising a further step of iteratively applying steps (a) to (e) to the prediction of additional frames. ### Claim 3 (depends on 1, 2) The method of claim 1 or claim 2, wherein the motion data is tracking metadata, wherein the spatial domains are tracked regions of interest and where domain correspondences are obtained from the tracking semantics. ### Claim 4 (depends on 3) The method of claim 3, where the parametric motion models are obtained from the geometry of the tracked regions of interest. ### Claim at position 5 5-7. (canceled) ### Claim 8 (depends on 1) The method of claim 1, wherein successive iterations of the steps predict frames in a hierarchical fashion, such that each frame is predicted using two reference frames and the frames that are predicted in one level of the hierarchy may be selected as reference frames when predicting other frames in the next level of the hierarchy. ### Claim at position 7 9-17. (canceled) ### Claim 18 (depends on 1) The method of claim 1, wherein the validity information within some reference frames is obtained by motion compensating the validity information found for a corresponding domain within another reference frame. ### Claim 19 (depends on 1) The method of claim 1, wherein the predicted frame is obtained by motion compensating each reference frame, within each of the overlapping spatial domains, using the associated parametric motion models and forming a weighted combination of the resulting motion compensated sample values, where the weights are based on the validity information for each domain. ### Claim at position 10 20-21. (canceled) ### Claim 22 (depends on 1, 5) The method of claim 1, wherein the video frame sample data is available in the form of images compressed using the JPEG2000 image compression standard, and a portion of the compressed representation and the auxiliary metadata is communicated to a client using the JPIP standard, that portion being used by the method of claim 5 to reconstruct content that has not been communicated. ### Claim 23 A multiresolution method for comparing two images over a spatial domain of interest to determine a set of likelihood ratios for each resolution level, in which each location within the resolution level has its own likelihood ratio that expresses the probability that the spatial features of the two images are matched at said location, divided by the probability that the spatial features of the two images are not matched at said location, comprising the steps of: (a) decomposing each image into a multi-resolution hierarchy; (b) determining a first set of likelihood ratios for each resolution level based on spatial neighbourhoods of the associated locations in each of the two images within said resolution level; (c) determining a second set of likelihood ratios for each resolution level by combining the first set of likelihood ratios with the final set of likelihood ratios determined at a lower resolution level, except at the lowest resolution level, where the first and second sets of likelihood ratios are the same. ### Claim 24 (depends on 23) A method in accordance with claim 23, comprising the further step of: determining a final set of likelihood ratios for each location in each resolution level by applying an edge refinement process to the second set of likelihood ratios. ### Claim at position 14 25-28. (canceled) ### Claim 29 (depends on 23) The method of claim 23, wherein determination of the first set of likelihood ratios for each resolution level involves the steps of: (a) determining structural measure values for each image at each location in said resolution level; (b) determining similarity feature values at each location in said resolution level; (c) forming a structural similarity likelihood ratio at each location in said resolution level based on the two structural measure values at that location, one from each image; (d) forming a conditional similarity likelihood ratio at each resolution in said resolution level from the similarity feature values at that location, conditioned upon the structural measure values at that location; (e) forming the first set of likelihood ratios for the resolution level by multiplying the structural similarity likelihood ratios by the conditional similarity likelihood ratios at each location. ### Claim at position 16 30-38. (canceled) ### Claim 39 (depends on 23) The method of claim 23, wherein the probability ratios are found using lookup tables that are populated through an off line modeling procedure. ### Claim 40 (depends on 24) The method of claim 24, wherein the likelihood ratios are expressed in a logarithmic domain and the first, second and final sets of likelihood ratios for each resolution level correspond to first, second and final sets of log-likelihood ratios. ### Claim 41 (depends on 24) The method of claim 24, wherein the second set of log-likelihood ratios at a given resolution level are obtained by interpolating and multiplying the final set of log-likelihood ratios from a lower resolution level by a set of scaling factors and adding these scaled and interpolated log-likelihood values to those from the first set of log-likelihood ratios at the given resolution level. ### Claim 42 (canceled) ### Claim 43 (depends on 23) The method of claim 23, wherein the likelihoods are derived by applying a transducer function to the likelihood ratios found by applying the method to the first reference frame and a motion compensated version of the second reference frame. ### Claim 44 An apparatus for recovering motion information within multiple frame media content, using video frame sample data for some frames, together with motion data that carries some information about the motion between frames, the apparatus comprising a processing apparatus arranged to implement the following steps: (a) selection of two or more reference frames to be used in predicting a further frame; (b) using the motion data to identify at least two spatial domains within a first reference frame, to group each of these with a corresponding spatial domain in each other reference frame, and to determine a parametric representation of motion between corresponding domains in different reference frames; (c) using the motion representations, domain correspondences and reference video frame sample values to determine validity information for each overlapping domain within the first reference frame; (d) using the parametric motion representations, domain correspondences and reference video frame sample values to determine validity information for each overlapping domain within each other reference frame; (e) using the parametric motion representations, validity information and reference frame sample values to form a prediction of said further frame. ### Claim 45 (depends on 44) An apparatus in accordance with claim 44, the apparatus comprising a video signal decoder. ### Claim 46 (canceled) ### Claim 47 (depends on 1) A non-transient computer readable medium, comprising instructions for controlling a computer to implement a method in accordance with claim 1. ### Claim at position 26 48-49. (canceled) ### Claim 50 An apparatus for comparing two images over a spatial domain of interest to determine a set of likelihood ratios for each resolution level, in which each location within the resolution level has its own likelihood ratio that expresses the probability that the spatial features of the two images are matched at said location, divided by the probability that the spatial features of the two images are not matched at said location, the apparatus comprising a processor configured to implement the steps of: (a) decomposing each image into a multi-resolution hierarchy; (b) determining a first set of likelihood ratios for each resolution level based on spatial neighbourhoods of the associated locations in each of the two images within said resolution level; (c) determining a second set of likelihood ratios for each resolution level by combining the first set of likelihood ratios with the final set of likelihood ratios determined at a lower resolution level, except at the lowest resolution level, where the first and second sets of likelihood ratios are the same. ### Claim 51 (depends on 50) An apparatus in accordance with claim 50, the processor being configured to implement the further step of: determining a final set of likelihood ratios for each location in each resolution level by applying an edge refinement process to the second set of likelihood ratios. ### Claim 52 (depends on 50, 51) An apparatus in accordance with claim 50 or claim 51, comprising a video signal decoder. ### Claim 53 (canceled) ### Claim 54 (depends on 23) A non-transient computer readable medium, comprising instructions for controlling a computer to implement a method in accordance with claim 23. ### Claim at position 32 55-67. (canceled)