# US 10,372,970 B2 · Automatic scene calibration method for video analytics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,970 B2 - Document ID: US10372970B2 - Status: Granted - Kind: B2 - Application number: 15/266,747 - Filed: 2016-09-15 - Published: 2018-03-15 - Granted: 2019-08-06 - Earliest claimed: 2016-09-15 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Lei Wang; Dashan Gao; Lei Ma; Chinchuan Chiu - Web page: https://patentgrep.com/patent/US10372970B2 ## CPC - G06V 40/165: {using facial parts and geometric relationships} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/85: {Stereo camera calibration} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - H04N 13/261: with monoscopic-to-stereoscopic image conversion - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T 2207/30232: Surveillance - G06T 2207/30244: Camera pose - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract To determine real-world information about objects moving in a scene, the camera capturing the scene is typically calibrated to the scene. Automatic scene calibration can be accomplished using people that are found moving about in the scene. During a calibration period, a video content analysis system processing video frames from a camera can identify blobs that are associated with people. Using an estimated height of a typical person, the video content analysis system can use the location of the person's head and feet to determine a mapping between the person's location in the 2-D video frame and the person's location in the 3-D real world. This mapping can be used to determine a cost for estimated extrinsic parameters for the camera. Using a hierarchical global estimation mechanism, the video content analysis system can determine the estimated extrinsic parameters with the lowest cost. ## Claims ### Claim 1 A method for automated scene calibration, comprising: determining a blob from a current video frame; identifying the blob as associated with an object, the blob including pixels that represent at least a portion of the object; determining, using the blob, a ground plane for the current video frame, wherein the ground plane represents a surface upon which the object is positioned; selecting approximate three-dimensional points on the ground plane; estimating extrinsic parameters for a camera model; determining, using the camera model and the estimated extrinsic parameters, two-dimensional coordinates within the current video frame that correspond to the approximate three-dimensional points; and determining, using the two-dimensional coordinates and the ground plane, values for a homographic matrix, wherein a homographic transformation using the values for the homographic matrix provides a mapping from the two-dimensional coordinates in the current video frame to three-dimensional real-world points. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the two-dimensional coordinates includes using virtual intrinsic parameters, wherein the virtual intrinsic parameters include at least a focal length and an optical center. ### Claim 3 (depends on 1) The method of claim 1, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least an upper body of the person, and wherein determining the ground plane includes using an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 4 (depends on 1) The method of claim 1, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least a face of the person, and wherein determining the ground plane includes using an estimated distance between eyes of the person and an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 5 (depends on 1) The method of claim 1, further comprising: using random sample consensus to modify the estimated extrinsic parameters. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining, using a cost function, a cost value for the estimated extrinsic parameters, wherein determining the cost value includes: determining an estimated height of an object in the current video frame using the estimated extrinsic parameters; determining a detected height of the object using coordinates of the object within the current video frame; and comparing the estimated height and the detected height using the cost function. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the estimated height includes: determining, using the homographic matrix, a three-dimensional point for two-dimensional coordinates of a bottom the object, wherein the two-dimensional coordinates are within the current video frame; and determining two-dimensional coordinates of a top of the object using the camera model and an estimated real-world height of the object. ### Claim 8 (depends on 6) The method of claim 6, further comprising: determining a plurality cost values for a plurality of extrinsic parameters, the plurality of cost values including the cost value; and identifying from the plurality of cost values a set of extrinsic parameters with a lowest cost value. ### Claim 9 (depends on 6) The method of claim 6, wherein the cost function is a size-pose-based cost function. ### Claim 10 An apparatus, comprising: a memory configured to store video data; and a processor configured to: determine a blob from a current video frame; identify the blob as associated with an object, the blob including pixels that represent at least a portion of the object; determine, using the blob, a ground plane for the current video frame, wherein the ground plane represents a surface upon which the object is positioned; select approximate three-dimensional points on the ground plane; estimate extrinsic parameters for a camera model; determine, using the camera model and the estimated extrinsic parameters, two-dimensional coordinates within the current video frame that correspond to the approximate three-dimensional points; and determine, using the two-dimensional coordinates and the ground plane, values for a homographic matrix, wherein a homographic transformation using the values for the homographic matrix provides a mapping from the two-dimensional coordinates in the current video frame to three-dimensional real-world points. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the camera model provides a mapping from three-dimensional real-world points to two-dimensional coordinates in the current video frame. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein homographic transformation provides a mapping from one coordinate system to another coordinate system. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein extrinsic parameters include at least three rotational parameters and two translational parameters. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the processor is configured to determine the two-dimensional coordinates using virtual intrinsic parameters, wherein the virtual intrinsic parameters include at least a focal length and an optical center. ### Claim 15 (depends on 10) The apparatus of claim 10, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least an upper body of the person, and wherein the processor is configured to determine the ground plane using an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 16 (depends on 10) The apparatus of claim 10, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least a face of the person, and wherein the processor is configured to determine the ground plane using an estimated distance between eyes of the person and an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 17 (depends on 10) The apparatus of claim 10, wherein the processor is further configured to: use random sample consensus to modify the estimated extrinsic parameters. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein the processor is further configured to: determine, using a cost function, a cost value for the estimated extrinsic parameters, wherein determining the cost value includes: determining an estimated height of an object in the current video frame using the estimated extrinsic parameters; determining a detected height of the object using coordinates of the object within the current video frame; and comparing the estimated height and the detected height using the cost function. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the processor is configured to determine the estimated height by: determining, using the homographic matrix, a three-dimensional point for two-dimensional coordinates of a bottom the object, wherein the two-dimensional coordinates are within the current video frame; and determining two-dimensional coordinates of a top of the object using the camera model and an estimated real-world height of the object. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein the processor is further configured to: determine a plurality cost values for a plurality of extrinsic parameters, the plurality of cost values including the cost value; and identify from the plurality of cost values a set of extrinsic parameters with a lowest cost value. ### Claim 21 (depends on 18) The apparatus of claim 18, wherein the cost function is a size-pose-based cost function. ### Claim 22 (depends on 10) The apparatus of claim 10, wherein the processor is further configured to: using the estimated extrinsic parameters for tracking objects in a video. ### Claim 23 A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determine a blob from a current video frame; identify the blob as associated with an object, the blob including pixels that represent at least a portion of the object; determine, using the blob, a ground plane for the current video frame, wherein the ground plane represents a surface upon which the object is positioned; select approximate three-dimensional points on the ground plane; estimate extrinsic parameters for a camera model; determine, using the camera model and the estimated extrinsic parameters, two-dimensional coordinates within the current video frame that correspond to the approximate three-dimensional points s; and determine, using the two-dimensional coordinates and the ground plane, values for a homographic matrix, wherein a homographic transformation using the values for the homographic matrix provides a mapping from the two-dimensional coordinates in the current video frame to three-dimensional real-world points. ### Claim 24 (depends on 23) The non-transitory computer-readable medium of claim 23, wherein determining the two-dimensional coordinates includes using virtual intrinsic parameters, wherein the virtual intrinsic parameters include at least a focal length and an optical center. ### Claim 25 (depends on 23) The non-transitory computer-readable medium of claim 23, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least an upper body of the person, and wherein determining the ground plane includes using an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 26 (depends on 23) The non-transitory computer-readable medium of claim 23, wherein at least the portion of the object is less than a whole of the object, wherein the object includes a person, wherein the pixels included in the blob include at least a face of the person, and wherein determining the ground plane includes using an estimated distance between eyes of the person and an estimated height of the person to locate an approximate position of one or both feet of the person. ### Claim 27 (depends on 23) The non-transitory computer-readable medium of claim 23, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determine, using a cost function, a cost value for the estimated extrinsic parameters, wherein determining the cost value includes: determine an estimated height of an object in the current video frame using the estimated extrinsic parameters; determine a detected height of the object using coordinates of the object within the current video frame; and compare the estimated height and the detected height using the cost function. ### Claim 28 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein determining the estimated height includes: determining, using the homographic matrix, a three-dimensional point for two-dimensional coordinates of a bottom the object, wherein the two-dimensional coordinates are within the current video frame; and determining two-dimensional coordinates of a top of the object using the camera model and an estimated real-world height of the object. ### Claim 29 (depends on 1) The method of claim 1, wherein the extrinsic parameters include one or more of a translation and a rotation of a camera. ### Claim 30 (depends on 1) The method of claim 1, wherein the homographic matrix is used for testing accuracies of one or more sets of extrinsic parameters.