# US 11,432,619 B2 · System, method, and apparatus for modelling feet and selecting footwear > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,432,619 B2 - Document ID: US11432619B2 - Status: Granted - Kind: B2 - Application number: 16/486,605 - Filed: 2018-02-16 - Published: 2020-01-02 - Granted: 2022-09-06 - Earliest claimed: 2017-02-18 - Assignees: Digital Animal Interactive Inc. - Applicants: Digital Animal Interactive Inc. - Inventors: Jamie Roy Sherrah; Michael Henson; William Ryan Smith - Web page: https://patentgrep.com/patent/US11432619B2 ## CPC - A43D 1/025: {comprising optical means, e.g. mirrors, photo-electric cells, for measuring or inspecting feet} - A43B 3/34: with electrical or electronic arrangements - A61B 5/107: Measuring physical dimensions, e.g. size of the entire body or parts thereof - G01B 11/002: {for measuring two or more coordinates} - G06T 7/60: Analysis of geometric attributes - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - A43B 13/04: Plastics, rubber or vulcanised fibre - A43D 2200/60: Computer aided manufacture of footwear, e.g. CAD or CAM - A61B 5/1074: {Foot measuring devices} - A61B 5/6807: {Footwear} - A43D: MACHINES, TOOLS, EQUIPMENT OR METHODS FOR MANUFACTURING OR REPAIRING FOOTWEAR - A43B: CHARACTERISTIC FEATURES OF FOOTWEAR; PARTS OF FOOTWEAR - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract One aspect of this disclosure is a method comprising: receiving, from a camera, a video feed depicting a pair of feet and a scaling object; capturing, with the camera, images of the feet and the scaling object based on the video feed; identifying foot features in each captured image; determining camera positions for each captured image by triangulating the foot features; generating a point cloud in a three-dimensional space by positioning each foot feature in the three-dimensional space based on the camera positions; scaling the point cloud based on the scaling object; segmenting the point cloud into at least a right-foot cluster and a left-foot cluster; fitting a first three-dimensional morphable model to the right-foot cluster according to first foot parameters; and fitting a second three-dimensional morphable model to the left-foot cluster according to second foot parameters. Related systems, apparatus, and methods also are described. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, from a camera, a video feed depicting a pair of feet and a scaling object; capturing, with the camera, images of the feet and the scaling object based on the video feed; identifying foot features in each captured image; determining camera positions for each captured image by triangulating the foot features; generating a point cloud in a three-dimensional space by positioning each foot feature in the three-dimensional space based on the camera positions; scaling the point cloud based on the scaling object; segmenting the point cloud into at least a right-foot cluster and a left-foot cluster; fitting a first three-dimensional morphable model to the right-foot cluster according to first foot parameters by a first process comprising: (i) placing the first three-dimensional morphable model over the right-foot cluster; (ii) modifying the first three-dimensional morphable model without deformation to fit the right-foot cluster; and (iii) modifying the first three-dimensional morphable model with deformation by iteratively adjusting the first foot parameters individually or in combination to reduce an error value between the first three-dimensional morphable model and the right-foot cluster; fitting a second three-dimensional morphable model to the left-foot cluster according to second foot parameters by a second process comprising: (i) placing the second three-dimensional morphable model over the left-foot cluster; (ii) modifying the second three-dimensional morphable model without deformation to fit the left-foot cluster; and (iii) modifying the second three-dimensional morphable model with deformation by iteratively adjusting the second foot parameters individually or in combination to reduce an error value between the second three-dimensional morphable model and the left-foot cluster; and outputting data comprising right-foot data and left-foot data. ### Claim 2 (depends on 1) The method of claim 1, further comprising processing the video feed to distinguish foot pixels from non-foot pixels. ### Claim 3 (depends on 2) The method of claim 2, wherein processing the video feed comprises: generating a tracker model based on one of the captured images; and tracking the foot pixels in the video feed based on the tracker model. ### Claim 4 (depends on 3) The method of claim 3, wherein capturing the images comprises adjusting a setting of the camera based on the foot pixels. ### Claim 5 (depends on 1) The method of claim 1, further comprising: matching the first three-dimensional morphable model with a first exemplar foot based on the first foot parameters, and matching the second three-dimensional morphable model with a second exemplar foot based on the second foot parameters. ### Claim 6 (depends on 5) The method of claim 5, further comprising selecting footwear based on at least one of the first exemplar foot and the second exemplar foot without reference to a standard notational size of the footwear. ### Claim 7 (depends on 1) The method of claim 1, wherein receiving the video feed further comprises moving the camera in a sweeping movement around the pair of feet, and capturing images during the sweeping movement. ### Claim 8 (depends on 7) The method of claim 7, wherein moving the camera in the sweeping movement comprises: holding the camera with an outstretched arm; orienting the camera towards the pair of feet and the scaling object; and rotating the camera around the pair of feet and the scaling object with the outstretched arm to maintain a distance between the camera and the pair of feet and the scaling object during the rotation. ### Claim 9 (depends on 8) The method of claim 8, wherein the sweeping movement comprises a plurality of rotational positions, and the method comprises determining whether an image has been captured for each rotational position. ### Claim 10 (depends on 9) The method of claim 9, further comprising receiving position data from a sensor, wherein determining whether the image has been captured for each rotational position is based on the position data. ### Claim 11 (depends on 1) The method of claim 1, further comprising receiving the first three-dimensional morphable model and the second three-dimensional morphable model from a remote data source. ### Claim 12 (depends on 1) The method of claim 1, further comprising segmenting the point cloud into a ground cluster and removing the ground cluster from the point cloud. ### Claim 13 (depends on 12) The method of claim 12, wherein scaling the point cloud comprises: determining a scale from each captured image based on the scaling-object; and determining a point cloud scale by averaging the scale of each captured image. ### Claim 14 (depends on 10) The method of claim 10, further comprising segmenting the point cloud into a ground cluster and removing the ground cluster from the point cloud based on the point cloud scale. ### Claim 15 A computer-implemented system comprising: means for receiving, from a camera, a video feed depicting a pair of feet and a scaling object; means for capturing, with the camera, images of the feet and the scaling object based on the video feed; means for identifying foot features in each captured image; means for determining camera positions for each captured image by triangulating the foot features; means for generating a point cloud in a three-dimensional space by positioning the foot features in the three-dimensional space based on the camera positions; means for scaling the point cloud based on the scaling object; means for segmenting the point cloud into at least a right-foot cluster and a left-foot cluster; means for fitting a first three-dimensional morphable model to the right-foot cluster according to first foot parameters by a first process comprising: (i) placing the first three-dimensional morphable model over the right-foot cluster; (ii) modifying the first three-dimensional morphable model without deformation to fit the right-foot cluster; and (iii) modifying the first three-dimensional morphable model with deformation by iteratively adjusting the first foot parameters individually or in combination to reduce an error value between the first three-dimensional morphable model and the right-foot cluster; means for fitting a second three-dimensional morphable model to the left-foot cluster according to second foot parameters by a second process comprising: (i) placing the second three-dimensional morphable model over the left-foot cluster; (ii) modifying the second three-dimensional morphable model without deformation to fit the left-foot cluster; and (iii) modifying the second three-dimensional morphable model with deformation by iteratively adjusting the second foot parameters individually or in combination to reduce an error value between the second three-dimensional morphable model and the left-foot cluster; and means for outputting data comprising right-foot data and left-foot data.