# US 2022/0036604 A1 · Generation of images with tooth color determined using depth information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036604 A1 - Document ID: US20220036604A1 - Status: Published application - Kind: A1 - Application number: 17/503,175 - Filed: 2021-10-15 - Published: 2022-02-03 - Earliest claimed: 2018-09-28 - Assignees: Align Technology, Inc. - Applicants: Align Technology, Inc. - Inventors: Yun Gao; Yingjie Li; Chao Shi - Web page: https://patentgrep.com/patent/US20220036604A1 ## CPC - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations - A61C 9/004: {Means or methods for taking digitized impressions} - A61C 9/0053: {Optical means or methods, e.g. scanning the teeth by a laser or light beam} - A61C 13/0004: {Computer-assisted sizing or machining of dental prostheses} - A61C 13/082: {Cosmetic aspects, e.g. inlays; Determination of the colour (A61C19/10 takes precedence)} - A61C 13/34: Making or working of models, e.g. preliminary castings, trial dentures; Dowel pins [4] - G01J 3/508: {measuring the colour of teeth} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/70: Denoising; Smoothing - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones} - A61C 2201/002: using colour effect, e.g. for identification purposes - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30036: Dental; Teeth - G06T 2210/41: Medical - G06T 2219/2004: Aligning objects, relative positioning of parts - G06T 2219/2016: Rotation, translation, scaling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61C: DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE (non-driven toothbrushes A46B; {tongue scrapers A61B17/24;} preparations for dentistry A61K6/00; preparations for cleaning the teeth or mouth A61K8/00, A61Q11/00) - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) ## Abstract A method includes determining depth values associated with a first set of pixel locations in a first image of a mouth. One or more function is generated for one or more color channels based on intensities of the one or more color channels at the first set of pixel locations and depth values associated with the first set of pixel locations. Image data comprising a new representation of the teeth is received, wherein the image data comprises a second set of pixel locations and new depth values associated with the second set of pixel locations. A new image is generated based on the image data and the one or more functions. ## Claims ### Claim 1 A method comprising: determining, from a first image of a mouth, a first region of the first image comprising a representation of teeth, wherein the first region comprises a first set of pixel locations in the first image; determining depth values associated with one or more of pixel locations in the first set of pixel locations; generating one or more functions for one or more color channels based at least in part on depth values associated with the one or more pixel locations in the first set of pixel locations; receiving image data comprising a new representation of the teeth in a second region, wherein one or more of the teeth have a different position in the image data than in the first image, wherein the second region comprises a second set of pixel locations for the teeth that is different than the first set of pixel locations, and wherein the image data further comprises new depth values associated with pixel locations in the second set of pixel locations; and generating a new image based on the image data and the one or more functions. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more functions for the one or more color channels are generated based on a) intensities of the one or more color channels at the one or more pixel locations in the first set of pixel locations and b) depth values associated with the one or more pixel locations in the first set of pixel locations. ### Claim 3 (depends on 1) The method of claim 1, wherein each of the one or more functions comprise a first variable for a first image axis, a second variable for a second image axis, and a third variable for a third image axis. ### Claim 4 (depends on 1) The method of claim 1, wherein a shape of the teeth is based on the image data and a color of the teeth is based on applying the one or more functions to the second set of pixel locations for the teeth and the new depth values associated with the second set of pixel locations for the teeth. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more functions comprise a first function for a first color channel and a second function for a second color channel. ### Claim 6 (depends on 5) The method of claim 5, wherein the one or more functions further comprise a third function for a third color channel. ### Claim 7 (depends on 1) The method of claim 1, wherein each of the one or more functions comprise a first sub-function f(x,y) for x,y pixel locations and a second sub-function g(z) for depth values. ### Claim 8 (depends on 1) The method of claim 1, further comprising: generating a blurred color representation of the teeth by applying the one or more functions to the second set of pixel locations for the teeth and the new depth values associated with the second set of pixel locations for the teeth; and processing the blurred color representation of the teeth and the received image data by a neural network to generate the new image. ### Claim 9 (depends on 1) The method of claim 1, further comprising: generating a first mask for the first image, wherein the first mask identifies the first set of pixel locations of the first region that are associated with the teeth; generating a second mask for the image data; and determining the second region comprising the teeth in the image data using the second mask, wherein the second mask identifies the second set of pixel locations of the second region that are associated with the teeth. ### Claim 10 A system comprising: a memory device; and a processing device operatively coupled to the memory device, the processing device to: determine, from a first image of a mouth, a first region of the first image comprising a representation of teeth, wherein the first region comprises a first set of pixel locations in the first image; determine depth values associated with one or more of pixel locations in the first set of pixel locations; generate one or more functions for one or more color channels based at least in part on depth values associated with the one or more pixel locations in the first set of pixel locations; receive image data comprising a new representation of the teeth in a second region, wherein one or more of the teeth have a different position in the image data than in the first image, wherein the second region comprises a second set of pixel locations for the teeth that is different than the first set of pixel locations, and wherein the image data further comprises new depth values associated with pixel locations in the second set of pixel locations; and generate a new image based on the image data and the one or more functions. ### Claim 11 (depends on 10) The system of claim 10, wherein the one or more functions for the one or more color channels are generated based on a) intensities of the one or more color channels at the one or more pixel locations in the first set of pixel locations and b) depth values associated with the one or more pixel locations in the first set of pixel locations. ### Claim 12 (depends on 10) The system of claim 10, wherein the one or more functions comprise a first variable for a first image axis, a second variable for a second image axis, and a third variable for a third image axis. ### Claim 13 (depends on 10) The system of claim 10, wherein a shape of the teeth is based on the image data and a color of the teeth is based on applying the one or more functions to the second set of pixel locations for the teeth and the new depth values associated with the second set of pixel locations for the teeth. ### Claim 14 (depends on 10) The system of claim 10, wherein the one or more functions comprise a first function for a first color channel and a second function for a second color channel. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more functions further comprise a third function for a third color channel. ### Claim 16 (depends on 10) The system of claim 10, wherein each of the one or more functions comprise a first sub-function f(x,y) for x,y pixel locations and a second sub-function g(z) for depth values. ### Claim 17 (depends on 10) The system of claim 10, wherein the processing device is further to: generate a blurred color representation of the teeth by applying the one or more functions to the second set of pixel locations for the teeth and the new depth values associated with the second set of pixel locations for the teeth; and process the blurred color representation of the teeth and the received image data by a neural network to generate the new image. ### Claim 18 (depends on 10) The system of claim 10, wherein the processing device is further to: generate a first mask for the first image, wherein the first mask identifies the first set of pixel locations of the first region that are associated with the teeth; generate a second mask for the image data; and determine the second region comprising the teeth in the image data using the second mask, wherein the second mask identifies the second set of pixel locations of the second region that are associated with the teeth. ### Claim 19 A computer readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: determining, from a first image of a mouth, a first region of the first image comprising a representation of teeth, wherein the first region comprises a first set of pixel locations in the first image; determining depth values associated with one or more of pixel locations in the first set of pixel locations; generating a plurality of functions, wherein the plurality of functions comprises a function for each color channel of a plurality of color channels, the function being based at least in part on depth values associated with the one or more pixel locations in the first set of pixel locations; receiving image data comprising a new representation of the teeth in a second region, wherein one or more of the teeth have a different position in the image data than in the first image, wherein the second region comprises a second set of pixel locations for the teeth that is different than the first set of pixel locations, and wherein the image data further comprises new depth values associated with pixel locations in the second set of pixel locations; and generating a new image based on the image data and the plurality of functions. ### Claim 20 (depends on 19) The computer readable storage medium of claim 19, wherein each of the plurality of functions are generated based on a) intensities of a respective color channel at the one or more pixel locations in the first set of pixel locations and b) depth values associated with the one or more pixel locations in the first set of pixel locations.