# US 12,006,612 B2 · Automated lens cleaning for a camera assembly based on image recognition in a washing machine appliance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,006,612 B2 - Document ID: US12006612B2 - Status: Granted - Kind: B2 - Application number: 17/330,802 - Filed: 2021-05-26 - Published: 2022-12-01 - Granted: 2024-06-11 - Earliest claimed: 2021-05-26 - Assignees: Haier US Appliance Solutions, Inc. - Applicants: Haier US Appliance Solutions, Inc. - Inventors: Je Kwon Yoon; Hoyoung Lee; Khalid Jamal Mashal; JaeHyo Lee; Minsu Kim - Web page: https://patentgrep.com/patent/US12006612B2 ## CPC - D06F 33/47: Responding to irregular working conditions, e.g. malfunctioning of pumps (safety arrangements D06F37/42) - D06F 23/025: {with a rotatable imperforate tub} - D06F 23/04: and rotating or oscillating about a vertical axis - D06F 34/14: Arrangements for detecting or measuring specific parameters - D06F 39/02: Devices for adding soap or other washing agents - D06F 39/06: Arrangements for preventing or destroying scum - D06F 39/088: {Liquid supply arrangements} - D06F 39/14: Doors or covers; Securing means therefor - G02B 27/0006: {with means to keep optical surfaces clean, e.g. by preventing or removing dirt, stains, contamination, condensation (G02B1/18 takes precedence; cleaning in general B08B)} - G06T 7/00: Image analysis - G06T 7/0002: {Inspection of images, e.g. flaw detection} - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - D06F 2103/64: Radiation, e.g. microwaves - D06F 2105/02: Water supply - G06T 2207/10016: Video; Image sequence - G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30168: Image quality inspection - G06T 2207/30204: Marker - D06F: LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A washing machine appliance includes a camera assembly mounted in view of a wash chamber for obtaining images of the wash chamber. The lens of the camera assembly is positioned adjacent a water outlet of a water dispenser and a controller is configured to facilitate an automated cleaning method for the camera assembly, the method including obtaining one or more images of the wash chamber using the camera assembly, analyzing the one or more images to determine an image quality, determining that the image quality has dropped below a quality threshold, and operating the water dispenser to perform a lens cleaning operation in response to determining that the image quality has dropped below the quality threshold. ## Claims ### Claim 1 A washing machine appliance comprising: a wash tub positioned within a cabinet; a wash basket rotatably mounted within the wash tub and defining a wash chamber configured for receiving a load of clothes; a water dispenser mounted within the cabinet for selectively providing a flow of water into the wash tub through a water outlet; a camera assembly mounted within the cabinet in view of the wash chamber, the camera assembly being positioned such that a lens of the camera assembly is positioned adjacent the water outlet; and a controller operably coupled to the camera assembly and the water dispenser, the controller being configured to: obtain one or more images of the wash chamber using the camera assembly; analyze the one or more images using an image processing algorithm or a machine learning image recognition process to determine an image quality; determine that the image quality has dropped below a quality threshold by comparing the image quality of the one or more images to the quality threshold, the quality threshold being predetermined and stored in the controller; and operate the water dispenser to perform a lens cleaning operation in response to determining that the image quality has dropped below the quality threshold. ### Claim 2 (depends on 1) The washing machine appliance of claim 1, wherein the image processing algorithm comprises using image differentiation or blur detection algorithms to compare the one or more images to a reference image. ### Claim 3 (depends on 1) The washing machine appliance of claim 1, wherein analyzing the machine learning image recognition process comprises at least one of a convolution neural network (“CNN”), a region-based convolution neural network (“R-CNN”), a deep belief network (“DBN”), or a deep neural network (“DNN”) image recognition process. ### Claim 4 (depends on 1) The washing machine appliance of claim 1, wherein the wash basket comprises one or more reference marks, and wherein analyzing the one or more images comprises: identifying the image quality based at least in part on the one or more reference marks. ### Claim 5 (depends on 1) The washing machine appliance of claim 1, wherein operating the water dispenser to perform a lens cleaning operation comprises: opening a water supply valve to provide the flow of water for a predetermined cleaning time. ### Claim 6 (depends on 5) The washing machine appliance of claim 5, wherein detergent is added to the flow of water during the predetermined cleaning time. ### Claim 7 (depends on 5) The washing machine appliance of claim 5, wherein the predetermined cleaning time is between about 0.1 second and 10 minutes. ### Claim 8 (depends on 1) The washing machine appliance of claim 1, wherein the controller is further configured to: wait a predetermined amount of time after performing the lens cleaning operation; obtain one or more verification images of the wash chamber using the camera assembly; analyze the one or more verification images to determine the image quality; and determine that the image quality exceeds the quality threshold. ### Claim 9 (depends on 1) The washing machine appliance of claim 1, wherein the washing machine appliance is a top load washing machine appliance, the water outlet is positioned above the wash tub proximate a rear of the wash tub, and a lens of the camera assembly is positioned below the water outlet and over the wash tub. ### Claim 10 (depends on 9) The washing machine appliance of claim 9, wherein the lens of the camera assembly protrudes out into the wash tub and toward an axis of rotation of the wash basket further than the water outlet. ### Claim 11 (depends on 1) The washing machine appliance of claim 1, wherein the washing machine appliance is a front load washing machine appliance, the water outlet is positioned above a door of the front load washing machine appliance, and the camera assembly is mounted to the door. ### Claim 12 (depends on 1) The washing machine appliance of claim 1, wherein the controller is further configured to: provide a user with instructions to clean a lens of the camera assembly. ### Claim 13 (depends on 1) The washing machine appliance of claim 1, further comprising: a tub light for illuminating the wash chamber, wherein the controller is further configured to turn on the tub light prior to obtaining the one or more images of the wash chamber. ### Claim 14 A method of operating a washing machine appliance, the washing machine appliance comprising a wash basket rotatably mounted within a wash tub and defining a wash chamber configured for receiving a load of clothes, a water dispenser for selectively providing a flow of water into the wash tub through a water outlet, and a camera assembly mounted adjacent the water dispenser and in view of the wash chamber, the method comprising: obtaining one or more images of the wash chamber using the camera assembly; analyzing the one or more images using an image processing algorithm or a machine learning image recognition process to determine an image quality; determining that the image quality has dropped below a quality threshold by comparing the image quality of the one or more images to the quality threshold, the quality threshold being predetermined and stored in the controller; and operating the water dispenser to perform a lens cleaning operation in response to determining that the image quality has dropped below the quality threshold. ### Claim 15 (depends on 14) The method of claim 14, wherein the image processing algorithm comprises using image differentiation or blur detection algorithms to compare the one or more images to a reference image. ### Claim 16 (depends on 14) The method of claim 14, wherein analyzing the machine learning image recognition process comprises at least one of a convolution neural network (“CNN”), a region-based convolution neural network (“R-CNN”), a deep belief network (“DBN”), or a deep neural network (“DNN”) image recognition process. ### Claim 17 (depends on 14) The method of claim 14, wherein operating the water dispenser to perform a lens cleaning operation comprises: opening a water supply valve to provide the flow of water for a predetermined cleaning time. ### Claim 18 (depends on 14) The method of claim 14, further comprising: waiting a predetermined amount of time after performing the lens cleaning operation; obtaining one or more verification images of the wash chamber using the camera assembly; analyzing the one or more verification images to determine the image quality; and determining that the image quality exceeds the quality threshold.