# US 12,180,636 B2 · Leftover load detection method using image recognition in a laundry appliance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,180,636 B2 - Document ID: US12180636B2 - Status: Granted - Kind: B2 - Application number: 17/220,180 - Filed: 2021-04-01 - Published: 2022-10-06 - Granted: 2024-12-31 - Earliest claimed: 2021-04-01 - Assignees: Haier US Appliance Solutions, Inc. - Applicants: Haier US Appliance Solutions, Inc. - Inventors: Kyoung-june Yi; Je Kwon Yoon; Seung-Yeong Park - Web page: https://patentgrep.com/patent/US12180636B2 ## CPC - D06F 34/14: Arrangements for detecting or measuring specific parameters - D06F 34/18: Condition of the laundry, e.g. nature or weight - G06T 7/11: Region-based segmentation - D06F 34/05: for wireless communication between components, e.g. for remote monitoring or control - D06F 37/266: {Gaskets mounted between tub and casing around the loading opening} - D06F 2103/04: Quantity, e.g. weight or variation of weight - D06F 2103/38: Time, e.g. duration - D06F 2103/40: Opening or locking status of doors - D06F 2103/64: Radiation, e.g. microwaves - D06F 2105/58: Indications or alarms to the control system or to the user - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - D06F: LAUNDERING, DRYING, IRONING, PRESSING OR FOLDING TEXTILE ARTICLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A laundry appliance includes a basket rotatably mounted within a cabinet and defining a chamber configured for receiving a load of clothes and a camera assembly mounted within the cabinet in view of the chamber. A controller is configured to implement a leftover load detection method including determining that an operating cycle of the laundry appliance has been completed and that a user has removed the load of clothes. A camera then obtains one or more images of the chamber and the images are analyzed to determine whether any articles of clothing remain in the chamber. If the image analysis identifies a leftover article remaining in the chamber, the method includes implementing a responsive action, such as providing a user notification. ## Claims ### Claim 1 A laundry appliance comprising: a cabinet; a basket rotatably mounted within the cabinet and defining a chamber configured for receiving a load of clothes; a door rotatably mounted to the cabinet for providing selective access to the chamber; a camera assembly mounted within the cabinet and in view of the chamber; and a controller operably coupled to the camera assembly, the controller being configured to: determine that an operating cycle of the laundry appliance has been completed by identifying at least one of the following: determining that a last function or sub cycle of a washing machine appliance has been performed for a particular load of clothes; or determining that a door is unlocked after the operating cycle; determine that a user has removed the load of clothes by identifying at least one of the following: determining that the door has been opened and determining that the door has been closed; determining that the door has been opened and determining that no other action has been taken within a predetermined time; wherein the laundry appliance is a washing machine appliance, and determining that a user has initiated a cycle selection for an associated dryer appliance; or determining that one or more buttons on the laundry appliance have been depressed to begin initiation of another operating cycle; obtain one or more images of the chamber using the camera assembly; identify a leftover article remaining in the chamber by analyzing the one or more images of the chamber; and implement a responsive action in response to identifying the leftover article remaining in the chamber. ### Claim 2 (depends on 1) The laundry appliance of claim 1, wherein the predetermined time is three minutes. ### Claim 3 (depends on 1) The laundry appliance of claim 1, wherein analyzing the one or more images of the chamber comprises: implementing an image processing algorithm on the one or more images. ### Claim 4 (depends on 3) The laundry appliance of claim 3, wherein the image processing algorithm utilizes image differentiation. ### Claim 5 (depends on 1) The laundry appliance of claim 1, wherein analyzing the one or more images of the chamber comprises: analyzing the one or more images using a machine learning image recognition process, the machine learning image recognition process being programmed into the controller. ### Claim 6 (depends on 5) The laundry appliance of claim 5, wherein the machine learning image recognition process comprises at least one of a convolution neural network (“CNN”), a region-based convolution neural network (“R-CNN”), a deep belief network (“DBN”), or a deep neural network (“DNN”) image recognition process. ### Claim 7 (depends on 1) The laundry appliance of claim 1, wherein implementing the responsive action comprises: providing a user notification that the leftover article remains in the chamber. ### Claim 8 (depends on 7) The laundry appliance of claim 7, further comprising: a user interface panel, wherein the user notification is provided through the user interface panel. ### Claim 9 (depends on 7) The laundry appliance of claim 7, wherein the controller is in operative communication with a remote device through an external network, and wherein the user notification is provided through the remote device. ### Claim 10 (depends on 9) The laundry appliance of claim 9, wherein the user notification includes displaying the one or more images on the remote device. ### Claim 11 (depends on 1) The laundry appliance of claim 1, further comprising: a gasket positioned between wherein the door and the cabinet, wherein the camera assembly is mounted in the gasket or on an inner surface of the door. ### Claim 12 (depends on 1) The laundry appliance of claim 1, wherein the laundry appliance is a washing machine appliance, a dryer appliance, or a combination washer/dryer appliance. ### Claim 13 A method of operating a laundry appliance, the laundry appliance comprising a basket rotatably mounted within a cabinet and defining a chamber configured for receiving a load of clothes, a door rotatably mounted to the cabinet for providing selective access to the chamber, and a camera assembly mounted within the cabinet in view of the chamber, the method comprising: determining that an operating cycle of the laundry appliance has been completed by identifying at least one of the following: determining that a last function or sub cycle of a washing machine appliance has been performed for a particular load of clothes; or determining that a door is unlocked after the operating cycle; determining that a user has removed the load of clothes by identifying at least one of the following: determining that the door has been opened and determining that the door has been closed; determining that the door has been opened and determining that no other action has been taken within a predetermined time; wherein the laundry appliance is a washing machine appliance, and determining that a user has initiated a cycle selection for an associated dryer appliance; or determining that one or more buttons on the laundry appliance have been depressed to begin initiation of another operating cycle; obtaining one or more images of the chamber using the camera assembly; identifying a leftover article remaining in the chamber by analyzing the one or more images of the chamber; and implementing a responsive action in response to identifying the leftover article remaining in the chamber. ### Claim 14 (depends on 13) The method of claim 13, wherein analyzing the one or more images of the chamber comprises: implementing an image processing algorithm on the one or more images. ### Claim 15 (depends on 13) The method of claim 13, wherein analyzing the one or more images of the chamber comprises: analyzing the one or more images using a machine learning image recognition process.