# US 10,311,409 B1 · Detection and interpretation of visual indicators > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,409 B1 - Document ID: US10311409B1 - Status: Granted - Kind: B1 - Application number: 14/751,508 - Filed: 2015-06-26 - Granted: 2019-06-04 - Earliest claimed: 2015-06-26 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Ilya Vladimirovich Brailovskiy; Paul Berenberg - Web page: https://patentgrep.com/patent/US10311409B1 ## CPC - G06Q 10/20: Administration of product repair or maintenance - G06F 16/683: using metadata automatically derived from the content - G06Q 10/08: Logistics, e.g. warehousing, loading or distribution; Inventory or stock management - G06Q 30/0631: {Recommending goods or services} - G06T 7/90: Determination of colour characteristics - G06V 10/56: relating to colour - G06V 20/20: in augmented reality scenes - G06T 2207/10024: Color image - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Devices, systems and methods are disclosed that detect and interpret the meaning of visual and/or audio indicators of devices, and provide information, solutions to potential malfunctions, and/or automatic ordering of consumable products to users. For example, sequences of visual indicators (such as LEDs) and/or audio indicators (such as speakers that output beeps) generated by home electronics and other devices may be monitored using an image capture component (such as a camera) or audio capture component (such as a microphone), and the sequences may be interpreted to provide users with information of what is wrong with a device, how to fix a device, purchase a new device, offer repair services, etc. The systems and methods may also provide users with notifications relating to such indicators when the user is away from home. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving a device identifier corresponding to an image capture device associated with a user profile; receiving, from the image capture device, image data representing an object including an illuminated indicator; determining an object identifier corresponding to the object; determining, based in part on the object identifier, stored data indicating a location of a known indicator with respect to the object and information associated with the known indicator; determining, using the image data and the stored data, a meaning of the illuminated indicator based at least in part on a luminescence of a pixel within a bounded region of the image data, the bounded region corresponding to the location; determining an action associated with the meaning; and performing the action. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication to replace a consumable product associated with the object identifier; identifying a user address associated with the user profile; and ordering the consumable product to be shipped to the user address. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication to replace a consumable product associated with the object identifier; identifying a user device associated with the user profile; and sending options of consumable products to the user device for purchase. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending instruction corresponding to resolving the malfunction to the user device. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending options of replacement objects to the user device for purchase. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending contact information for repair services to the user device. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein determining the action and performing the action includes: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user address associated with the user profile; and contacting a repair service to resolve the malfunction. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving, from the image capture device, audio data corresponding to an audible indicator; determining an audio fingerprint of the audio data; comparing the audio fingerprint to a stored audio fingerprint associated with a stored audio sequence associated with the object identifier; and performing an action associated with the stored audio sequence. ### Claim 9 (depends on 1) The method of claim 1, wherein the device identifier corresponds to a camera positioned with the object in the camera's field of view. ### Claim 10 A system, comprising: at least one processor; and at least one memory including instructions operable to be executed by the at least one processor to perform a set of actions, configuring the at least one processor to: receive a device identifier corresponding to an image capture device associated with a user profile; receive, from the image capture device, image data representing an object including an illuminated indicator; process the image data to identify the object; determine an object identifier corresponding to the object; determine, based in part on the object identifier, stored data indicating a location of a known indicator with respect to the object and information associated with the known indicator; determine, using the image data and the stored data, a meaning of the illuminated indicator in the image data based at least in part on a luminescence of a pixel within a bounded region corresponding to the location; determine an action associated with the meaning; and perform the action. ### Claim 11 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication to replace a consumable product associated with the object identifier; identifying a user address associated with the user profile; and ordering the consumable product to be shipped to the user address. ### Claim 12 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication to replace a consumable product associated with the object identifier; identifying a user device associated with the user profile; and sending options of consumable products to the user device for purchase. ### Claim 13 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending instruction corresponding to resolving the malfunction to the user device. ### Claim 14 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending options of replacement objects to the user device for purchase. ### Claim 15 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user device associated with the user profile; and sending contact information for repair services to the user device. ### Claim 16 (depends on 10) The system of claim 10, wherein the at least one processor is further configured to determine the action and perform the action by: determining that the meaning corresponds to an indication of a malfunction of the object; identifying a user address associated with the user profile; and contacting a local repair service to resolve the malfunction. ### Claim 17 (depends on 10) The system of claim 10, wherein the device identifier corresponds to a camera positioned with the object in the camera's field of view.