# US 11,113,568 B2 · Devices, systems, and methods for device-error analysis using computer vision and machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,568 B2 - Document ID: US11113568B2 - Status: Granted - Kind: B2 - Application number: 16/442,184 - Filed: 2019-06-14 - Published: 2020-12-17 - Granted: 2021-09-07 - Earliest claimed: 2019-06-14 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Nigel Brady - Web page: https://patentgrep.com/patent/US11113568B2 ## CPC - H04N 1/00037: {Detecting, i.e. determining the occurrence of a predetermined state (H04N1/00031 takes precedence)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/20: Ensemble learning - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Devices, systems, and methods obtain a video of a device, wherein the device includes one or more light emitters that are visible in the video; input the video to a first machine-learning model and executing the first machine-learning model, wherein the first machine-learning model outputs a time series of light-emitting states that indicate respective light-emitting states of the light emitters at respective times in the time series; and input the time series of light-emitting states to a second machine-learning model and executing the second machine-learning model, wherein the second machine-learning model outputs a status of the device. ## Claims ### Claim 1 An apparatus comprising: one or more computer-readable storage media; and one or more processors that are configured to cause the apparatus to perform operations including: obtaining a video of a device, wherein the device includes one or more light emitters that are visible in the video; inputting the video to a first machine-learning model and executing the first machine-learning model, wherein the first machine-learning model outputs a time series of light-emitting states that indicates respective light-emitting states of the light emitters at respective times in the time series; and inputting the time series of light-emitting states to a second machine-learning model and executing the second machine-learning model, wherein the second machine-learning model outputs a status indicator of the device. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the status indicator indicates an error status, a network-communication status, or a power status. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the device is a printer, a scanner, or a multi-function device. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the video is input to the first machine-learning model as a series of time-indexed frames. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the light-emitting states indicate the respective light-emitting states of the light emitters in each time-indexed frame in the series of time-indexed frames. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the time series of light-emitting states is output into a buffer in the one or more computer-readable media, and wherein inputting the time series of light-emitting states to the second machine-learning model includes inputting contents of the buffer to the second machine-learning model. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the video includes a plurality of frames, and wherein inputting the video to the first machine-learning model and executing the first machine-learning model includes invoking the first machine-learning model on each frame of the plurality of frames, wherein, for each frame of the plurality of frames, the first machine-learning model outputs a respective time-indexed light-emitting state for each of the one or more light emitters, and wherein the time series of light-emitting states includes the respective time-indexed light-emitting state for each of the one or more light emitters for each frame of the plurality of frames. ### Claim 8 A method comprising: obtaining a video of a device, wherein the device includes one or more light emitters that are visible in the video; generating a series of time-indexed frames based on the video; executing a first machine-learning model using the series of time-indexed frames as inputs, wherein the first machine-learning model outputs a time-indexed series of light-emitting states that indicate respective light-emitting states of the one or more light emitters at respective times in the time-indexed series; and executing a second machine-learning model using the time-indexed series of light-emitting states as inputs, wherein the second machine-learning model outputs a status indicator of the device. ### Claim 9 (depends on 8) The method of claim 8, wherein the light-emitting states indicate the respective light-emitting states of the light emitters in the time-indexed frames. ### Claim 10 (depends on 8) The method of claim 8, further comprising: storing the time-indexed series of light-emitting states as contents in a buffer on a computer-readable medium, and inputting the contents of the buffer to the second machine-learning model. ### Claim 11 (depends on 8) The method of claim 8, wherein at least one of the one or more light emitters is configured to blink when the device is in an error state. ### Claim 12 (depends on 8) The method of claim 8, further comprising: identifying a model of the device. ### Claim 13 One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining training data, wherein the training data includes time-indexed frames of at least one video of a device, wherein the device includes one or more light emitters, wherein the one or more light emitters are visible in the time-indexed frames, and wherein the training data indicates a respective light-emitting state of each of the one or more light emitters in each of the time-indexed frames; training a first machine-learning model based on the training data, wherein the first machine-learning model accepts time-indexed frames as inputs and outputs a time-indexed series of light-emitting states that indicates respective light-emitting states of the one or more light emitters at respective times; and training a second machine-learning model based on a plurality of time-indexed series of light-emitting states and on data that maps operating statuses of the device to respective time-indexed series of light-emitting states, wherein the operating statuses include one or more error statuses, and wherein the second machine-learning model accepts a time-indexed series of light-emitting states as an input and outputs an operating status of the device. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein the device is a printer. ### Claim 15 (depends on 14) The one or more non-transitory computer-readable media of claim 14, wherein the one or more error statuses include at least one of the following: paper jam, out of paper, out of toner, and out of ink. ### Claim 16 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the device includes at least two light emitters. ### Claim 17 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein at least one of the first machine-learning model and the second machine-learning model is a neural network. ### Claim 18 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the training data includes time-indexed frames of at least two videos of the device.