# US 12,112,551 B2 · Vehicles, systems and methods for automatically detecting a state of signal lights of a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,551 B2 - Document ID: US12112551B2 - Status: Granted - Kind: B2 - Application number: 17/667,876 - Filed: 2022-02-09 - Published: 2023-08-10 - Granted: 2024-10-08 - Earliest claimed: 2022-02-09 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Naoki Nagasaka; Blake Wulfe; Kuan-Hui Lee; Jia-En Marcus Pan - Web page: https://patentgrep.com/patent/US12112551B2 ## CPC - G06V 20/584: {of vehicle lights or traffic lights} - G06T 7/254: involving subtraction of images - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In one embodiment, a signal light state detection system includes one or more processors, an a non-transitory memory module storing computer-readable instructions. The computer-readable instructions are configured to cause the one or more processors to receive a first image of a vehicle and receiving a second image of the vehicle, wherein the second image is later in time from the first image, and generate a warped image from the first image and the second image, wherein the warped image has individual pixels of one of the first image and the second image that are shifted to locations of corresponding pixels of the other of the first image and the second image. The one or more processors further generate a difference image from the warped image and one of the first image and the second image, and determine, using a classifier module, a probability of a state of vehicle signal lights. ## Claims ### Claim 1 A signal light state detection system comprising: one or more processors; and a non-transitory memory module storing computer-readable instructions configured to cause the one or more processors to: receive a first image of a vehicle and receiving a second image of the vehicle, wherein the second image is later in time from the first image; generate a warped image from one of the first image and the second image, wherein the warped image has individual pixels of one of the first image and the second image that are shifted to locations of corresponding pixels of the other of the first image and the second image such that the individual pixels of the warped image are aligned with the individual pixels of the other of the first image and the second image; generate a difference image from the warped image and one of the first image and the second image; and determine, using a classifier module, a probability of a state of vehicle signal lights. ### Claim 2 (depends on 1) The signal light state detection system of claim 1, wherein generating the warped image comprises shifting individual pixels of the first image to locations of corresponding pixels of the second image. ### Claim 3 (depends on 1) The signal light state detection system of claim 1, wherein generating the warped image comprises: calculating an optical flow from the first image and the second image; and shifting individual pixels of one of the first image and the second image based at least in part on the optical flow. ### Claim 4 (depends on 3) The signal light state detection system of claim 3, wherein the optical flow is calculated by PWCNet. ### Claim 5 (depends on 1) The signal light state detection system of claim 1, wherein the classifier module comprises: a convolutional neural network producing a plurality of spatial features; a long short-term memory module that receives the plurality of spatial features of the convolutional neural network as input and outputs a probability of the state of vehicle signal lights for each spatial feature; and a classifier that receives the probability of the state of vehicle signal lights and selects the state of vehicle signal lights of a spatial feature having a highest probability as a selected state of vehicle signal lights. ### Claim 6 (depends on 5) The signal light state detection system of claim 5, calculating a cross entropy loss for the probabilities of the state of vehicle signal lights for the plurality of spatial features. ### Claim 7 (depends on 1) The signal light state detection system of claim 1, calculating a photometric loss of between the warped image and one of the first image and the second image. ### Claim 8 (depends on 1) The signal light state detection system of claim 1, wherein: the vehicle further comprises a drive system; and the computer-readable instructions further cause the one or more processors to control the drive system according to the probability of the state of vehicle signal lights. ### Claim 9 A vehicle comprising: one or more processors; and a non-transitory memory module storing computer-readable instructions configured to cause the one or more processors to: receive a first image of a road vehicle and receiving a second image of the vehicle, wherein the second image is later in time from the first image; generate a warped image from one of the first image and the second image, wherein the warped image has individual pixels of one of the first image and the second image that are shifted to locations of corresponding pixels of the other of the first image and the second image such that the individual pixels of the warped image are aligned with the individual pixels of the other of the first image and the second image; generate a difference image from the warped image and one of the first image and the second image; and determine, using a classifier module, a probability of a state of vehicle signal lights. ### Claim 10 (depends on 9) The vehicle of claim 9, wherein generating the warped image comprises shifting individual pixels of the first image to locations of corresponding pixels of the second image. ### Claim 11 (depends on 9) The vehicle of claim 9, wherein generating the warped image comprises: calculating an optical flow from the first image and the second image; and shifting individual pixels of one of the first image and the second image based at least in part on the optical flow. ### Claim 12 (depends on 11) The vehicle of claim 11, wherein the optical flow is calculated by PWCNet. ### Claim 13 (depends on 9) The vehicle of claim 9, wherein the classifier module comprises: a convolutional neural network producing a plurality of spatial features; a long short-term memory module that receives the plurality of spatial features of the convolutional neural network as input and outputs a probability of the state of vehicle signal lights for each spatial feature; and a classifier that receives the probability of the state of vehicle signal lights and selects the state of vehicle signal lights of a spatial feature having a highest probability as a selected state of vehicle signal lights. ### Claim 14 (depends on 9) The vehicle of claim 9, wherein: the vehicle further comprises a drive system; and the computer-readable instructions further cause the one or more processors to control the drive system according to the probability of the state of vehicle signal lights. ### Claim 15 A method of detecting a state of vehicle signal lights, the method comprising: receiving a first image of a vehicle and receiving a second image of the vehicle, wherein the second image is later in time from the first image; generating a warped image from one of the first image and the second image, wherein the warped image has individual pixels of one of the first image and the second image that are shifted to locations of corresponding pixels of the other of the first image and the second image such that the individual pixels of the warped image are aligned with the individual pixels of the other of the first image and the second image; generating a difference image from the warped image and one of the first image and the second image; and determining, using a classifier module, a probability of a state of vehicle signal lights. ### Claim 16 (depends on 15) The method of claim 15, wherein generating the warped image comprises shifting individual pixels of the first image to locations of corresponding pixels of the second image. ### Claim 17 (depends on 15) The method of claim 15, wherein generating the warped image comprises: calculating an optical flow from the first image and the second image; and shifting individual pixels of one of the first image and the second image based at least in part on the optical flow. ### Claim 18 (depends on 15) The method of claim 15, wherein the classifier module comprises: a convolutional neural network producing a plurality of spatial features; a long short-term memory module that receives the plurality of spatial features of the convolutional neural network as input and outputs a probability of the state of vehicle signal lights for each spatial feature; and a classifier that receives the probability of the state of vehicle signal lights and selects the state of vehicle signal lights of a spatial feature having a highest probability as a selected state of vehicle signal lights. ### Claim 19 (depends on 18) The method of claim 18, calculating a cross entropy loss for the probabilities of the state of vehicle signal lights for the plurality of spatial features. ### Claim 20 (depends on 15) The method of claim 15, calculating a photometric loss of between the warped image and one of the first image and the second image.