# US 11,113,961 B2 · Driver behavior monitoring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,961 B2
- Document ID: US11113961B2
- Status: Granted
- Kind: B2
- Application number: 16/704,520
- Filed: 2019-12-05
- Published: 2020-04-09
- Granted: 2021-09-07
- Earliest claimed: 2016-01-11
- Assignees: Netradyne, Inc.
- Applicants: Netradyne, Inc.
- Inventors: David Jonathan Julian; Avneesh Agrawal
- Web page: https://patentgrep.com/patent/US11113961B2

## CPC

- G08G 1/052: with provision for determining speed or overspeed {(speed measuring in general G01P)}
- G06F 18/24: Classification techniques
- G06F 18/24133: {Distances to prototypes}
- G06V 10/40: Extraction of image or video features
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)}
- G07C 5/0866: {the electronic data carrier being a digital video recorder in combination with video camera}
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G08G 1/0129: {for creating historical data or processing based on historical data}
- G08G 1/0133: {for classifying traffic situation}
- G08G 1/0137: {for specific applications}
- G08G 1/015: with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
- G08G 1/04: using optical or ultrasonic detectors
- B60W 40/09: Driving style or behaviour
- B60W 2420/403: Image sensing, e.g. optical camera
- G06Q 40/08: Insurance
- G06V 20/44: {Event detection}
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods provide, implement, and use using a computer-vision based methods of context-sensitive monitoring and characterization of driver behavior. Additional systems and methods are provided for unsupervised learning of action values, monitoring of a driver's environment, and transmitting visual information from a client to a server.

## Claims

### Claim 1

A method comprising:
detecting, by at least one processor of a computing device in communication with a camera, the presence of a traffic light, a state of the traffic light, and an intersection marking in a field of view of the camera, wherein the camera is mounted on or in a vehicle; and
determining, by the at least one processor, that the vehicle passed the intersection marking after the traffic light turned red, based at least in part on first visual data captured by the camera,
wherein detecting the presence of the traffic light further comprises:
initially receiving, by the at least one processor, the first visual data; and
identifying, by the at least one processor, a location of the traffic light in the first visual data, and
wherein determining that the vehicle passed the intersection marking after the traffic light turned red further comprises:
identifying, by the at least one processor, that the traffic light is in a red state in the first visual data;
determining, by the at least one processor, that the vehicle was not in the intersection at a first time corresponding to when the visual data was captured;
receiving, by the at least one processor, second visual data captured by the camera, wherein the second visual data is captured by the camera subsequent to the capture of the first visual data;
identifying, by the at least one processor, a second location of the traffic light in the second visual data; based at least in part on the first location of the traffic light in the first visual data;
identifying, by the at least one processor, that the traffic light is in a red state in the second visual data; and
determining, by the at least one processor, that the vehicle was in the intersection at a second time; wherein the second time corresponds to when the second visual data was captured.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining that the vehicle passed the intersection marking after the traffic light turned red further comprises:
determining, by the at least one processor, a distance between the vehicle and the intersection at the first time, wherein the traffic light is associated with the intersection; and
determining, by the at least one processor, that the vehicle did not stop before passing through the intersection.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining that the vehicle passed the intersection marking after the traffic light turned red further comprises:
recognizing, by the at least one processor, an intersection marking in the visual data; and
determining, by the at least one processor, that the vehicle did not stop before passing through the intersection.

### Claim 4 (depends on 1)

The method of claim 1, wherein identifying the location of the traffic light comprises:
determining, by the at least one processor, a bounding box for the traffic light; and
determining, by the at least one processor, a probability that a portion of the visual data associated with the bounding box contains a traffic light.

### Claim 5 (depends on 1)

The method of claim 1, wherein the computing device in communication with the camera is located in or on the vehicle.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining, by the at least one processor, that the vehicle did not stop before passing through the intersection based on the visual data captured by the camera alone.

### Claim 7

A method comprising:
detecting, by at least one processor of a computing device in communication with a camera, at a first time, the presence of a traffic light, a state of the traffic light, and an intersection marking in a field of view of the camera, wherein the camera is mounted on or in a vehicle;
detecting, by the at least one processor, that the traffic light turned yellow at a second time subsequent to the first time;
determining, by the at least one processor, a distance between the vehicle and the intersection marking at the second time, wherein the traffic light is associated with the intersection marking; and
determining, by the at least one processor, a speed of the vehicle at the second time;
determining, by the at least one processor, that the vehicle did not stop at the intersection;
determining, by the at least one processor, that the traffic light turned red before the vehicle was in the intersection; and
determining, by the at least one processor, that the vehicle appropriately ran a red light based on at least one of: the determined distance between the vehicle and the intersection at the first time, the speed of the vehicle at the second time, or some combination thereof.

### Claim 8 (depends on 7)

The method of claim 7, wherein detecting the presence of a traffic light further comprises:
initially receiving, by the at least one processor, first visual data captured by the camera at the first time;
identifying, by the at least one processor, a first location of the traffic light in the first visual data; and
determining, by the at least one processor, that the traffic light is in a green state in the first visual data;
and wherein detecting that the traffic light turned yellow further comprises:
receiving, by the at least one processor, second visual data captured by the camera, wherein the second visual data is captured by the camera at the second time;
identifying, by the at least one processor, a second location of the traffic light in the second visual data; based at least in part on the first location of the traffic light in the first visual data; and
identifying, by the at least one processor, that the traffic light is in a yellow state in the second visual data.

### Claim 9 (depends on 7)

A method of claim 7, further comprising:
detecting, by at least one processor of a computing device in communication with a camera, at a first time, a presence of a traffic light, wherein the camera is mounted on or in a vehicle;
determining, by the at least one processor, that the vehicle stopped at the traffic light;
determining, by the at least one processor, that the traffic light turned green;
measuring, by the at least one processor, an elapsed time between the light turning green and the vehicle starting forward;
assessing, by the at least one processor, that a driver of the vehicle is being responsive to traffic signals based at least in part on the elapsed time;
detecting, by the at least one processor, a pedestrian based on visual data captured by the camera; and
grading, by the at least one processor, the driver positively for waiting at a green light based on the detected pedestrian.

### Claim 10 (depends on 9)

The method of claim 9, further comprising determining an extent to which a driver obeys traffic lights, based at least in part on the determination that the vehicle stopped at the traffic light.