# US 11,999,366 B2 · System and methods thereof for monitoring proper behavior of an autonomous vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,999,366 B2 - Document ID: US11999366B2 - Status: Granted - Kind: B2 - Application number: 17/122,124 - Filed: 2020-12-15 - Published: 2021-06-17 - Granted: 2024-06-04 - Earliest claimed: 2019-12-17 - Assignees: Foretellix Ltd. - Applicants: Foretellix Ltd. - Inventors: Yoav Hollander; Oded Doron Hirschfeld; Yaron Kashai - Web page: https://patentgrep.com/patent/US11999366B2 ## CPC - B60W 50/045: {Monitoring control system parameters} - G06F 11/3698: Environments for analysis, debugging or testing of software - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - G06F 11/3684: {for test design, e.g. generating new test cases} - G06F 30/15: Vehicle, aircraft or watercraft design - G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) - B60W 2050/0018: {Method for the design of a control system} - B60W 2050/0082: {for initialising the control system} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system and methods thereof for monitoring proper behavior of an autonomous vehicle are provided. The method includes generating a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT, generating a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents, and monitoring an interaction between the plurality of agents and the DUT agent for a scenario modeling the respective agent. ## Claims ### Claim 1 A method performed by a computer-implemented monitoring system for monitoring behavior of a simulated device under test (DUT), comprising: receiving a data stream of an at least one physical object; generating a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT and wherein an agent for the at least one physical object is one of the plurality of agents, and wherein generating the agent for the at least one physical object is based on the received data stream; generating a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents; monitoring an interaction between the plurality of agents other than the DUT agent and the DUT agent for a scenario modeling the agents; generating an error indication based on the monitoring when there is a failure of the DUT agent to respond in a manner priorly defined as acceptable during at least one of the plurality of scenarios; generating an alert corresponding to the error indication when the error indication is generated; and supplying information to enable recreation of a sequence that led to the error indication when an error indication is generated. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one physical object includes more than one physical object. ### Claim 3 (depends on 2) The method of claim 2, further comprising: generating at least a scenario for at least one object of the more than one physical objects. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one physical object describes at least one of: a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a traffic cone, a barrier, a bicycle, a train, and a weather element. ### Claim 5 (depends on 1) The method of claim 1, wherein the DUT agent is an autonomous vehicle (AV). ### Claim 6 (depends on 1) The method of claim 1, wherein a first scenario of the plurality of scenarios includes at least a second scenario of the plurality of scenarios. ### Claim 7 (depends on 5) The method of claim 5, wherein a scenario of the plurality of scenarios describes at least one of: a cut-in in front of another vehicle, a cut from a left lane of another vehicle, a cut from a right lane of another vehicle, cuts from two lanes simultaneously, slowing in front of the autonomous vehicle, cutting and slowing in front of the DUT, and traffic light changes. ### Claim 8 (depends on 1) The method of claim 1, further comprising: generating, based on the monitoring, at least one of: a notification upon detection of a monitored event, a key performance indicator (KPI), and a coverage information. ### Claim 9 (depends on 1) The method of claim 1, wherein the data stream comprises one or more video segments. ### Claim 10 (depends on 1) The method of claim 1, further comprising: executing by the computer-implemented monitoring system at least two of the plurality of agents in parallel. ### Claim 11 (depends on 1) The method of claim 1, further comprising: executing by the computer-implemented monitoring system at least two of the plurality of scenarios in parallel. ### Claim 12 (depends on 1) The method of claim 1, wherein each scenario of the plurality of scenarios and each agent of the plurality of agents are described in a measurable scenario descriptive language (MSDL). ### Claim 13 (depends on 12) The method of claim 12, wherein the MSDL is a declarative programming language for describing the behavior of the DUT and physical objects. ### Claim 14 (depends on 1) A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute the method of claim 1. ### Claim 15 A system for monitoring behavior of a simulated device under test (DUT), comprising: a network interface; an input/output (I/O) interface; a database; a processing unit communicatively connected to the network interface, the I/O interface and the database, the processing unit being adapted to execute a plurality of instructions provided thereto; a memory, a portion of which contains instructions for execution, wherein upon execution of the instructions by the processing unit, the monitoring system is adapted to: receive a data stream of an at least one physical object; generate a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT and wherein an agent for the at least one physical object is one of the plurality of agents, and wherein generating the agent for the at least one physical object is based on the received data stream; generate a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents; monitor an interaction between the plurality of agents other than the DUT agent and the DUT agent for a scenario modeling the agents; generate an error indication based on the monitoring when there is a failure of the DUT agent to respond in a manner priorly defined as acceptable during at least one of the plurality of scenarios; generate an alert corresponding to the error indication when the error indication is generate; and supply information to enable recreation of a sequence that led to the error indication when an error indication is generated. ### Claim 16 (depends on 15) The system of claim 15, wherein the at least one physical object includes more than one physical object. ### Claim 17 (depends on 16) The system of claim 16, wherein the system is further configured to: generate at least a scenario for at least one object of the more than one physical objects. ### Claim 18 (depends on 15) The system of claim 15, wherein the at least one physical object describes at least one of: a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a traffic cone, a barrier, a bicycle, a train and a weather element. ### Claim 19 (depends on 15) The system of claim 15, wherein the DUT agent is an autonomous vehicle (AV). ### Claim 20 (depends on 15) The system of claim 15, wherein a first scenario of the plurality of scenarios includes at least a second scenario of the plurality of scenarios. ### Claim 21 (depends on 19) The system of claim 19, wherein a scenario of the plurality of scenarios describes at least one of: a cut-in in front of another vehicle, a cut from a left lane of another vehicle, a cut from a right lane of another vehicle, cuts from two lanes simultaneously, slowing in front of the autonomous vehicle, cutting and slowing in front of the DUT, and traffic light changes. ### Claim 22 (depends on 15) The system of claim 15, wherein the system is further configured to: generate, based on the monitoring, at least one of: a notification upon detection of a monitored event, a key performance indicator (KPI), and a coverage information. ### Claim 23 (depends on 15) The system of claim 15, wherein the data stream comprises one or more video segments. ### Claim 24 (depends on 15) The system of claim 15, wherein the system is further configured to: execute by the system at least two of the plurality of agents in parallel. ### Claim 25 (depends on 15) The system of claim 15, wherein the system is further configured to: execute by the system at least two of the plurality of scenarios in parallel. ### Claim 26 (depends on 15) The system of claim 15, wherein each scenario of the plurality of scenarios and each agent of the plurality of agents are described in a measurable scenario descriptive language (MSDL). ### Claim 27 (depends on 26) The system of claim 26, wherein the DUT agent is an autonomous vehicle (AV) and wherein the MSDL is a declarative programming language for describing the behavior of the autonomous vehicle and physical objects.