# US 12,005,919 B2 · Varying extended reality content based on risk level of a driving environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,005,919 B2 - Document ID: US12005919B2 - Status: Granted - Kind: B2 - Application number: 17/677,125 - Filed: 2022-02-22 - Published: 2023-08-24 - Granted: 2024-06-11 - Earliest claimed: 2022-02-22 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Hiroshi Yasuda; Simon A. I. Stent - Web page: https://patentgrep.com/patent/US12005919B2 ## CPC - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60K 35/50: Instruments characterised by their means of attachment to or integration in the vehicle (B60K35/231 takes precedence) - B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10) - B60W 40/02: related to ambient conditions - G02B 27/017: {Head mounted} - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/22: Display screens - B60K 35/235: with means for detecting the driver's gaze direction or eye points - B60K 35/81: for controlling displays - B60K 2360/165: Videos and animations - B60K 2360/176: Camera images - B60K 2360/177: Augmented reality - B60K 2360/178: Warnings - B60K 2360/1868: according to driving situations - B60W 2050/146: {Display means} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Extended reality content in a video can be varied based on a risk level of an external environment of a vehicle. The video can be of an external environment of a vehicle can be presented in real-time on a display located within the vehicle. The display can be a video pass through display. The display can be an in-vehicle display, or it can be a part of a video pass-through extended reality headset. The video can present a view of an external environment of the vehicle as well as extended reality content. A risk level of the external environment of the vehicle can be determined. An amount of the extended reality content presented in the video can be varied based on the risk level. ## Claims ### Claim 1 A method of varying extended reality content in a video presented on a display located within a vehicle, the display being a video pass through display, the video including extended reality content, the method comprising: determining a risk level of an external environment of the vehicle based on a complexity of the external environment of the vehicle, the complexity being based on at least one of: a number of objects detected in the external environment, a density of objects detected in the external environment, a number of different directions that objects detected in the external environment are moving, a number of different types of objects detected in the external environment, a number of a particular type of object detected in the external environment, an amount of occlusions of individual objects detected in the external environment, and a degree of agreement in data captured by one or more sensors of the vehicle; and causing an amount of the extended reality content presented in the video to be varied based on the risk level, the amount being proportional to the risk level of the external environment of the vehicle, the amount including: a number of augmented objects in the video; or a number of extended reality overlays over one or more objects and/or one or more portions of the external environment in the video, whereby varying the amount of extended reality content signals to a human driver that greater or lesser attention can be paid to the external environment due to the risk level. ### Claim 2 (depends on 1) The method of claim 1, further including: determining whether the risk level meets a threshold; and responsive to determining that the risk level meets the threshold, causing the amount of extended reality content presented in the video to be reduced. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the risk level includes determining the risk level based on a likelihood of the vehicle being impinged upon by another object in the external environment of the vehicle. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the risk level includes determining the risk level based on a complexity of the external environment of the vehicle. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the risk level includes determining the risk level based on an operational mode of one or more vehicles in the external environment of the vehicle. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the risk level includes determining the risk level based on a behavior of one or more vehicles in the external environment of the vehicle. ### Claim 7 (depends on 1) The method of claim 1, wherein the display is part of: a head mounted display; or an in-vehicle display. ### Claim 8 (depends on 1) The method of claim 1, further including: capturing, using one or more cameras, video of an external environment of the vehicle; augmenting with video with extended reality content; and causing the video with extended reality content to be presented on the display. ### Claim 9 (depends on 1) The method of claim 1, wherein the extended reality content includes: extended reality overlays over corresponding real-world objects or environment; or extended reality modifications to real-world objects or environment. ### Claim 10 (depends on 9) The method of claim 9, wherein the extended reality content is applied in a selected style. ### Claim 11 A system for varying extended reality content in a video, the system comprising: a display located within a vehicle, the display being a video pass through display; one or more cameras; and one or more processors operatively connected to the one or more cameras and to the display, the one or more processors being programmed to initiate executable operations comprising: causing the video captured by the one or more cameras to be presented on the display, the video including extended reality content; determining a risk level of an external environment of the vehicle based on a complexity of the external environment of the vehicle, the complexity being based on at least one of: a number of objects detected in the external environment, a density of objects detected in the external environment, a number of different directions that objects detected in the external environment are moving, a number of different types of objects detected in the external environment, a number of a particular type of object detected in the external environment, an amount of occlusions of individual objects detected in the external environment, and a degree of agreement in data captured by one or more sensors of the vehicle; and causing an amount of the extended reality content presented in the video to be varied based on the risk level, the amount being proportional to the risk level of the external environment of the vehicle, the amount including: a number of augmented objects in the video; or a number of extended reality overlays over one or more objects and/or one or more portions of the external environment in the video, whereby varying the amount of extended reality content signals to a human driver that greater or lesser attention can be paid to the external environment due to the risk level. ### Claim 12 (depends on 11) The system of claim 11, further including: determining whether the risk level meets a threshold; and responsive to determining that the risk level meets the threshold, causing the amount of extended reality content presented in the video to be reduced. ### Claim 13 (depends on 11) The system of claim 11, wherein determining the risk level includes determining the risk level based on a likelihood of the vehicle being impinged upon by another object in the external environment of the vehicle. ### Claim 14 (depends on 11) The system of claim 11, wherein determining the risk level includes determining the risk level based on a complexity of the external environment of the vehicle. ### Claim 15 (depends on 11) The system of claim 11, wherein determining the risk level includes determining the risk level based on an operational mode of one or more vehicles in the external environment of the vehicle. ### Claim 16 (depends on 11) The system of claim 11, wherein determining the risk level includes determining the risk level based on a behavior of one or more vehicles in the external environment of the vehicle. ### Claim 17 (depends on 11) The system of claim 11, wherein the display is part of: a head mounted display; or an in-vehicle display. ### Claim 18 (depends on 11) The system of claim 11, wherein the extended reality content includes: extended reality overlays over corresponding real-world objects or environment; or extended reality modifications to real-world objects or environment. ### Claim 19 (depends on 18) The system of claim 18, wherein the extended reality content is applied in a selected style. ### Claim 20 (depends on 11) The system of claim 11, wherein the one or more cameras are mounted on the vehicle.