# US 2023/0264697 A1 · Varying extended reality content based on driver attentiveness > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0264697 A1 - Document ID: US20230264697A1 - Status: Published application - Kind: A1 - Application number: 17/676,995 - Filed: 2022-02-22 - Published: 2023-08-24 - Earliest claimed: 2022-02-22 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Hiroshi Yasuda; Simon A.I. Stent - Web page: https://patentgrep.com/patent/US20230264697A1 ## CPC - B60W 40/08: related to drivers or passengers - B60K 35/235: with means for detecting the driver's gaze direction or eye points - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60K 35/50: Instruments characterised by their means of attachment to or integration in the vehicle (B60K35/231 takes precedence) - B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10) - B60W 60/0057: {Estimation of the time available or required for the handover} - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 2360/165: Videos and animations - B60K 2360/175: Autonomous driving - B60K 2360/176: Camera images - B60K 2360/177: Augmented reality - B60K 2360/21: using cameras - B60W 2040/0863: {due to erroneous selection or response of the driver} - B60W 2540/22: Psychological state; Stress level or workload - B60W 2540/225: Direction of gaze - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES ## Abstract Extended reality content in a video can be varied based on driver attentiveness. The video can be of an external environment of a vehicle and can be presented in real-time on a display located within the vehicle. The display can be a video pass through display. The display can be an in-vehicle display, or it can be a part of a video pass-through extended reality headset. The video can present a view of an external environment of the vehicle as well as extended reality content. A level of attentiveness of a driver of the vehicle can be determined. An amount of the extended reality content presented in the video can be varied based on the level of attentiveness. ## Claims ### Claim 1 A method of varying extended reality content in a video presented on a display located within a vehicle, the display being a video pass through display, the video including extended reality content, the method comprising: determining a level of attentiveness of a driver of the vehicle; and causing an amount of the extended reality content presented in the video to be varied based on the level of attentiveness. ### Claim 2 (depends on 1) The method of claim 1, further including: determining whether the level of attentiveness meets a threshold; and responsive to determining that the level of attentiveness does not meet the threshold, causing the amount of extended reality content presented in the video to be reduced. ### Claim 3 (depends on 2) The method of claim 2, further including: responsive to determining that the level of attentiveness does not meet the threshold, causing the amount of extended reality content presented in the video to remain the same or to be increased. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the level of attentiveness includes: determining the level of attentiveness based on an amount of attention a driver pays to the video presented on the display. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the level of attentiveness includes: determining the level of attentiveness based on a response time of the driver. ### Claim 6 (depends on 5) The method of claim 5, wherein, when the vehicle is in an autonomous operational mode, the response time of the driver is an amount of time for a driver to take control of the vehicle after being requested to take control. ### Claim 7 (depends on 1) The method of claim 1, wherein the display is part of: a head mounted display; or an in-vehicle display. ### Claim 8 (depends on 1) The method of claim 1, further including: capturing, using one or more cameras, video of an external environment of the vehicle; augmenting with video with extended reality content; and causing the video with extended reality content to be presented on the display. ### Claim 9 (depends on 1) The method of claim 1, wherein the extended reality content includes: extended reality overlays over corresponding real-world objects or environment; or extended reality modifications to real-world objects or environment. ### Claim 10 (depends on 9) The method of claim 9, wherein the extended reality content is applied in a selected style. ### Claim 11 A system for varying extended reality content in a video, the system comprising: a display located within a vehicle, the display being a video pass through display; one or more cameras; and one or more processors operatively connected to the one or more cameras and to the display, the one or more processors being programmed to initiate executable operations comprising: causing the video captured by the one or more cameras to be presented on the display, the video including extended reality content; determining a level of attentiveness of a driver of the vehicle; and causing an amount of the extended reality content presented in the video to be varied based on the level of attentiveness. ### Claim 12 (depends on 11) The system of claim 11, further including: determining whether the level of attentiveness meets a threshold; and responsive to determining that the level of attentiveness does not meet the threshold, causing the amount of extended reality content presented in the video to be reduced. ### Claim 13 (depends on 12) The system of claim 12, further including: responsive to determining that the level of attentiveness does not meet the threshold, causing the amount of extended reality content presented in the video to remain the same or to be increased. ### Claim 14 (depends on 11) The system of claim 11, wherein determining the level of attentiveness includes: determining the level of attentiveness based on an amount of attention the driver pays to the video presented on the display. ### Claim 15 (depends on 11) The system of claim 11, wherein determining the level of attentiveness includes: determining the level of attentiveness based on a response time of the driver. ### Claim 16 (depends on 15) The system of claim 15, wherein, when the vehicle is in an autonomous operational mode, the response time of the driver is an amount of time for a driver to take control of the vehicle after being requested to take control. ### Claim 17 (depends on 11) The system of claim 11, wherein the display is part of: a head mounted display; or an in-vehicle display. ### Claim 18 (depends on 11) The system of claim 11, wherein the extended reality content includes: extended reality overlays over corresponding real-world objects or environment; or extended reality modifications to real-world objects or environment. ### Claim 19 (depends on 18) The system of claim 18, wherein the extended reality content is applied in a selected style. ### Claim 20 (depends on 11) The system of claim 11, wherein the one or more cameras are mounted on the vehicle.