# US 10,455,165 B2 · Systems, methods, and computer-readable media for using a video capture device to alleviate motion sickness via an augmented display for a passenger > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,455,165 B2 - Document ID: US10455165B2 - Status: Granted - Kind: B2 - Application number: 15/392,283 - Filed: 2016-12-28 - Published: 2018-06-28 - Granted: 2019-10-22 - Earliest claimed: 2016-12-28 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Daniel Gray Goldstein - Web page: https://patentgrep.com/patent/US10455165B2 ## CPC - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34) - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04N 7/183: {for receiving images from a single remote source} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, methods, and computer-readable media are disclosed for using a video capture device to alleviate motion sickness via an augmented display for a passenger. One method includes receiving, from a video capture device of a vehicle, video data of a view outside of the vehicle while the vehicle is in motion; displaying, on a display device for a passenger of the vehicle, a request for a mode of display of the video data, the mode of displaying including a passive video mode and an active video mode; receiving a selection of the mode of display; displaying a request for a video display type when the passive video mode is selected; and displaying the video data including a menu, the menu including a plurality of selectable icons to interact with the video data when the active video mode is selected. ## Claims ### Claim 1 A computer-implemented method for using a video capture device to alleviate motion sickness via an augmented display for a passenger, the method comprising: receiving, from a video capture device of a vehicle, video data of a view outside of the vehicle while the vehicle is in motion; displaying, on a display device for a passenger of the vehicle, a request for a mode of display of the video data, the mode of displaying including a passive video mode and an active video mode; receiving, from the passenger, a selection of the mode of display; displaying, on the display device for the passenger, the video data as a transparent background of a desktop of the display device when the passive video mode is selected; analyzing, when the active video mode is selected, video data of the video capture device of the view outside of the vehicle while the vehicle is in motion to identify at least one target object in the video data representing a real object in the view outside of the vehicle; displaying, on the display device for the passenger when the active video mode is selected, the video data including a menu, the menu including a plurality of selectable icons to interact with the identified at least one target object in the video data, wherein the plurality of selectable icons are based on at least one of the identified real object and the analyzed video data; and displaying, on the display device for the passenger when the active video mode is selected, augment video data that includes at least one virtual object on or around the identified real object based on a selection of a first selectable icon of the plurality of selectable icons, wherein the at least one virtual object includes one or more of i) walls, ii) pot holes, iii) oil spills, iv) slime, and v) simulated objects in which at least one of the vehicle, a virtual vehicle, and a neighboring vehicle interacts. ### Claim 2 (depends on 1) The method according to claim 1, wherein the video capture device of the vehicle is one of mounted in the vehicle or on the vehicle. ### Claim 3 (depends on 1) The method according to claim 1, wherein the video data of the video capture device is of a forward-facing view outside of the vehicle while the vehicle is in motion. ### Claim 4 (depends on 1) The method according to claim 1, further comprising: receiving, from the passenger, the selection of the first selectable icon of the plurality of selectable icons, wherein the selection of the first selectable icon identifies the at least one virtual object to include in the augmented video data. ### Claim 5 (depends on 1) The method according to claim 1, wherein displaying augment video data includes: creating, by the display device, an augmented video data by one of merging, augmenting, and fusing the video data of the video capture device with the at least one virtual object, wherein the at least one virtual object are based on the selection of the first selectable icon by the passenger. ### Claim 6 A system for using a video capture device to alleviate motion sickness via an augmented display for a passenger, the system including: a data storage device that stores instructions for using a video capture device to alleviate motion sickness via an augmented display for a passenger; and a processor configured to execute the instructions to perform a method including: receiving, from a video capture device of a vehicle, video data of a view outside of the vehicle while the vehicle is in motion; displaying, on a display for a passenger of the vehicle, a request for a mode of display of the video data, the mode of displaying including a passive video mode and an active video mode; receiving, from the passenger, a selection of the mode of display; displaying, on the display for the passenger, the video data as a transparent background of a desktop of the display device when the passive video mode is selected; analyzing, when the active video mode is selected, video data of the video capture device of the view outside of the vehicle while the vehicle is in motion to identify at least one target object in the video data representing a real object in the view outside of the vehicle; displaying, on the display device for the passenger when the active video mode is selected, the video data including a menu, the menu including a plurality of selectable icons to interact with the identified at least one target object in the video data, wherein the plurality of selectable icons are based on at least one of the identified real object and the analyzed video data; and displaying, on the display device for the passenger when the active video mode is selected, augment video data that includes at least one virtual object on or around the identified real object based on a selection of a first selectable icon of the plurality of selectable icons, wherein the at least one virtual object includes one or more of i) walls, ii) pot holes, iii) oil spills, iv) slime, and v) simulated objects in which at least one of the vehicle, a virtual vehicle, and a neighboring vehicle interacts. ### Claim 7 (depends on 6) The system according to claim 6, wherein the video capture device of the vehicle is one of mounted in the vehicle or on the vehicle. ### Claim 8 (depends on 6) The system according to claim 6, wherein the video data of the video capture device is of a forward-facing view outside of the vehicle while the vehicle is in motion. ### Claim 9 (depends on 6) The system of claim 6, wherein the processor is further configured to execute the instructions to perform the method including: further comprising: receiving, from the passenger, the selection of the first selectable icon of the plurality of selectable icons, wherein the selection of the first selectable icon identifies the at least one virtual object to include in the augmented video data. ### Claim 10 (depends on 6) The system according to claim 6, wherein displaying augment video data includes: creating, by the display device, an augmented video data by one of merging, augmenting, and fusing the video data of the video capture device with the at least one virtual object, wherein the at least one virtual object are based on the selection of the first selectable icon by the passenger. ### Claim 11 A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for using a video capture device to alleviate motion sickness via an augmented display for a passenger, the method including: receiving, from a video capture device of a vehicle, video data of a view outside of the vehicle while the vehicle is in motion; displaying, on a display device for a passenger of the vehicle, a request for a mode of display of the video data, the mode of displaying including a passive video mode and an active video mode; receiving, from the passenger, a selection of the mode of display; displaying, on the display device for the passenger, the video data as a transparent background of a desktop of the display device when the passive video mode is selected; and analyzing, when the active video mode is selected, video data of the video capture device of the view outside of the vehicle while the vehicle is in motion to identify at least one target object in the video data representing a real object in the view outside of the vehicle; displaying, on the display device for the passenger when the active video mode is selected, the video data including a menu, the menu including a plurality of selectable icons to interact with the identified at least one target object in the video data, wherein the plurality of selectable icons are based on at least one of the identified real object and the analyzed video data; and displaying, on the display device for the passenger when the active video mode is selected, augment video data that includes at least one virtual object on or around the identified real object based on a selection of a first selectable icon of the plurality of selectable icons, wherein the at least one virtual object includes one or more of i) walls, ii) pot holes, iii) oil spills, iv) slime, and v) simulated objects in which at least one of the vehicle, a virtual vehicle, and a neighboring vehicle interacts. ### Claim 12 (depends on 11) The computer-readable medium of claim 11, wherein the video capture device of the vehicle is one of mounted in the vehicle or on the vehicle. ### Claim 13 (depends on 11) The computer-readable medium of claim 11, wherein the video data of the video capture device is of a forward-facing view outside of the vehicle while the vehicle is in motion. ### Claim 14 (depends on 11) The computer-readable medium of claim 11, further comprising: receiving, from the passenger, the selection of the first selectable icon of the plurality of selectable icons, wherein the selection of the first selectable icon identifies the at least one virtual object to include in the augmented video data. ### Claim 15 (depends on 1) The method according to claim 1, wherein the transparent background allows the passenger to both i) view the video data of the view outside of the vehicle while the vehicle is in motion and ii) interact with the desktop of a graphical user interface of the display device. ### Claim 16 (depends on 6) The system according to claim 6, wherein the transparent background allows the passenger to both i) view the video data of the view outside of the vehicle while the vehicle is in motion and ii) interact with the desktop of a graphical user interface of the display device. ### Claim 17 (depends on 11) The computer-readable medium of claim 11, wherein the transparent background allows the passenger to both i) view the video data of the view outside of the vehicle while the vehicle is in motion and ii) interact with the desktop of a graphical user interface of the display device.