# US 2019/0172267 A1 · Virtual work of expression within a virtual environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172267 A1 - Document ID: US20190172267A1 - Status: Published application - Kind: A1 - Application number: 16/252,706 - Filed: 2019-01-20 - Published: 2019-06-06 - Earliest claimed: 2015-01-23 - Applicants: YouMap, Inc. - Inventors: Stephen Constantinides - Web page: https://patentgrep.com/patent/US20190172267A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/174: {Facial expression recognition} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method for displaying a virtual three-dimensional environment, including: displaying at least a portion of the virtual three-dimensional environment in a head-mounted display, where the head-mounted display includes a retinal tracking device; enabling a user to interact with the virtual three-dimensional environment; displaying a virtual representation of a written work within the virtual three-dimensional environment in the head-mounted display; enabling the user to interact with the virtual representation of the written work; tracking the user's reading position in the representation of the written work via the retinal tracking device; determining the content of the representation of the written book at the user's reading position; and modifying the virtual three-dimensional environment based at least in part on the content at the user's reading position. ## Claims ### Claim 1 A system for displaying a virtual three-dimensional environment, the system comprising: a computer processor; a head-mounted display including a retinal tracking device; and a content engine executing on the computer processor and configured to: display at least a portion of the virtual three-dimensional environment in the head-mounted display; enable a user to interact with the virtual three-dimensional environment; display a virtual representation of a written work within the virtual three-dimensional environment in the head-mounted display; enable the user to interact with the virtual representation of the written work; track the user's reading position in the representation of the written work via the retinal tracking device; determine the content of the representation of the written book at the user's reading position; and modify the virtual three-dimensional environment based at least in part on the content at the user's reading position. ### Claim 2 (depends on 1) The system of claim 1: further comprising accessing a set of resources that provide contextual information associated with the content of the representation of the written book at the user's reading position; and wherein modifying the virtual three-dimensional environment is based at least in part on the contextual information. ### Claim 3 (depends on 2) The system of claim 2, wherein the set of resources comprises at least one selected from a group consisting of an Internet website, encyclopedia, research database, patent, government database, dictionary, periodical, magazine, and newspaper. ### Claim 4 (depends on 1) The system of claim 1, wherein modifying the virtual three-dimensional environment comprises producing an audio recitation in a character's voice in synchrony with the content at the user's reading position. ### Claim 5 (depends on 1) The system of claim 1, wherein modifying the virtual three-dimensional environment comprises modifying a visual scene. ### Claim 6 (depends on 1) The system of claim 1, wherein the virtual three-dimensional environment comprises at least one selected from a group consisting of a virtual reality environment, augmented reality environment, and mixed reality environment. ### Claim 7 (depends on 1) The system of claim 1, wherein the retinal tracking device comprises at least one selected from a group consisting of a camera, infrared camera, and EEG sensor. ### Claim 8 (depends on 1) The system of claim 1, wherein the written work comprises at least one selected from a group consisting of a book, journal, work of literature, work of expression, sheet music, text document. ### Claim 9 A method for displaying a virtual three-dimensional environment, the method comprising: displaying at least a portion of the virtual three-dimensional environment in a head-mounted display, wherein the head-mounted display includes a retinal tracking device; enabling a user to interact with the virtual three-dimensional environment; displaying a virtual representation of a written work within the virtual three-dimensional environment in the head-mounted display; enabling the user to interact with the virtual representation of the written work; tracking the user's reading position in the representation of the written work via the retinal tracking device; determining the content of the representation of the written book at the user's reading position; and modifying, by a computer processor, the virtual three-dimensional environment based at least in part on the content at the user's reading position. ### Claim 10 (depends on 9) The system of claim 9: further comprising accessing a set of resources that provide contextual information associated with the content of the representation of the written book at the user's reading position; and wherein modifying the virtual three-dimensional environment is based at least in part on the contextual information. ### Claim 11 (depends on 10) The system of claim 10, wherein the set of resources comprises at least one selected from a group consisting of an Internet website, encyclopedia, research database, patent, government database, dictionary, periodical, magazine, and newspaper. ### Claim 12 (depends on 9) The system of claim 9, wherein modifying the virtual three-dimensional environment comprises producing an audio recitation in a character's voice in synchrony with the content at the user's reading position. ### Claim 13 (depends on 9) The system of claim 9, wherein modifying the virtual three-dimensional environment comprises modifying a visual scene. ### Claim 14 (depends on 9) The system of claim 9, wherein the virtual three-dimensional environment comprises at least one selected from a group consisting of a virtual reality environment, augmented reality environment, and mixed reality environment. ### Claim 15 (depends on 9) The system of claim 9, wherein the retinal tracking device comprises at least one selected from a group consisting of a camera, infrared camera, and EEG sensor. ### Claim 16 (depends on 9) The system of claim 9, wherein the written work comprises at least one selected from a group consisting of a book, journal, work of literature, work of expression, sheet music, text document. ### Claim 17 A non-transitory computer-readable storage medium comprising a plurality of instructions for displaying a virtual three-dimensional environment, the plurality of instructions configured to execute on at least one computer processor to enable the at least one computer processor to: display at least a portion of the virtual three-dimensional environment in a head-mounted display, wherein the head-mounted display includes a retinal tracking device; enable a user to interact with the virtual three-dimensional environment; display a virtual representation of a written work within the virtual three-dimensional environment in the head-mounted display; enable the user to interact with the virtual representation of the written work; track the user's reading position in the representation of the written work via the retinal tracking device; determine the content of the representation of the written book at the user's reading position; and modify the virtual three-dimensional environment based at least in part on the content at the user's reading position. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the plurality of instructions are configured to execute on the at least one computer processor to further enable the at least one computer processor to: access a set of resources that provide contextual information associated with the content of the representation of the written book at the user's reading position, and wherein modifying the virtual three-dimensional environment is based at least in part on the contextual information. ### Claim 19 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein modifying the virtual three-dimensional environment comprises producing an audio recitation in a character's voice in synchrony with the content at the user's reading position. ### Claim 20 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein modifying the virtual three-dimensional environment comprises modifying a visual scene.