# US 2021/0233322 A1 · System and method for implementing computer-simulated reality interactions between users and publications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233322 A1
- Document ID: US20210233322A1
- Status: Published application
- Kind: A1
- Application number: 17/230,922
- Filed: 2021-04-14
- Published: 2021-07-29
- Earliest claimed: 2016-06-03
- Assignees: A Big Chunk of Mud LLC
- Applicants: A Big Chunk of Mud LLC
- Inventors: J. Michelle HAINES
- Web page: https://patentgrep.com/patent/US20210233322A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06V 20/20: in augmented reality scenes
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06V 2201/131: Book
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and media for enhancing one or more publications by receiving live video captured by a user, the live video comprising video of a publication, the publication comprising copyrighted content; identifying at least one first trigger in the live video, identifying one or more first three-dimensional, interactive media associated with the at least one first trigger and pertaining to the copyrighted content, and presenting to the user the first three-dimensional, interactive media; and identifying at least one second trigger in the first three-dimensional, interactive media, identifying one or more second three-dimensional, interactive media associated with the at least one second trigger and pertaining to the copyrighted content, and presenting to the user the second three-dimensional, interactive media to progressively deepen and enrich the engagement with the copyrighted content of the publication.

## Claims

### Claim 1

A computer-implemented method of enhancing a publication comprising:
receiving one or more live camera feeds comprising image data of a publication that includes a plurality of triggers and image data of the eyes of a user;
identifying, automatically, at least one trigger in the one or more live camera feeds that provides one or more augmented reality display media;
determining a field of view of the user based on the image data of the eyes of the user;
presenting, via a visual output device, the one or more augmented reality display media as a virtual overlay aligned with a portion of the publication included in the field of view of the user;
identifying one or more virtual reality environments pertaining to the publication caused by user engagement with at least one digital trigger in a portion of the one or more augmented reality display media within the user field of view; and
terminating presentation of the one or more augmented reality display media and presenting via the visual output device, the one or more virtual reality environments.

### Claim 2 (depends on 1)

The method of claim 1, further comprising identifying, in the image data of the eyes of the user, a user response to at least one of a portion of the one or more augmented reality display media or a portion of the one or more virtual reality environments; and
dynamically shifting between the one or more augmented reality display media and the one or more virtual reality environments based on the user response.

### Claim 3 (depends on 1)

The method of claim 1, wherein the publication is a non-fiction publication.

### Claim 4 (depends on 1)

The method of claim 1, wherein the augmented reality display media includes a 3D interactive animated graphic representation of one or more items described in the publication.

### Claim 5 (depends on 1)

The method of claim 1, wherein the at least one trigger is a graphic in the publication.

### Claim 6 (depends on 1)

The method of claim 1, further comprising generating at least one of the one or more augmented reality display media or the one or more virtual reality environments by adapting content included in the publication into a 3D immersive scene.

### Claim 7 (depends on 1)

The method of claim 1, further comprising creating a new scene based on the content included in the publication; and
generating the one or more augmented reality display media or the one or more virtual reality environments by adapting the new scene into an immersive scene.

### Claim 8 (depends on 1)

The method of claim 1, further comprising tracking time spent and content viewed by the user in a location included in at least one of the one or more augmented reality display media or the one or more virtual reality environments; and
modifying a presentation of at least one of the one or more augmented reality display media or the one or more virtual reality environments based on the time spent and content viewed by the user in the location.

### Claim 9 (depends on 1)

The method of claim 1, further comprising receiving a user input indicating a selection of how at least one of the one or more augmented reality display media or the one or more virtual reality environments should be presented; and
modifying a presentation of at least one of the one or more augmented reality display media or the one or more virtual reality environments based on the user input.

### Claim 10 (depends on 1)

The method of claim 1, wherein the at least one digital trigger is an audio trigger.

### Claim 11 (depends on 10)

The method of claim 10, wherein the audio trigger comprises at least one of audio key words associated with the text of the publication, sub-vocal keywords as a hidden track underneath an audio track included in the one or more augmented reality display media, audible tones, audible tunes, sub-audible tones, and sub-audible tunes.

### Claim 12

A computer-implemented method of enhancing a publication comprising:
receiving one or more live camera feeds comprising image data of a publication that includes a plurality of triggers;
identifying, automatically, at least one trigger in the one or more live camera feeds that provides one or more augmented reality display media;
presenting, via a visual output device, the one or more augmented reality display media;
identifying a piece of content pertaining to the publication caused by user engagement with at least one digital trigger in a portion of the one or more augmented reality display media;
receiving a user input indicating a selection of how the piece of content should be presented;
determining to present the piece of content as a virtual reality environment based on the user input and a determining that the piece of content is not already being presented as a virtual reality environment; and
terminating presentation of the one or more augmented reality display media and presenting via the visual output device, the virtual reality environment.

### Claim 13 (depends on 12)

The method of claim 12, wherein the user input comprises at least one of a voice input, a tactile input, an attempted tactile input, a gesture, and an eye movement.

### Claim 14

A computer-implemented method of enhancing a publication comprising:
receiving a live camera feed comprising image data of a publication that includes a plurality of triggers;
identifying, automatically, at least one trigger in the live camera feed that provides one or more augmented reality display media;
presenting, via a visual output device, the one or more augmented reality display media;
identifying one or more virtual reality environments pertaining to the publication caused by user engagement with at least one digital trigger in a portion of the one or more augmented reality display media;
terminating presentation of the one or more augmented reality display media and presenting via the visual output device, the one or more virtual reality environments;
identifying one or more mixed reality media pertaining to the publication based on a determination that the one or more virtual reality environments are not being presented as a mixed reality presentation; and
shifting the presentation of the one or more virtual reality environments to a mixed realty presentation that embeds the one or more augmented reality display media within the one or more virtual reality environments.

### Claim 15 (depends on 14)

The method of claim 14, wherein the publication is a non-fiction publication.

### Claim 16 (depends on 14)

The method of claim 14, wherein the augmented reality display media includes a 3D interactive animated graphic representation of one or more items described in the publication.

### Claim 17 (depends on 14)

The method of claim 14, wherein the at least one trigger is a graphic in the publication.

### Claim 18 (depends on 14)

The method of claim 14, further comprising generating at least one of the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation by adapting content included in the publication into an immersive scene.

### Claim 19 (depends on 14)

The method of claim 14, further comprising creating a new scene based on the content included in the publication; and
generating the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation by adapting the new scene into a 3D immersive scene.

### Claim 20 (depends on 14)

The method of claim 14, further comprising tracking time spent and content viewed by the user in a location included in at least one of the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation; and
modifying a presentation of at least one of the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation based on the time spent and content viewed by the user in the location.

### Claim 21 (depends on 14)

The method of claim 14, further comprising receiving a user input indicating a selection of how at least one of the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation should be presented; and
modifying a presentation of at least one of the one or more augmented reality display media, the one or more virtual reality environments, or the mixed reality presentation based on the user input.

### Claim 22 (depends on 14)

The method of claim 14, wherein the at least one digital trigger is an audio trigger.

### Claim 23 (depends on 22)

The method of claim 22, wherein the audio trigger comprises at least one of audio key words associated with the text of the publication, sub-vocal keywords as a hidden track underneath an audio track included in the one or more augmented reality display media or the mixed reality presentation, audible tones, audible tunes, sub-audible tones, and sub-audible tunes.
