# US 10,504,233 B1 · Augmented displays for multi-surface and three-dimensional live performances > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,233 B1 - Document ID: US10504233B1 - Status: Granted - Kind: B1 - Application number: 16/037,283 - Filed: 2018-07-17 - Granted: 2019-12-10 - Earliest claimed: 2018-06-08 - Assignees: Particle Ink, LLC - Applicants: Particle Ink, LLC - Inventors: Cassandra Sanford-Rosenthal; Sephi Joseph Shapira - Web page: https://patentgrep.com/patent/US10504233B1 ## CPC - H04N 9/3194: {including sensor feedback} - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - H04N 5/74: Projection arrangements for image reproduction, e.g. using eidophor - H04N 9/3147: {Multi-projection systems (video walls G06F3/1446)} - H04N 9/3179: {Video signal processing therefor} - G06T 2207/10048: Infrared image - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed are improvements to augmented reality technology by augmenting a physical scene of a live performance via images projected onto three-dimensional physical scenes with real-time IR signal detection for driving augmentation of the live performance. For example, an improved projection mapping technique is disclosed in which a physical object is augmented with a projection image that is projected onto a physical scene. In some instances, the projection may be made in accordance with a timeline. The timeline may be used to synchronize other actions and behaviors of the system as well as the actions of live performers such as actors and users such as audience members. In some instances, the projection images may be driven by audience or other inputs to the system, such as sensor-based inputs. The sensor-based or other inputs may also or alternatively drive physical components such as actuators that impact the real-world objects. ## Claims ### Claim 1 A computer implemented method of augmenting a physical scene of a live performance via projection overlays for driving augmentation of the live performance, wherein the physical scene is part of a multiple scene set comprising at least a second physical scene adjacent to the physical scene in the multiple scene set, and wherein the live performance includes a storyline that spans the multiple scene set, the method comprising: accessing, by a processor, sensor information from a sensor device; identifying, by the processor, a location on the physical scene at which a user device is directed based on the sensor information; generating, by the processor, one or more projection images to be newly projected onto the physical scene to augment the physical scene at the location; causing, by the processor, a projection device to project the one or more projection images onto the physical scene at the location; identifying, by the processor, a first projection image associated with a first portion of the storyline, the first projection image assigned to be projected by the projection device; causing, by the processor, the projection device to project the first projection image onto the physical scene; identifying, by the processor, a second projection image associated with a second portion of the storyline, the second projection image assigned to be projected by a second projection device; and causing, by the processor, the second projection device to project the second projection image onto the second physical scene after the first projection image is projected onto the physical scene. ### Claim 2 (depends on 1) The computer implemented method of claim 1, the method further comprising: adjusting the behavior of the one or more projection images; and causing the projection device to project the one or more projection images based on the adjusted behavior. ### Claim 3 (depends on 1) The computer implemented method of claim 1, wherein the multiple scene set is part of a site having multiple sets through which audience members traverse to experience the storyline such that the storyline spans the multiple sets, the method comprising: coordinating the projection of the first projection image at the physical scene with the projection of the second projection image at a second physical scene of a second set of the multiple sets based on a timeline of the storyline. ### Claim 4 (depends on 1) The computer implemented method of claim 1, wherein the live performance follows a storyline that includes an audience interactive element that requires an audience member to perform a physical action to advance the storyline, the method further comprising: detecting that the physical action has been performed; and controlling the projection device to project at least one projection image that corresponds to the physical action to give the appearance that the physical action caused a behavior of the least one projection image. ### Claim 5 (depends on 1) The computer implemented method of claim 1, the method further comprising: controlling the projection device to project at least one projection image that appears to interact with a physical object; and controlling an actuator device to move the physical object responsive to projection of the at least one projection image to give the appearance that the least one projection image caused the physical object to be moved. ### Claim 6 (depends on 1) The method of claim 1, further comprising: accessing a master timeline for synchronizing actions of the one or more projection images and a live performer; identifying a first projection image to be projected at a first time in the master timeline; and causing the projection device to project the first projection image based on the first time. ### Claim 7 (depends on 6) The method of claim 6, further comprising: recording a relevant scene and/or the one or more projection images based on the location of the physical scene at which the user device is directed. ### Claim 8 (depends on 6) The method of claim 6, further comprising: causing an actuator to activate to cause a physical change in the physical scene based on the timeline. ### Claim 9 A system of augmenting a physical scene of a live performance via projection overlays for driving augmentation of the live performance, wherein the physical scene is part of a multiple scene set comprising at least a second physical scene adjacent to the physical scene in the multiple scene set, and wherein the live performance includes a storyline that spans the multiple scene set, the system comprising: a processor programmed to: access sensor information from a sensor device; identify a location on the physical scene at which a user device is directed based on the sensor information; generate one or more projection images to be newly projected onto the physical scene to augment the physical scene at the location; cause a projection device to project the one or more projection images onto the physical scene at the location; identify a first projection image associated with a first portion of the storyline; cause a first projection device to project the first projection image onto the physical scene; identify a second projection image associated with a second portion of the storyline; and cause a second projection device to project the second projection image onto the second physical scene after the first projection image is projected onto the physical scene. ### Claim 10 (depends on 9) The system of claim 9, wherein to generate one or more projection images to augment the physical scene at the location, the processor is further programmed to: adjust the behavior of the one or more projection images; and cause the projection device to project the one or more projection images based on the adjusted behavior. ### Claim 11 (depends on 9) The system of claim 9, wherein the multiple scene set is part of a site having multiple sets through which audience members traverse to experience the storyline such that the storyline spans the multiple sets, wherein the processor is further programmed to: coordinate the projection of the first projection image at the physical scene with the projection of the second projection image at a second physical scene of a second set of the multiple sets based on a timeline of the storyline. ### Claim 12 (depends on 9) The system of claim 9, wherein the live performance follows a storyline that includes an audience interactive element that requires an audience member to perform a physical action to advance the storyline, wherein the processor is further programmed to: detect that the physical action has been performed; and control the projection device to project at least one projection image that corresponds to the physical action to give the appearance that the physical action caused a behavior of the least one projection image. ### Claim 13 (depends on 9) The system of claim 9, wherein the projector is further to: control the projection device to project at least one projection image that appears to interact with a physical object; wherein the system further comprises: an actuator device to move the physical object responsive to projection of the at least one projection image to give the appearance that the least one projection image caused the physical object to be moved. ### Claim 14 A computer implemented method of augmenting a physical scene of a live performance via projection overlays with real-time signal detection for driving augmentation of the live performance, wherein the physical scene is part of a multiple scene set comprising at least a second physical scene adjacent to the physical scene in the multiple scene set, and wherein the live performance includes a storyline that spans the multiple scene set, the method comprising: emitting, by a beacon device, a beacon signal for positioning a user device in the physical scene; accessing, by a processor, location information relating to a location of the user device, the location information based on the beacon signal; identifying, by the processor, a location on the physical scene at which the user device is directed based on the location information; generating, by the processor, one or more projection images to be newly projected onto the physical scene to augment the physical scene at the location; causing, by the processor, a projection device to project the one or more projection images at the location; identifying, by the processor, a first projection image associated with a first portion of the storyline, the first projection image assigned to be projected by the projection device; causing, by the processor, the projection device to project the first projection image onto the physical scene; identifying, by the processor, a second projection image associated with a second portion of the storyline, the second projection image assigned to be projected by a second projection device; and causing, by the processor, the second projection device to project the second projection image onto the second physical scene after the first projection image is projected onto the physical scene.