# US 2023/0410379 A1 · Augmenting shared digital content with dynamically generated digital content to improve meetings with multiple displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410379 A1
- Document ID: US20230410379A1
- Status: Published application
- Kind: A1
- Application number: 17/845,705
- Filed: 2022-06-21
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Mac Donald Scarpino JOHNSTON
- Web page: https://patentgrep.com/patent/US20230410379A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06T 7/50: Depth or shape recovery
- H04N 7/15: Conference systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure relates to systems, methods, and computer-readable media for utilizing a presentation enhancement system to provide enhancements for hybrid meetings (e.g., meetings where shared content is provided to multiple displays). For example, the presentation enhancement system generates dynamic digital content in response to detecting physical interactions of a presenting user with shared digital content on a display device. Further, the presentation enhancement system augments the shared digital content provided to a remote display device with the dynamic digital content. In addition to providing augmented shared digital content to remote display devices, the presentation enhancement system can perform several additional actions on the shared digital content based on the detected physical interactions.

## Claims

### Claim 1

A computer-implemented method comprising:
providing, to a first display device and a second display device, shared digital content, wherein the first display device projects a projected image of the shared digital content;
detecting, by a capture device, a physical interaction of a presenting user with the projected image of the shared digital content, wherein the physical interaction is a non-digital physical interaction that occurs separately from components of the first display device;
determining, utilizing a detection model that analyzes the physical interaction relative to the projected image, whether the physical interaction indicates a target location within the shared digital content at a nearby content item near the physical interaction or an away content item away from the physical interaction;
generating dynamic digital content with which to augment the shared digital content based on utilizing a detection model that analyzes the physical interaction of the presenting user with the projected image of the shared digital content;
generating the shared digital content augmented with the dynamic digital content that includes the dynamic digital content at:
the nearby content item when the target location is near the physical interaction; or
the away content item when the target location is away from the physical interaction; and
displaying the shared digital content augmented with the dynamic digital content on the second display device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the dynamic digital content comprises utilizing a movement detection model to determine a type of physical user movement outside of the first display device.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein generating the dynamic digital content comprises:
utilizing a pointer detection model to map the physical interaction of the presenting user, detected outside of the first display device, to the target location within the shared digital content; and
modifying the shared digital content by adding a visual element at the target location.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein:
the capture device comprises a depth camera that captures depth information of the presenting user; and
generating the dynamic digital content further comprises:
utilizing depth information with the pointer detection model to map the physical interaction of the presenting user to the target location within the shared digital content, wherein the target location is located away from the presenting user; and
modifying the shared digital content by adding the visual element at the target location.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the dynamic digital content is displayed on the second display device without being displayed on the first display device.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein generating the dynamic digital content comprises:
utilizing an annotation detection model to convert the physical interaction of the presenting user to digital text or shapes; and
modifying the shared digital content to comprise the digital text or shapes.

### Claim 7 (depends on 3)

The computer-implemented method of claim 3, wherein generating the dynamic digital content comprises:
utilizing a gesture detection model to convert the physical interaction of the presenting user to an action, wherein the physical interaction is with a content item of the shared digital content; and
modifying the content item of the shared digital content based on the action.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein detecting the physical interaction of the presenting user comprises detecting a finger, hand, or arm gesture by the presenting user corresponding to manipulating the shared digital content.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the second display device is located at a same location as the first display device.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the second display device is located at a remote location from the first display device.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein:
the first display device is a projector; and
the capture device is physically attached next to the projector.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein:
the first display device is a digital display monitor; and
the capture device is located away from the first display device.

### Claim 13

A system comprising:
a first display device that projects a projected image of shared digital content on a physical surface;
a second display device that displays the shared digital content;
a capture device that captures user movement with the projected image of the shared digital content on the physical surface;
at least one processor; and
a physical computer memory device comprising instructions that, when executed by the at least one processor, cause the system to:
detect, by the capture device that captures an image of at least a portion of a user and the projected image of the shared digital content on the physical surface, a non-digital physical interaction of a presenting user with the projected image of the shared digital content, wherein the non-digital physical interaction occurs separately from components of the first display device;
determine, utilizing a detection model that analyzes the non-digital physical interaction relative to the projected image, whether the non-digital physical interaction indicates a target location within the shared digital content at a nearby content item near the non-digital physical interaction or an away content item away from the non-digital physical interaction;
generate dynamic digital content with which to augment the shared digital content based on utilizing a detection model that analyzes the non-digital physical interaction of the presenting user with the projected image of the shared digital content;
generate the shared digital content augmented with the dynamic digital content that includes the dynamic digital content at:
the nearby content item when the target location is near the non-digital physical interaction; or
the away content item when the target location is away from the non-digital physical interaction; and
display the shared digital content augmented with the dynamic digital content on the second display device.

### Claim 14 (depends on 13)

The system of claim 13, wherein detecting the non-digital physical interaction of the presenting user comprises utilizing a facial detection model to identify the presenting user from other users.

### Claim 15 (depends on 13)

The system of claim 13, further comprising instructions that, when executed by the at least one processor, cause the system to generate a depth map from stereo images captured by the capture device.

### Claim 16 (depends on 15)

The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to determine from the depth map that the non-digital physical interaction of the presenting user occurs away from the projected image of the shared digital content.

### Claim 17 (depends on 16)

The system of claim 16, wherein generating the dynamic digital content with which to augment the shared digital content comprises:
determining the target location on the shared digital content away from the presenting user based on the depth map and the non-digital physical interaction of the presenting user; and
generating the dynamic digital content by adding a pointer visual element to the target location.

### Claim 18

A computer-implemented method for comprising:
providing, to a projector and a remote computing device, shared digital content, wherein the projector projects a projected image of the shared digital content on a wall or projection screen;
detecting, by a camera of a capture device that captures an image of at least a portion of a user, a physical interaction of a presenting user with the wall or projection screen displaying the projected image of the shared digital content, wherein the physical interaction is a non-digital physical interaction;
determining, utilizing a detection model that analyzes the physical interaction relative to the projected image, whether the physical interaction indicates a target location within the shared digital content at a nearby content item near the physical interaction or an away content item away from the physical interaction;
determining a particular action based on the image of the physical interaction of the presenting user;
generating dynamic digital content with which to augment the shared digital content based on the particular action;
generating the shared digital content augmented with the dynamic digital content that includes the dynamic digital content at:
the nearby content item when the target location is near the physical interaction; or
the away content item when the target location is away from the physical interaction; and
displaying the shared digital content augmented with the dynamic digital content on the remote computing device.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, further comprising:
detecting, by the capture device, that the presenting user moves to a second area away from the projected image of the shared digital content;
detecting user movement by the presenting user with a surface in the second area;
generating an additional shared digital content based on the user movement detected with the surface in the second area; and
providing the additional shared digital content on the remote computing device.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, further comprising providing the additional shared digital content to the projector to be projected on the wall or projection screen.
