# US 2021/0082083 A1 · Dynamic adaptation of images for projection, and/or of projection parameters, based on user(s) in environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082083 A1
- Document ID: US20210082083A1
- Status: Published application
- Kind: A1
- Application number: 17/107,250
- Filed: 2020-11-30
- Published: 2021-03-18
- Earliest claimed: 2018-04-17
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Jian Wei Leong
- Web page: https://patentgrep.com/patent/US20210082083A1

## CPC

- H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06T 3/00: Geometric image transformations in the plane of the image
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 3/4084: in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
- G06T 3/608: by skew deformation, e.g. two-pass or three-pass rotation
- G06V 10/753: {Transform-based matching, e.g. Hough transform}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- H04N 9/3188: {Scale or resolution adjustment (scaling in general G06T3/40; resolution modifying circuits for displays in general G09G5/391)}
- H04N 9/3194: {including sensor feedback}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G09G 2320/0261: in the context of movement of objects on the screen or movement of the observer relative to the screen
- G09G 2340/04: Changes in size, position or resolution of an image
- G09G 2340/0492: Change of orientation of the displayed image, e.g. upside-down, mirrored
- G09G 2360/144: the light being ambient light
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Implementations relate to dynamic adaptation of images for projection by a projector, based on one or more properties of user(s) that are in an environment with the projector. The projector can be associated with an automated assistant client of a client device. In some versions of those implementations, a pose of a user in the environment is determined and, based on the pose, a base image for projecting onto a surface is warped to generate a transformed image. The transformed image, when projected onto a surface and viewed from the pose of the user, mitigates perceived differences relative to the base image. The base image (on which the transformed image is based) can optionally be generated in dependence on a distance of the user. Some implementations additionally or alternatively relate to dynamic adaptation of projection parameters (e.g., a location for projection, a size of projection) based on one or more properties of user(s) that are in an environment with the projector.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
identifying a plurality of users that are in an environment with a computing device, where an automated assistant client is associated with the computing device;
determining, using sensor data from at least one sensor accessible to the computing device, that a given user, of the plurality of users, is an active user for the automated assistant client;
determining a pose of the given user, wherein determining the pose is based on the sensor data or additional sensor data from at least one additional sensor accessible to the computing device;
identifying, by the automated assistant client, a base image for rendering to the given user via one or more output devices accessible to the automated assistant client; and
based on the determining the given user is the active user, tailoring rendering of output to the pose of the given user, the output including the base image or a transformed image that is a transformation of the base image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more output devices accessible to the automated assistant client is a projector, and further comprising:
causing, based on one or more output parameters, the projector to project the output onto a surface.

### Claim 3 (depends on 1)

The method of claim 1, wherein the at least one sensor accessible to the computing device includes a presence sensor, and wherein determining, using the sensor data from the at least one sensor accessible to the computing device, that the given user is the active user for the automated assistant client comprises:
identifying gesture data captured using the presence sensor; and
determining the active user for the automated assistant based on the gesture data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the sensor data or the additional sensor data from at least one additional sensor accessible to the computing device is gaze data that identifies the gaze of the given user, and wherein determining the pose of the active user comprises:
determining the pose of the active user based on the gaze data.

### Claim 5 (depends on 4)

The method of claim 4, wherein, based on the determining the given user is the active user, tailoring rendering of output to the pose of the given user, the output including a base image or a transformed image that is a transformation of the base image comprises:
tailoring rendering of the output to the pose of the given user, based on the gaze data that identifies the gaze of the given user.

### Claim 6

A computing device comprising,
one or more processors,
one or more sensors; and
memory configured to store instructions that, when executed by the one or more processors cause the one or more processors to perform operations that include:
identifying a plurality of users that are in an environment with the computing device, where an automated assistant client is associated with the computing device;
determining, using sensor data from at least a given sensor, of the one or more sensors, that a given user is an active user for the automated assistant client;
determining a pose of the given user, wherein determining the pose is based on the sensor data or additional sensor data from at least one additional sensor of the one or more sensors;
identifying, by the automated assistant client, a base image for rendering to the given user via one or more output devices accessible to the automated assistant client; and
based on the determining the given user is the active user, tailoring rendering of output to the pose of the given user, the output including a base image or a transformed image that is a transformation of the base image.

### Claim 7 (depends on 6)

The computing device of claim 6, wherein the one or more output devices accessible to the automated assistant client is a projector, and wherein the operations further include:
causing, based on one or more output parameters, the projector to project the output onto a surface.

### Claim 8 (depends on 5)

The computing device of claim 5, wherein the at least one sensor accessible to the computing device includes a presence sensor, and wherein determining, using the sensor data from the at least the given sensor accessible to the computing device, that the given user is the active user for the automated assistant client comprises:
identifying gesture data captured using the presence sensor; and
determining the active user for the automated assistant based on the gesture data.

### Claim 9 (depends on 5)

The computing device of claim 5, wherein the sensor data or the additional sensor data from at least one additional sensor accessible to the computing device is gaze data, identifying a gaze of the given user, and wherein determining the pose of the given user comprises:
determining the pose of the active user based on the gaze data.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein, based on the determining of the given user, tailoring rendering of output to the pose of the given user, the output including a base image or a transformed image that is a transformation of the base image comprises:
tailoring rendering of the output to the pose of the given user, based on the gaze data identifying the gaze of the given user.

### Claim 11

A non-transitory computer readable storage medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include:
identifying a plurality of users that are in an environment with a computing device, where an automated assistant client is associated with the computing device;
determining, using sensor data from at least one sensor accessible to the computing device, that a given user, of the plurality of users, is an active user for the automated assistant client;
determining a pose of the given user, wherein determining the pose is based on the sensor data or additional sensor data from at least one additional sensor accessible to the computing device;
identifying, by the automated assistant client, a base image for rendering to the given user via one or more output devices accessible to the automated assistant client; and
based on the determining the given user is the active user, tailoring rendering of output to the pose of the given user, the output including the base image or a transformed image that is a transformation of the base image.

### Claim 12 (depends on 12)

The non-transitory computer readable storage medium of claim 12, wherein the one or more output devices accessible to the automated assistant client is a projector, and wherein the one or more operations further include:
causing, based on one or more output parameters, the projector to project the output onto a surface.

### Claim 13 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the at least one sensor accessible to the computing device includes a presence sensor, and wherein determining, using the sensor data from the at least one sensor accessible to the computing device, that the given user is the active user for the automated assistant client comprises:
identifying gesture data captured using the presence sensor; and
determining the active user for the automated assistant based on the gesture data.

### Claim 14 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the sensor data or the additional sensor data from at least one additional sensor accessible to the computing device is gaze data that identifies the gaze of the given user, and wherein determining the pose of the user comprises:
determining the pose of the active user based on the gaze data.

### Claim 15 (depends on 14)

The non-transitory computer readable storage medium of claim 14, wherein, based on the determining of the given user is the active user, tailoring rendering of output to the pose of the given user, the output including a base image or a transformed image that is a transformation of the base image comprises:
tailoring rendering of the output to the pose of the given user, based on the gaze data identifying the gaze of the given user.
