# US 12,112,441 B2 · Content transformations based on reflective object recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,441 B2
- Document ID: US12112441B2
- Status: Granted
- Kind: B2
- Application number: 18/214,575
- Filed: 2023-06-27
- Published: 2024-01-04
- Granted: 2024-10-08
- Earliest claimed: 2022-06-30
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Yutaka Yokokawa; Devin W. Chalmers; Brian W. Temple; Rahul Nair; Thomas G. Salter
- Web page: https://patentgrep.com/patent/US12112441B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06T 7/75: {involving models}
- G06V 20/20: in augmented reality scenes
- G06V 20/50: Context or environment of the image
- G06V 40/161: {Detection; Localisation; Normalisation}
- H04S 7/303: {Tracking of listener position or orientation}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30201: Face
- H04R 2420/07: Applications of wireless loudspeakers or wireless microphones
- H04S 7/304: {For headphones}
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04S: STEREOPHONIC SYSTEMS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Various implementations disclosed herein include devices, systems, and methods that present virtual content based on detecting a reflective object and determining a three-dimensional (3D) position of the reflective object in a physical environment. For example, an example process may include obtaining sensor data (e.g., image, sound, motion, etc.) from a sensor of an electronic device in a physical environment that includes one or more objects. The method may further include detecting a reflective object amongst the one or more objects based on the sensor data. The method may further include determining a 3D position of the reflective object in the physical environment (e.g., where the plane of the mirror is located). The method may further include presenting virtual content in a view of the physical environment. The virtual content may be positioned at a 3D location based on the 3D position of the reflective object.

## Claims

### Claim 1

A method comprising:
at an electronic device having a processor and a sensor:
obtaining sensor data from the sensor of the electronic device in a physical environment that includes one or more objects;
detecting a reflective surface of a reflective object amongst the one or more objects based on the sensor data and detecting a reflection of the electronic device upon the reflective surface of the reflective object;
determining a three-dimensional (3D) position of the reflective object in the physical environment based on determining a 3D position of the reflection of the electronic device; and
presenting virtual content within a view of an extended reality (XR) environment that includes a view of the physical environment, wherein the virtual content is positioned at a 3D location within the view of the XR environment based on the determined 3D position of the reflective object in the physical environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the 3D position of the reflective object in the physical environment comprises detecting a plane of the surface of the reflective object, and wherein the virtual content comprises one or more interactable elements that are presented on the detected plane of the surface of the reflective object.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the 3D position of the reflective object in the physical environment comprises
determining a mid-point location between the 3D position of the reflection of the electronic device and a 3D position of the electronic device.

### Claim 4 (depends on 1)

The method of claim 1, wherein, in accordance with the electronic device rotating about a vertical axis during a first period of time, determining the 3D position of the reflective object in the physical environment comprises:
detecting a reflection of a rotation of the electronic device upon the reflective surface of the reflective object;
determining 3D position data of the reflection of the rotation of the electronic device during the first period of time; and
determining the 3D position of the reflective object based on comparing 3D position data of the electronic device and the 3D position data of the reflection of the rotation of the electronic device during the first period of time.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determined 3D position of the reflective object in the physical environment is determined based on depth data from the sensor data and the 3D location of the virtual content within the view of the XR environment is based on the depth data associated with the 3D position of the reflective object.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the determined 3D position of the reflective object in the physical environment comprises a first distance from the electronic device;
the 3D location of the virtual content within the view of the XR environment is at a second distance from the electronic device that is greater than the first distance; and
presenting the virtual content within the view of the XR environment comprises presenting spatialized audio content at a perceived distance to a sound source based on the 3D location of the virtual content within the view of the XR environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the sensor data comprises images of a head of a user of the electronic device, and wherein detecting the reflective surface of the reflective object is based on determining that:
the head of the user as seen in the images is rotating about a vertical axis by an amount that is double an amount of rotation of the electronic device; or
the head of the user as seen in the images is not rotating about a forward axis.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
detecting a facial reflection in the reflective object amongst the one or more objects based on the sensor data, wherein detecting the facial reflection is based on tracking facial features of a user of the electronic device or facial recognition of the user.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
in accordance with detecting the reflective surface of the reflective object,
determining a context associated with a use of the electronic device in the physical environment based on the sensor data, and
presenting the virtual content within the view of the XR environment based on the context.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
the context comprises a time of day, and presenting the virtual content within the view of the XR environment is based on the time of day;
the context comprises movements of a user of the electronic device with respect to a reflection on the surface of the reflective object, and presenting the virtual content within the view of the XR environment is based on the movements of the user; or
the context comprises a user interaction with the reflection, and presenting the virtual content within the view of the XR environment is based on the user interaction with the reflection.

### Claim 11 (depends on 9)

The method of claim 9, wherein determining the context comprises:
determining use of the electronic device in a new location;
determining use of the electronic device during a type of activity; or
determining that the electronic device is within a proximity threshold distance of a location, an object, another electronic device, or a person.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
determining a scene understanding of the physical environment based on the sensor data;
determining, based on the scene understanding, that a user of the electronic device is an only user within an area associated with the view of the XR environment; and
presenting the virtual content within the view of the XR environment based on a user preference setting associated with the user being the only user within the area associated with the view of the physical environment.

### Claim 13 (depends on 1)

The method of claim 1, wherein the virtual content comprises a visual representation of another user based on a communication session with another electronic device.

### Claim 14 (depends on 1)

The method of claim 1, wherein a depth position of the 3D location of the virtual content within the view of the XR environment is the same as a depth of a reflected object detected in the reflection.

### Claim 15 (depends on 1)

The method of claim 1, wherein the electronic device is a head-mounted device (HMD).

### Claim 16

A device comprising:
a sensor;
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
obtaining sensor data from the sensor of the device in a physical environment that includes one or more objects;
detecting a reflective surface of a reflective object amongst the one or more objects based on the sensor data and detecting a reflection of the device upon the reflective surface of the reflective object;
determining a three-dimensional (3D) position of the reflective object in the physical environment based on determining a 3D position of the reflection of the device; and
presenting virtual content within a view of an extended reality (XR) environment that includes a view of the physical environment, wherein the virtual content is positioned at a 3D location within the view of the XR environment based on the determined 3D position of the reflective object in the physical environment.

### Claim 17 (depends on 16)

The device of claim 16, wherein determining the 3D position of the reflective object in the physical environment comprises detecting a plane of the surface of the reflective object, and wherein the virtual content comprises one or more interactable elements that are presented on the detected plane of the surface of the reflective object.

### Claim 18 (depends on 16)

The device of claim 16, wherein determining the 3D position of the reflective object in the physical environment comprises:
determining a mid-point location between the 3D position of the reflection of the device and a 3D position of the device.

### Claim 19 (depends on 16)

The device of claim 16, wherein the sensor data comprises images of a head of a user of the device, and wherein detecting the reflective surface of the reflective object is based on determining that:
the head of the user as seen in the images is rotating about a vertical axis by an amount that is double an amount of rotation of the device; or
the head of the user as seen in the images is not rotating about a forward axis.

### Claim 20

A non-transitory computer-readable storage medium, storing computer-executable program instructions on a device to perform operations comprising:
obtaining sensor data from the sensor of an electronic device in a physical environment that includes one or more objects;
detecting a reflective surface of a reflective object amongst the one or more objects based on the sensor data and detecting a reflection of the electronic device upon the reflective surface of the reflective object;
determining a three-dimensional (3D) position of the reflective object in the physical environment based on determining a 3D position of the reflection of the electronic device; and
presenting virtual content within a view of an extended reality (XR) environment that includes a view of the physical environment, wherein the virtual content is positioned at a 3D location within the view of the XR environment based on the determined 3D position of the reflective object in the physical environment.

### Claim 21 (depends on 20)

The non-transitory computer-readable storage medium of claim 20, wherein determining the 3D position of the reflective object in the physical environment comprises detecting a plane of the surface of the reflective object, and wherein the virtual content comprises one or more interactable elements that are presented on the detected plane of the surface of the reflective object.

### Claim 22 (depends on 20)

The non-transitory computer-readable storage medium of claim 20, wherein determining the 3D position of the reflective object in the physical environment comprises:
determining a mid-point location between the 3D position of the reflection of the electronic device and a 3D position of the electronic device.

### Claim 23 (depends on 20)

The non-transitory computer-readable storage medium of claim 20, wherein the sensor data comprises images of a head of a user of the electronic device, and wherein detecting the reflective surface of the reflective object is based on determining that:
the head of the user as seen in the images is rotating about a vertical axis by an amount that is double an amount of rotation of the electronic device; or
the head of the user as seen in the images is not rotating about a forward axis.