# US 2023/0156314 A1 · Gaze-based camera auto-capture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0156314 A1
- Document ID: US20230156314A1
- Status: Published application
- Kind: A1
- Application number: 18/053,280
- Filed: 2022-11-07
- Published: 2023-05-18
- Earliest claimed: 2021-11-15
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Sebastian Sztuk; Salvael Ortega Estrada; Sapna Shroff; Jun Hu; Yilei Li
- Web page: https://patentgrep.com/patent/US20230156314A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 18/2323: based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/48: using social graphs
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 20/20: in augmented reality scenes
- G06V 40/193: {Preprocessing; Feature extraction}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/675: {comprising setting of focusing regions}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for capturing a scene in a virtual environment for an immersive reality application running in a headset is provided. The method includes determining initiation of an auto-capture session in a headset by a user, the headset running an immersive reality application hosted by a remote server, executing a gaze model based on the initiation, detecting through the gaze model a gaze of the user, tracking the gaze of the user, capturing a scene in a virtual environment based on the gaze of the user, and storing the scene as a media file in storage. A headset and a memory storing instructions to cause the headset and a remote server to perform the above method are also provided.

## Claims

### Claim 1

A computer-implemented method for capturing a scene in a virtual environment, comprising:
determining initiation of an auto-capture session in a headset by a user, the headset running an immersive reality application hosted by a remote server;
executing a gaze model based on the initiation;
detecting through the gaze model a gaze of the user;
tracking the gaze of the user;
capturing a scene in a virtual environment based on the gaze of the user; and
storing the scene as a media file in storage.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein executing the gaze model comprises detecting that the gaze of the user is longer than a pre-selected threshold.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising initiating a next level of a confirmation model to confirm that there is a meaningful object in the gaze of the user.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising automatically initiating the capturing of the scene.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein tracking the gaze of the user comprises identifying an object that is a target in the gaze of the user.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein capturing the scene in the virtual environment comprises identifying an object in the scene and verifying a privacy setting of the object in a user account.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising performing auto-focusing and/or auto-zooming for an object that is a target in the gaze of the user.

### Claim 8

A system configured for relaying a message through a social network, the system comprising:
a one or more processors configured by machine-readable instructions to cause the system to:
determine an initiation of an auto-capture session by a user of a headset, the user being a subscriber of the social network;
execute a gaze model based on the initiation;
detect through the gaze model a gaze of the user;
track the gaze of the user;
capture a scene in a virtual environment based on the gaze of the user; and
store the scene as a media file in storage.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to detect that the gaze of the user is longer than a threshold and to confirm that there is a meaningful object in the gaze of the user.

### Claim 10 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to automatically initiate a capture of the scene.

### Claim 11 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to track an object in the gaze of the user.

### Claim 12 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to identify an object in the scene and to verify a privacy setting of the object in a user account.

### Claim 13 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to identify a person in the scene and verifying a content setting of the person in the social network.

### Claim 14 (depends on 8)

The system of claim 8, wherein the one or more processors further cause the system to perform an auto-focus and/or an auto-zoom for an object that is a target in the gaze of the user.

### Claim 15

A non-transient, computer-readable storage medium having instructions which, when executed by a processor, cause a computer to:
determine initiation of an auto-capture session in a headset by a user, the headset running an immersive reality application hosted by a remote server;
execute a gaze model based on the initiation;
detect through the gaze model a gaze of the user;
track the gaze of the user;
capture a scene in a virtual environment based on the gaze of the user; and
store the scene as a media file in storage.

### Claim 16 (depends on 15)

The non-transient, computer-readable storage medium of claim 15, wherein to execute the gaze model, the processor further executes instructions to cause the computer to detect that the gaze of the user is longer than a threshold.

### Claim 17 (depends on 15)

The non-transient, computer-readable storage medium of claim 15, wherein the processor further executes instructions to cause the computer to verify that there is a meaningful object in the gaze of the user.

### Claim 18 (depends on 15)

The non-transient, computer-readable storage medium of claim 15, wherein the processor further executes instructions to cause the computer to automatically initiate the capture of the scene automatically.

### Claim 19 (depends on 15)

The non-transient, computer-readable storage medium of claim 15, wherein the processor further executes instructions to cause the computer to track an object that is a target in the gaze of the user.

### Claim 20 (depends on 15)

The non-transient, computer-readable storage medium of claim 15, wherein to capture the scene in the virtual environment, the processor further executes instructions to cause the computer to identify an object in the scene and verifying a privacy setting of the object in a user account.
