# US 2023/0408832 A1 · Augmented Reality Content Creation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0408832 A1
- Document ID: US20230408832A1
- Status: Published application
- Kind: A1
- Application number: 18/240,307
- Filed: 2023-08-30
- Published: 2023-12-21
- Earliest claimed: 2015-02-25
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Brian Mullins; Matthew Kammerait
- Web page: https://patentgrep.com/patent/US20230408832A1

## CPC

- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for generating a virtual content to a physical object is described. The system accesses a video that was captured by a camera. The video depicts a user performing a physical operation in relation to a physical object. The system determines metadata identifying positions of the camera relative to the physical object during the capturing of the video, identifies feature points of the physical object in the video, and retrieves a virtual object based on the feature points of the physical object. The system generates virtual content based on the video, the metadata, and the virtual object, and stores the object identifier and the virtual content relationally in a data storage that stores multiple sets of virtual content associated with multiple physical objects and their corresponding object identifiers.

## Claims

### Claim 1

A computing device comprising:
one or more processors; and
a non-transitory storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
access a video that was captured by a camera, the video depicting a user performing a physical operation in relation to a physical object, the physical object associated with an object identifier;
determine metadata identifying positions of the camera relative to the physical object during the capturing of the video;
identify feature points of the physical object in the video;
retrieve a virtual object based on the feature points of the physical object;
generate virtual content based on the video, the metadata, and the virtual object; and
store the object identifier and the virtual content relationally in a data storage that stores a plurality of sets of virtual content associated with a plurality of physical objects and corresponding object identifiers.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein an operation on the physical object is recorded by the camera in response to at least a verbal recording instruction from a user of the camera to form the virtual content.

### Claim 3 (depends on 1)

The computing device of claim 1, wherein the metadata further comprise at least an orientation of the camera relative to a physical object, an operational status of the physical object, or a temperature range related to the physical object.

### Claim 4 (depends on 1)

The computing device of claim 1, wherein the instructions further cause the one or more processors to:
receive an image of the physical object from a client device;
identify the object identifier based on the image of the physical object;
retrieve the virtual content based on the object identifier;
cause the virtual content to be displayed at a display of the client device.

### Claim 5 (depends on 4)

The computing device of claim 4, wherein the image of the physical object includes a quick response (QR) code, the QR code corresponding to the object identifier of the physical object.

### Claim 6 (depends on 5)

The computing device of claim 5, wherein the instructions further cause the one or more processors to:
extract the QR code from the image of the physical object; and
identify the object identifier based on the QR code.

### Claim 7 (depends on 1)

The computing device of claim 1, wherein physical object is a machine of a particular type, the operation of the physical operation in relation to the physical object includes an operation of the machine, and the virtual content includes a video illustrating an operation of the particular type of machine.

### Claim 8 (depends on 1)

The computing device of claim 1, wherein physical object is a machine, and the operation of the physical operation in relation to the physical object includes an operation of the machine.

### Claim 9 (depends on 1)

The computing device of claim 1, wherein the instructions further cause the one or more processors to:
determine a geographic location of the camera;
access a library containing information describing geographic locations of a plurality of physical objects and respective object identifiers; and
determine an object identifier based on the geographic location of the camera and the library.

### Claim 10

A computer implemented method, the method comprising:
accessing a video that was captured by a camera, the video depicting a user performing a physical operation in relation to a physical object, the physical object associated with an object identifier;
determining metadata identifying positions of the camera relative to the physical object during the capturing of the video;
identifying feature points of the physical object in the video;
retrieving a virtual object based on the feature points of the physical object;
generating virtual content based on the video, the metadata, and the virtual object; and
storing the object identifier and the virtual content relationally in a data storage that stores a plurality of sets of virtual content associated with a plurality of physical objects and corresponding object identifiers.

### Claim 11 (depends on 10)

The method of claim 10, wherein an operation on the physical object is recorded by the camera in response to at least a verbal recording instruction from a user of the camera to form the virtual content.

### Claim 12 (depends on 10)

The method of claim 10, wherein the metadata further comprise at least an orientation of the camera relative to a physical object, an operational status of the physical object, or a temperature range related to the physical object.

### Claim 13 (depends on 10)

The method of claim 10, the method further comprising:
receiving an image of the physical object from a client device;
identifying the object identifier based on the image of the physical object;
retrieving the virtual content based on the object identifier;
causing the virtual content to be displayed at a display of the client device.

### Claim 14 (depends on 13)

The method of claim 13, wherein the image of the physical object includes a quick response (QR) code, the QR code corresponding to the object identifier of the physical object.

### Claim 15 (depends on 14)

The method of claim 14, the method further comprising:
extracting the QR code from the image of the physical object; and
identifying the object identifier based on the QR code.

### Claim 16 (depends on 10)

The method of claim 10, wherein physical object is a machine of a particular type, the operation of the physical operation in relation to the physical object includes an operation of the machine, and the virtual content includes a video illustrating an operation of the particular type of machine.

### Claim 17 (depends on 10)

The method of claim 10, wherein physical object is a machine, and the operation of the physical operation in relation to the physical object includes an operation of the machine.

### Claim 18 (depends on 10)

The method of claim 10, the method further comprising:
determining a geographic location of the camera;
accessing a library containing information describing geographic locations of a plurality of physical objects and respective object identifiers; and
determining an object identifier based on the geographic location of the camera and the library.

### Claim 19

A computer program product, comprising a non-transitory storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
access a video that was captured by a camera, the video depicting a user performing a physical operation in relation to a physical object, the physical object associated with an object identifier;
determine metadata identifying positions of the camera relative to the physical object during the capturing of the video;
identify feature points of the physical object in the video;
retrieve a virtual object based on the feature points of the physical object;
generate virtual content based on the video, the metadata, and the virtual object; and
store the object identifier and the virtual content relationally in a data storage that stores a plurality of sets of virtual content associated with a plurality of physical objects and corresponding object identifiers.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the instructions further cause the one or more processors to:
receive an image of the physical object from a client device;
identify the object identifier based on the image of the physical object;
retrieve the virtual content based on the object identifier;
cause the virtual content to be displayed at a display of the client device.
