# US 2021/0233320 A1 · Virtual item placement system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233320 A1
- Document ID: US20210233320A1
- Status: Published application
- Kind: A1
- Application number: 17/228,802
- Filed: 2021-04-13
- Published: 2021-07-29
- Earliest claimed: 2018-09-11
- Assignees: Houzz, Inc.
- Applicants: Houzz, Inc.
- Inventors: Shay Rom; Eli Konky
- Web page: https://patentgrep.com/patent/US20210233320A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06T 15/005: {General purpose rendering architectures}
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 30/13: Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
- G06T 15/04: Texture mapping
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2210/04: Architectural design, interior design
- G06T 2215/16: Using real world measurements to influence rendering
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A virtual model placement system can generate virtual floors and virtual walls to correspond to a real-world environment. Virtual items such as doors, windows, lamps, can be placed on the generated virtual walls and the system can generate dynamic updates of the placed items in response to movement by a user's device (e.g., a smartphone moving from right to left). The virtual items can be initially placed as primitives and then rendered to new positions upon the user selecting a render or lock instruction.

## Claims

### Claim 1

A method comprising:
generating, by a client device, an image sequence of a physical environment using an image sensor of the client device;
generating, on the client device, a virtual plane that coincides with a physical surface of the physical environment;
projecting a virtual item onto the physical surface by anchoring a side of the virtual item on the generated virtual plane in the image sequence;
detecting movement using one or more inertial sensors of the client device; and
in response to the detected movement, moving the projection of the virtual item on the physical surface of the image sequence by sliding the virtual item along the virtual plane while the side of the virtual item is anchored to the virtual plane.

### Claim 2 (depends on 1)

The method of claim 1, wherein the projection is moved to a new position, and wherein the method further comprises:
receiving a render instruction after moving the projection in response to the detected movement; and
in response to receiving the render instruction, rendering using the virtual item to generate a virtual item render having realistic surface texture and lighting at the new position in the image sequence.

### Claim 3 (depends on 1)

The method of claim 1, wherein the virtual item is a virtual panel that is smaller than the virtual plane.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the virtual plane comprises:
receiving one or more point placements on a virtual ground detected in the image sequence, wherein the virtual plane is generated to coincide with the one or more point placements.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating the virtual plane to coincide with the one or more point placements includes generating the virtual plane at the point placements and setting the virtual plane as orthogonal to the virtual ground detected in the image sequence.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtual plane is transparent, and the virtual item is visible in the image sequence.

### Claim 7 (depends on 6)

The method of claim 6, wherein the virtual item is a virtual door that is anchored to a bottom side of the virtual plane such that updating the projection of the virtual item simulates the virtual door sliding along the bottom side of the physical surface as viewed through the image sequence on the client device.

### Claim 8 (depends on 7)

The method of claim 7, wherein the bottom side of the virtual plane is an edge of the virtual plane, and a virtual door side of the virtual door is anchored to the edge of the virtual plane so as to slide along the edge of the virtual plane.

### Claim 9 (depends on 7)

The method of claim 7, wherein the side of a virtual plane is defined as an intersection of the virtual plane with a virtual floor generated by the client device.

### Claim 10 (depends on 9)

The method of claim 9, the method further comprising:
detecting a physical ground of the physical environment depicted in the image sequence by detecting a plurality of image features of the physical ground;
generating the virtual floor to correspond to the physical ground;
determining an intersection of the virtual floor and the virtual plane; and
anchoring the virtual door to the intersection of the virtual floor and the virtual plane.

### Claim 11 (depends on 1)

The method of claim 1, wherein the virtual item is a mesh of a model of a vertical virtual item.

### Claim 12 (depends on 1)

The method of claim 1, wherein the virtual item is a simplified geometric shape of a vertical virtual item.

### Claim 13 (depends on 1)

The method of claim 1, wherein the virtual item is a collection of vertices connected by lines that outline a shape of a vertical virtual item.

### Claim 14

A system comprising:
one or more processors of a machine;
an image sensor;
one or more inertial sensors;
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
generating an image sequence of a physical environment using the image sensor;
generating a virtual plane that coincides with a physical surface of the physical environment;
projecting a virtual item onto the physical surface by anchoring a side of the virtual item on the generated virtual plane in the image sequence;
detecting movement using one or more inertial sensors; and
in response to the detected movement, moving the projection of the virtual item on the physical surface of the image sequence by sliding the virtual item along the virtual plane while the side of the virtual item is anchored to the virtual plane.

### Claim 15 (depends on 14)

The system of claim 14, wherein the projection is moved to a new position, and wherein the operations further comprise:
receiving a render instruction after moving the projection in response to the detected movement; and
in response to receiving the render instruction, rendering using the virtual item to generate a virtual item render having realistic surface texture and lighting at the new position in the image sequence.

### Claim 16 (depends on 14)

The system of claim 14, wherein the virtual item is a virtual panel that is smaller than the virtual plane.

### Claim 17 (depends on 14)

The system of claim 14, wherein generating the virtual plane comprises:
receiving one or more point placements on a virtual ground detected in the image sequence, wherein the virtual plane generated to coincide with the one or more point placements.

### Claim 18 (depends on 17)

The system of claim 17, wherein generating the virtual plane to coincide with the one or more point placements includes generating the virtual plane at the point placements and setting the virtual plane as orthogonal to the virtual ground detected in the image sequence.

### Claim 19 (depends on 14)

The system of claim 14, wherein the virtual plane is transparent, and the virtual item is visible in the image sequence.

### Claim 20

A machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating an image sequence of a physical environment using an image sensor;
generating a virtual plane that coincides with a physical surface of the physical environment;
projecting a virtual item onto the physical surface by anchoring a side of the virtual item on the generated virtual plane in the image sequence;
detecting movement using one or more inertial sensors; and
in response to the detected movement, moving the projection of the virtual item on the physical surface of the image sequence by sliding the virtual item along the virtual plane while the side of the virtual item is anchored to the virtual plane.
