# US 2022/0277484 A1 · Software Engine Enabling Users to Interact Directly with a Screen Using a Camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277484 A1
- Document ID: US20220277484A1
- Status: Published application
- Kind: A1
- Application number: 17/682,512
- Filed: 2022-02-28
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: Subversus Interactive LLC
- Applicants: Subversus Interactive LLC
- Inventors: Vishal SRIVASTAVA
- Web page: https://patentgrep.com/patent/US20220277484A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/75: {involving models}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03545: {Pens or stylus}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06T 2207/30196: Human being; Person
- G06T 2207/30204: Marker
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The technology of this application relates to a software engine that enables a user to interact with a display (e.g., television) by using an image capture device (e.g., camera). In one non-limiting example, a user can operate a device (such as a mobile phone) to run an application that interfaces with the software engine, so that the device's camera can operate in unison with a separate display to enable unique types of user interactions. For example, the user may use such an application to tap a user interface button or draw a shape with their hand, directly on the surface of a larger display (e.g., television), in view of the mobile phone's camera; the application's visual state would update in response to the user's captured gestures and to the position and orientation of the mobile phone. In another non-limiting example, a user, while playing a game utilizing the software engine, swings a device (such as a mobile phone) in free space, to control the position and orientation of a paddle within the three dimensional scene of the game, presented on the larger display.

## Claims

### Claim 1

A system, comprising:
an image capture device; and
processing circuitry having at least one processor and at least one memory, wherein the processing circuitry is configured to:
capture image data associated with a target using the image capture device;
detect and track features in the captured image data associated with the target, wherein the target includes a surface;
determine a position and an orientation of the image capturing device with respect to the surface, using the detected features associated with at least a portion of the target; and
generate output data, using the determined position and orientation of the image capturing device with respect to the surface, to enable user interaction.

### Claim 2 (depends on 1)

The system of claim 1, wherein the surface includes a screen.

### Claim 3 (depends on 1)

The system of claim 1, wherein the image capture device is operable as a controller in a three-dimensional space.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processing circuitry is further configured to:
detect and track one or more recognizable markers in the captured image data;
track the surface from the captured image data;
unproject the one or more markers onto or near the surface; and
generate output data, using the one or more unprojected markers, to enable further user interaction.

### Claim 5 (depends on 4)

The system of claim 4, wherein the plane of the surface is positioned behind, with respect to a viewpoint of the image capture device, the one or more markers.

### Claim 6 (depends on 4)

The system of claim 4, wherein unprojecting the one or more markers includes translating the one or more markers in a three-dimensional scene along a ray from a viewpoint of the image capture device.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processing circuitry is further configured to:
extract an occluding object image from the captured image data, wherein the occluding object image includes a real world occluding object; and
generate output data, using the occluding object image, to enable further user interaction.

### Claim 8 (depends on 7)

The system of claim 7, wherein the processing circuitry is further configured to generate image data associated with the occluding object, and wherein the plane of the surface is positioned behind, with respect to the viewpoint of the image capture device, the occluding object.

### Claim 9 (depends on 2)

The system of claim 2, wherein the processing circuitry is further configured to display a tracking image, or a region of a tracking image, at least some of the time, on the screen, such that the displayed tracking image is the target that is tracked by the system.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processing circuitry is further configured to calculate a position and an orientation of the image capture device relative to the screen.

### Claim 11 (depends on 4)

The system of claim 4, wherein one or more detected markers are parts of a human body.

### Claim 12 (depends on 1)

The system of claim 1, wherein the generated output data includes a three-dimensional model of a robot whose position, orientation, and body pose mimic the user's own, relative to the surface.

### Claim 13

A method for generating a virtual environment, the method comprising:
capturing image data associated with a target using an image capture device;
determining a position and an orientation of the image capture device with respect to the captured image, using at least a portion of the target; and
generating an image for display in association with the determined position and orientation of the image capture device relative to the target.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
generating, for display, an image including the virtual environment;
generating, for display, a virtual object in the virtual environment in the displayed image; and
controlling the virtual object based on the determined position and orientation of the image capture device relative to the target.

### Claim 15 (depends on 14)

The method of claim 14, wherein the virtual object is controlled in three dimensions based on movement of the image capture device in three dimensions.

### Claim 16 (depends on 13)

The method of claim 13, wherein the determined position and orientation are output from a software engine.

### Claim 17

A system, comprising:
a processor; and
a memory configured to store computer readable instructions that, when executed by the processor, cause the system to:
detect and track features in image data captured by an image capture device, wherein the features are associated with a target, and the target includes a surface;
detect and track a recognizable real world object positioned between the image capture device and the surface;
determine a position and an orientation of the real world object with respect to the surface and/or with respect to the image capture device, using the detected features associated with real world object; and
generate output data, using the determined position and orientation, to enable user interaction.

### Claim 18 (depends on 17)

The system of claim 17, wherein the surface includes a screen.

### Claim 19 (depends on 17)

The system of claim 17, wherein the processing circuitry is further configured to:
detect and track one or more recognizable markers in the captured image data;
track the surface from the captured image data; and
unproject the one or more markers onto or near the surface; and
generate output data, using the one or more unprojected markers, to enable further user interaction.

### Claim 20 (depends on 17)

The system of claim 17, wherein the processing circuitry is further configured to display a tracking image, or a region of a tracking image, at least some of the time, on the screen, such that the displayed tracking image is the target that is tracked by the system.
