# US 2021/0279466 A1 · Virtualization of tangible interface objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279466 A1
- Document ID: US20210279466A1
- Status: Published application
- Kind: A1
- Application number: 16/938,913
- Filed: 2020-07-25
- Published: 2021-09-09
- Earliest claimed: 2012-10-15
- Assignees: Tangible Play, Inc.
- Applicants: Tangible Play, Inc.
- Inventors: Pramod Kumar Sharma; Jerome Scholler
- Web page: https://patentgrep.com/patent/US20210279466A1

## CPC

- G06V 20/20: in augmented reality scenes
- G06F 1/1632: {External expansion units, e.g. docking stations}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0317: {in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 23/51: Housings
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An example system includes a computing device located proximate to a physical activity surface, a video capture device, and a detector. The video capture device is coupled for communication with the computing device and is adapted to capture a video stream that includes an activity scene of the physical activity surface and one or more interface objects physically intractable with by a user. The detector processes the video stream to detect the one or more interface objects included in the activity scene, to identify the one or more interface objects that are detectable, to generate one or more events describing the one or more interface objects, and to provide the one or more events to an activity application configured to render virtual information on the one or more computing devices based on the one or more events.

## Claims

### Claim 1

(canceled)

### Claim 2

A method comprising:
capturing, using a rear facing camera of a computing device, a video stream of a field of view of the rear facing camera, the video stream including an activity scene and a tangible interface object present in the activity scene;
processing, using a processor of a computing device, the video stream for the tangible interface object and an attribute of the tangible interface object;
determining, using the processor of the computing device, an identity of the tangible interface object using the attribute of the tangible interface object;
generating, using the processor of the computing device, a virtual scene based on the attribute of the tangible interface object; and
rending, in a display of the computing device, the virtual scene in a region of the display.

### Claim 3 (depends on 2)

The method of claim 2, wherein the field of view of the rear facing camera captures substantially only the activity scene.

### Claim 4 (depends on 2)

The method of claim 2, wherein the field of view of the rear facing camera is redirected towards a front side of the computing device using a camera adapter.

### Claim 5 (depends on 4)

The method of claim 4, wherein the activity scene is located in front of the computing device and wherein the camera adapter redirects the field of view of the rear facing camera by wrapping light reflected off of the activity scene into the rear facing camera.

### Claim 6 (depends on 4)

The method of claim 4, wherein the camera adapter defines a side of the activity scene being captured.

### Claim 7 (depends on 2)

The method of claim 2, wherein the video stream of the field of view of the rear facing camera is a first video stream, the virtual scene is a first virtual scene, and the region is a first region, the method further comprising:
capturing, using a front facing video capture device, a second video stream of a field of the front facing camera, the second video stream capturing a portion of an activity surface situated in front of the computing device;
generating, using the processor of the computing device, a second virtual scene based on the second video stream of the field of view of the front facing camera; and
rendering, the second virtual scene in a second region of the display, the second region of the display being separate from the first region of the display.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second video stream of the field of view of the front facing camera also captures a user scene that includes a user situated in front of the computing device.

### Claim 9 (depends on 8)

The method of claim 8, wherein the second video stream of the field of view is a split field of view that splits the field of view into a first field of view that captures the user scene and a second field of view that captures a portion of the activity surface situated in front of the computing device.

### Claim 10

A physical activity surface visualization system comprising:
a stand configured to position a computing device having one or more processors;
a rear facing camera configured to capture a video stream of a field of view of the rear facing camera, the video stream including an activity scene and a tangible interface object present in the activity scene;
a detector executable by the one or more processors to detect the tangible interface object and an attribute of the tangible interface object in the video stream; and
an activity application executable by the one or more processors to determine an identity of the tangible interface object using the attribute of the tangible interface object, generate a virtual scene based on the attribute of the tangible interface object; and
a display configured to render the virtual scene in a region of the display.

### Claim 11 (depends on 10)

The physical activity surface visualization system of claim 10, wherein the field of view of the rear facing camera captures substantially only the activity scene.

### Claim 12 (depends on 10)

The physical activity surface visualization system of claim 10, wherein the field of view of the rear facing camera is redirected towards a front side of the computing device using a camera adapter.

### Claim 13 (depends on 12)

The physical activity surface visualization system of claim 12, wherein the activity scene is located in front of the computing device and wherein the camera adapter redirects the field of view of the rear facing camera by wrapping light reflected off of the activity scene into the rear facing camera.

### Claim 14 (depends on 13)

The physical activity surface visualization system of claim 13, wherein the camera adapter defines a side of the activity scene being captured.

### Claim 15 (depends on 10)

The physical activity surface visualization system of claim 10, wherein the video stream of the field of view of the rear facing camera is a first video stream, the virtual scene is a first virtual scene, and the region is a first region, the system further comprising:
a front facing camera configured to capture a second video stream of a field of view of the front facing camera, the second video stream capturing a portion of an activity surface situated in front of the computing device, and
wherein the activity application is further configured to generate a second virtual scene based on the second video stream of the field of view of the front facing camera, and
wherein the display is further configured to render the second virtual scene in a second region of the display.

### Claim 16 (depends on 15)

The physical activity surface visualization system of claim 15, wherein the second video stream of the field of view of the front facing camera also captures a user scene that includes a user situated in front of the computing device.

### Claim 17 (depends on 16)

The physical activity surface visualization system of claim 16, wherein the second video stream of the field of view is a split field of view that splits the field of view into a first field of view that captures the user scene and a second field of view that captures a portion of the activity surface situated in front of the computing device.

### Claim 18

A method comprising:
capturing, using a rear facing camera of a computing device, a first video stream of a field of view of the rear facing camera;
capturing, using a front facing camera of the computing device, a second video stream of a field of view of the front facing camera;
processing, using a processor of the computing device, the first video stream and the second video stream for a tangible interface objects present in one of the first video stream and the second video stream;
detecting, using the processor of the computing device, an attribute of the tangible interface object;
determining, using, the processor of the computing device, an identity of the tangible interface object using the attribute;
generating, using the processor of the computing device, a virtual object based on the identity of the tangible interface object;
rendering, using the processor of the computing device, a first virtual scene based on the first video stream and a second virtual scene based on the second video stream; and
presenting, in a display of the computing device, the first virtual scene in a first region, the second virtual scene in a second region, and the virtual object.

### Claim 19 (depends on 18)

The method of claim 18, wherein the tangible interface object is present in the first video stream and wherein presenting, the virtual object further comprises:
presenting using the processor of the computing device, the virtual object within the second virtual scene.

### Claim 20 (depends on 18)

The method of claim 18, wherein the field of view of the rear facing camera is redirected towards an activity surface located in front of the computing device using a camera adapter.

### Claim 21 (depends on 20)

The method of claim 20, wherein the camera adapter redirects the field of view of the rear facing camera by wrapping light reflected off of the activity surface into the rear facing camera.