US 10,096,168 B2Granted

Three-dimensional mixed-reality viewport

AssigneeMicrosoft Technology Licensing, LLC

InventorsAlexandre Da Veiga, Ben Sugden, Jon Christen

Life of this patent

Estimating the term…

Loading patent…

Abstract

An application running on a computing platform that employs three-dimensional (3D) modeling is extended using a virtual viewport into which 3D holograms are rendered by a mixed-reality head mounted display (HMD) device. The HMD device user can position the viewport to be rendered next to a real world 2D monitor and use it as a natural extension of the 3D modeling application. For example, the user can interact with modeled objects in mixed-reality and move objects between the monitor and the viewport. The 3D modeling application and HMD device are configured to exchange scene data for modeled objects (such as geometry, lighting, rotation, scale) and user interface parameters (such as mouse and keyboard inputs). The HMD device implements head tracking to determine where the user is looking so that user inputs are appropriately directed to the monitor or viewport.

Claims20

Dependent claims

A head mounted display (HMID) device operable by a user in a physical environment, comprising:

one or more processors;
a sensor package configured to dynamically provide sensor data used to determine a pose of the user's head in the physical environment;
a network connection configured to support an exchange of extensibility data between the HID device and a remote computing device, in which the remote computing device executes an application that renders a three-dimensional (3D) model on a two-dimensional (2D) desktop;
a see-through display configured for rendering a mixed-reality environment to the user, a view position of the user for the rendered mixed-reality environment being variable depending at least in part on the determined user head pose;
and
one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the HMID device to:
determine user head pose in the physical environment using the sensor data,
implement a 3D virtual viewport on the see-through display,
exchange extensibility data between the HMID device and the remote computing device, the extensibility data including scene data for the 3D model and user interface data for the 3D model, and wherein the extensibility data is configured to support multiple different rendering modes including 2D rendering on the desktop and 3D rendering on the viewport, in which the desktop and viewport each utilize the exchanged extensibility data to facilitate rendering of the 3D model in the different respective 2D rendering and 3D rendering supported by the desktop and the viewport, respectively,
selectively utilize one or more of the scene data or interface data to render the 3D model as a hologram in the viewport based on the view position,
enable user interaction with the hologram in the viewport, and
operate the see-through display based on the view position to enable user interaction with the 3D model on the desktop.

9 more elements

A computing device, comprising:

one or more processors;
an interface to a monitor, the monitor displaying a two-dimensional (2D) desktop;
a mouse interface for connecting to a mouse and receiving signals from the mouse indicating mouse movement and inputs to mouse controls from a user of the computing device;
a keyboard interface for connecting to a keyboard and receiving signals from the keyboard indicating keyboard inputs from the user;
a network interface for communicating with a remote head mounted display (HMD) device over a network connection, the HMD device being configured to support a three-dimensional (3D) virtual viewport;
and
one or more memory devices storing computer-readable instructions which, when executed by the one or more processors implement a three-dimensional (3D) modeling application and a user interface (UI) server configured for
tracking mouse messages that describe the mouse movements and inputs,
tracking keyboard messages that describe the keyboard inputs,
when a mouse movement indicates that a cursor associated with the mouse is moving beyond an edge of the monitor, taking control of the mouse messages and preventing propagation of the mouse messages to systems operating on the computing device,
sending the mouse messages to the HMD device over the network connection,
sending the keyboard messages to the HMD device over the network connection, and
sending extensibility data to the HMD device over the network, the extensibility data including scene data describing a 3D model supported by an application executing on the computing device, and wherein the extensibility data is configured to support multiple different rendering modes including 2D rendering on the desktop and 3D rendering on the viewport, in which the desktop and viewport each utilize the exchanged extensibility data to facilitate rendering of the 3D model in the different respective 2D rendering and 3D rendering supported by the desktop and the viewport, respectively.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/841,047Filed 13 Dec 2017 · continuation
Priority: 25 Jul 2014
Published: US 2018/0101994 A112 Apr 2018
Granted: 9 Oct 2018
Term ends: Not estimatedThe source records don’t support an estimate.

Classification29 codes

G06T 19/006
Manipulating three-dimensional [3D] models or images for computer graphics: mixed reality
G02B 27/017
Head-up displays: head mounted
G02B 27/0172
Head mounted: characterised by optical features
G06F 3/011
Input arrangements or combined input and output arrangements for interaction between user and computer: arrangements for interaction with the…

Family

Application62/029,351Us provisional application · filed 25 Jul 2014
Application14/688,817Parent of this continuation · filed 16 Apr 2015
US 2018/0101994 A1Pre-grant publication · published 12 Apr 2018
US 9,858,720 B2Parent of this continuation
US 2016/0027216 A1Published application
US 10,096,168 B2This patent

From the US filing history. Foreign family members aren’t listed.