# US 10,864,433 B2 · Using a portable device to interact with a virtual space > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,864,433 B2 - Document ID: US10864433B2 - Status: Granted - Kind: B2 - Application number: 15/482,669 - Filed: 2017-04-07 - Published: 2017-07-27 - Granted: 2020-12-15 - Earliest claimed: 2010-10-12 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Yunpeng Zhu; Jeffrey Roger Stafford; Steven Osman - Web page: https://patentgrep.com/patent/US10864433B2 ## CPC - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 13/2145: the surface being also a display device, e.g. touch screens - A63F 13/25: Output arrangements for video game devices - A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth - A63F 13/323: between game devices with different hardware characteristics, e.g. hand-held game devices connectable to game consoles or arcade machines - A63F 13/352: involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes - A63F 13/63: by the player, e.g. authoring using a level editor - A63F 13/86: Watching games played by other players - A63F 13/92: Video game devices specially adapted to be hand-held while playing - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/327: using wireless networks, e.g. Wi-Fi® or piconet - A63F 13/47: involving branching, e.g. choosing one of several possible scenarios at a given point in time - A63F 2300/301: using an additional display connected to the game console, e.g. on the controller - A63F 2300/403: Connection between platform and handheld device - A63F 2300/538: for performing operations on behalf of the game client, e.g. rendering - A63F 2300/632: by branching, e.g. choosing one of several possible story developments at a given point in time - A63F 2300/8082: Virtual reality - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR ## Abstract In one implementation, a portable device is provided, including: a sensor configured to generate sensor data for determining and tracking a position and orientation of the portable device during an interactive session of a virtual reality program presented on a main display, the interactive session being defined for interactivity between a user and the virtual reality program; a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive from the computing device an ancillary video stream of the interactive session that is generated based on a state of the virtual reality program and the tracked position and orientation of the portable device. ## Claims ### Claim 1 A portable device, comprising: a sensor configured to generate sensor data for determining and tracking a position and orientation of the portable device during an interactive session of a virtual reality program that is executed by a computing device and presented on a main display, the interactive session being defined for interactivity between a user and the virtual reality program; a communications module configured to send the sensor data to the computing device, the communications module being further configured to receive from the computing device an ancillary video stream of the interactive session that is generated based on a state of the virtual reality program and the tracked position and orientation of the portable device, the state of the virtual reality program being determined based on the interactivity between the user and the virtual reality program; a display configured to render the ancillary video stream; wherein a virtual viewpoint into a virtual space is presented on the main display by the virtual reality program, the main display provides a direct view from a first perspective into the virtual space, and the virtual viewpoint is changed when viewed via the portable device based on a location of the portable device, the change in the virtual viewpoint provides a side view from a second perspective into the virtual space that exposes a separation between objects in a foreground and objects in a background of the direct view, the second perspective being different from the first perspective; and wherein the ancillary video stream includes images of the virtual space captured from the second perspective of the virtual viewpoint so as to provide the side view; wherein the ancillary video stream is generated based on the position and orientation of the portable device relative to a tracked position of the user during the interactive session; wherein the position and orientation of the portable device relative to the position of the user define the change in the position and orientation of the virtual viewpoint within the virtual space defined by the virtual reality program. ### Claim 2 (depends on 1) The portable device of claim 1, wherein the state of the virtual reality program is determined from interactive input, the interactive input being received from a controller device during the interactive session. ### Claim 3 (depends on 1) The portable device of claim 1, wherein the interactive session defines interactivity between the user and a primary video stream rendered on the main display, the primary video stream being generated based on the state of the virtual reality program. ### Claim 4 (depends on 3) The portable device of claim 3, wherein the interactive session further defines interactivity between a second user and the ancillary video stream rendered on the display, the state of the virtual reality program being determined, at least in part, by the interactivity between the second user and the ancillary video stream. ### Claim 5 (depends on 1) The portable device of claim 1, wherein the sensor is defined by one or more of an accelerometer, a magnetometer, and/or a gyroscope; and wherein the sensor data includes inertial data generated by the sensor. ### Claim 6 (depends on 1) The portable device of claim 1, wherein the sensor is defined by an image capture device; and wherein the sensor data includes captured image data of an interactive environment in which the interactivity occurs. ### Claim 7 (depends on 6) The portable device of claim 6, wherein determining and tracking the position and orientation of the portable device is based on identification of one or more markers in the interactive environment. ### Claim 8 (depends on 1) The portable device of claim 1, wherein an object within a virtual environment of the virtual reality program is mapped to the position of the user, the object being included in the images of the ancillary video stream when the portable device is oriented towards the position of the user. ### Claim 9 (depends on 1) The portable device of claim 1, wherein the communications module is configured to send the sensor data wireles sly to the computing device, and wherein the computing device is separate from the portable device. ### Claim 10 (depends on 1) The portable device of claim 1, wherein the direct view and the side view define views of the virtual space from different virtual perspectives in accordance with a relative positioning of the user and a spectator that operates the portable device. ### Claim 11 A method, comprising: executing, by a computing device, an interactive session of a virtual reality program presented on a main display, the interactive session being defined for interactivity between a user and the virtual reality program; processing, by the computing device, sensor data received from a portable device to determine and track a position and orientation of the portable device; generating, by the computing device, an ancillary video stream of the interactive session based on a state of the virtual reality program and the tracked position and orientation of the portable device, the state of the virtual reality program being determined based on the interactivity between the user and the virtual reality program; sending the ancillary video stream from the computing device to the portable device for rendering on a display of the portable device; wherein a first virtual viewpoint into a virtual space is presented on the main display by the virtual reality program, the main display provides a direct view into the virtual space depicted from a perspective of the first virtual viewpoint, wherein a second virtual viewpoint is viewed via the portable device based on a location of the portable device, the portable device provides a side view into the virtual space depicted from a perspective of the second virtual viewpoint that exposes a separation between objects in a foreground and objects in a background of the direct view, the perspective of the second virtual viewpoint providing a different vantage point of the virtual space than the perspective of the first virtual viewpoint; and wherein the ancillary video stream includes images of the virtual space captured from the perspective of the second virtual viewpoint; wherein the ancillary video stream is generated based on the position and orientation of the portable device relative to a tracked position of the user during the interactive session; wherein the position and orientation of the portable device relative to the position of the user define the position and orientation of the second virtual viewpoint within the virtual space defined by the virtual reality program. ### Claim 12 (depends on 11) The method of claim 11, wherein the state of the virtual reality program is determined from interactive input, the interactive input being received from a controller device during the interactive session. ### Claim 13 (depends on 11) The method of claim 11, wherein the interactive session defines interactivity between the user and a primary video stream rendered on the main display, the primary video stream being generated based on the state of the virtual reality program. ### Claim 14 (depends on 13) The method of claim 13, wherein the interactive session further defines interactivity between a second user and the ancillary video stream rendered on the display, the state of the virtual reality program being determined, at least in part, by the interactivity between the second user and the ancillary video stream. ### Claim 15 (depends on 11) The method of claim 11, wherein the sensor data includes inertial data generated from one or more of an accelerometer, a magnetometer, and/or a gyroscope. ### Claim 16 (depends on 11) The method of claim 11, wherein the sensor data includes captured image data of an interactive environment in which the interactivity occurs. ### Claim 17 (depends on 16) The method of claim 16, wherein determining and tracking the position and orientation of the portable device is based on identification of one or more markers in the interactive environment. ### Claim 18 (depends on 11) The method of claim 11, wherein the sensor data is wirelessly received from the portable device; and wherein the computing device is separate from the portable device. ### Claim 19 (depends on 11) The method of claim 11, wherein the direct view and the side view define views of the virtual space from different virtual perspectives in accordance with a relative positioning of the user and a spectator that operates the portable device. ### Claim 20 A system, comprising: a computing device, the computing device configured to execute an interactive session of a video game, to generate a primary video stream presented on a main display, a state of the interactive session being determined based on interactivity with a first user that views the main display; a portable device, including, a sensor configured to generate sensor data, a communications module configured to send the sensor data to the computing device, and a display; wherein the computing device is configured to process the sensor data to determine and track a position and orientation of the portable device; wherein the computing device is configured to generate an ancillary video stream of the interactive session based on the state of the interactive session and the tracked position and orientation of the portable device, the ancillary video stream being presented on the portable device for viewing by a second user; wherein the primary video stream presented on the main display provides a direct view into the virtual space depicted from a perspective of a first virtual viewpoint; wherein the ancillary video stream presented on the portable device provides a side view into the virtual space depicted from a perspective of a second virtual viewpoint that exposes a separation between objects in a foreground and objects in a background of the direct view; wherein the perspective of the second virtual viewpoint being different than the perspective of the first virtual viewpoint, so that the primary video stream and the ancillary video stream provide viewing of the virtual space from different vantage points, wherein the ancillary video stream is generated based on the position and orientation of the portable device relative to a tracked position of the first user during the interactive session; wherein the position and orientation of the portable device relative to the position of the first user define the position and orientation of the second virtual viewpoint within the virtual space defined by the virtual reality program. ### Claim 21 (depends on 20) The system of claim 20, wherein the direct view and the side view define views of the virtual space from different virtual perspectives in accordance with a relative positioning of the first user and the second user.