# US 10,455,212 B1 · Projected pattern motion/vibration for depth sensing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,455,212 B1 - Document ID: US10455212B1 - Status: Granted - Kind: B1 - Application number: 14/583,903 - Filed: 2014-12-29 - Granted: 2019-10-22 - Earliest claimed: 2014-08-25 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Kurt Konolige; Ethan Rublee - Web page: https://patentgrep.com/patent/US10455212B1 ## CPC - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/167: Synchronising or controlling image signals - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Example implementations relate to determining depth information using stereo sensor data. An example system may include at least one projector coupled to a robotic manipulator and configured to project a texture pattern onto an environment. The system may further include a displacer coupled to the at least one texture projector and configured to repeatedly change a position of the texture pattern within the environment. The system may also include at least two optical sensors configured to capture stereo sensor data for the environment. And the system may include a computing device configured to determine, using the stereo sensor data, an output including a virtual representation of the environment. ## Claims ### Claim 1 A method comprising: causing, by one or more computing devices, a texture projector to project a texture pattern onto an environment; causing, by the one or more computing devices, a displacer coupled to the texture projector to repeatedly change a position of the texture pattern within the environment using a MicroElectroMechanical System (MEMS) device configured to repeatedly provide a back-and-forth motion that physically alters a position of the texture projector on a predefined path along a rail while the texture projector projects the texture pattern, wherein the MEMS device provides the back-and-forth motion at one or more frequencies based on a signal received by the MEMS device from the one or more computing devices; receiving, by the one or more computing devices, stereo sensor data for the environment, wherein the stereo sensor data comprises a plurality of image pairs corresponding to a plurality of different positions of the texture pattern, and wherein each image pair of the plurality of image pairs comprises a first image captured from a first viewpoint of a first optical sensor and a second image captured from a second viewpoint of a second optical sensor; and determining, by the one or more computing devices and using the stereo sensor data, an output including a virtual representation of the environment, wherein the output comprises a depth of at least one surface in the environment. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the output including the virtual representation of the environment using the stereo sensor data comprises: determining, using the plurality of image pairs, a plurality of depth images; and determining an output depth image that integrates information from the plurality of depth images, wherein the output depth image has a higher resolution than a resolution of individual depth images of the plurality of depth images. ### Claim 3 (depends on 1) The method of claim 1, wherein the displacer is positioned between the texture projector and the environment and configured to cause variations in the texture pattern projected onto the environment. ### Claim 4 (depends on 3) The method of claim 3, wherein the displacer comprises an electronically controllable optical element coupled to the texture projector. ### Claim 5 (depends on 4) The method of claim 4, wherein the electronically-controllable optical element comprises an electronically-controllable lens through which the texture pattern is projected, and wherein causing the displacer to repeatedly change the position of the texture pattern within the environment comprises causing variations in a shape of the electronically-controllable lens. ### Claim 6 (depends on 4) The method of claim 4, wherein the electronically-controllable optical element comprises an electronically-controllable prism through which the texture pattern is projected, and wherein causing the displacer to repeatedly change the position of the texture pattern within the environment comprises causing variations in at least one reflection angle of the electronically-controllable prism. ### Claim 7 (depends on 1) The method of claim 1, wherein causing the displacer to repeatedly change the position of the texture pattern within the environment comprises causing the displacer to repeatedly change the position of the texture pattern in a plurality of directions. ### Claim 8 A non-transitory computer-readable medium having stored therein instructions, that when executed by one or more processors of a computing device, cause the computing device to perform functions comprising: causing a texture projector to project a texture pattern onto an environment; causing a displacer coupled to the texture projector to repeatedly change a position of the texture pattern within the environment using a MicroElectroMechanical System (MEMS) device configured to repeatedly provide a back-and-forth motion that physically alters a position of the texture projector on a predefined path along a rail while the texture projector projects the texture pattern, wherein the MEMS device provides the back-and-forth motion at one or more frequencies based on a signal received by the MEMS device from the one or more computing devices; receiving stereo sensor data for the environment, wherein the stereo sensor data comprises a plurality of image pairs corresponding to a plurality of different positions of the texture pattern, and wherein each image pair of the plurality of image pairs comprises a first image captured from a first viewpoint of a first optical sensor and a second image captured from a second viewpoint of a second optical sensor; and determining, using the stereo sensor data, an output including a virtual representation of the environment, wherein the output comprises a depth of at least one surface in the environment. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein determining the output including the virtual representation of the environment using the stereo sensor data comprises: determining, using the plurality of image pairs, a plurality of depth images; and determining an output depth image that integrates information from the plurality of depth images, wherein the output depth image has a higher resolution than a resolution of individual depth images of the plurality of depth images. ### Claim 10 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the displacer is positioned between the texture projector and the environment and configured to cause variations in the texture pattern projected onto the environment. ### Claim 11 A system comprising: a robotic manipulator; at least one projector coupled to a rail on the robotic manipulator and configured to project a texture pattern onto an environment; a displacer coupled to the at least one projector and configured to repeatedly change a position of the texture pattern within the environment using a MicroElectroMechanical System (MEMS) device configured to repeatedly provide a back-and-forth motion that physically alters a position of the texture projector on a predefined path along the rail while the at least one projector projects the texture pattern, wherein the MEMS device is configured to provide the back-and-forth motion at one or more frequencies based on a signal received by the MEMS device from a computing device; at least two optical sensors configured to capture stereo sensor data for the environment, wherein the stereo sensor data comprises a plurality of image pairs corresponding to a plurality of different positions of the texture pattern during a predetermined time period, and wherein each image pair of the plurality of image pairs comprises a first image captured from a first viewpoint of a first optical sensor and a second image captured from a second viewpoint of a second optical sensor; and the computing device configured to determine, using the stereo sensor data, an output including a virtual representation of the environment, wherein the output comprises a depth of at least one surface in the environment. ### Claim 12 (depends on 11) The system of claim 11, wherein the computing device is configured to determine the output including the virtual representation of the environment using the stereo sensor data by: determining, using the plurality of image pairs, a plurality of depth images; and determining an output depth image that integrates information from the plurality of depth images, wherein the output depth image has a higher resolution than a resolution of individual depth images of the plurality of depth images. ### Claim 13 (depends on 11) The system of claim 11, wherein the displacer comprises an electronically controllable optical element positioned between the at least one projector and the environment and configured to cause variations in the texture pattern projected onto the environment. ### Claim 14 (depends on 13) The system of claim 13, wherein the electronically-controllable optical element comprises an electronically-controllable lens through which the texture pattern is projected. ### Claim 15 (depends on 13) The system of claim 13, wherein the electronically-controllable optical element comprises an electronically-controllable prism through which the texture pattern is projected. ### Claim 16 (depends on 11) The system of claim 11, wherein the displacer is configured to repeatedly change the position of the texture pattern in a plurality of directions during the predetermined time period. ### Claim 17 (depends on 1) The method of claim 1, wherein causing the displacer coupled to the texture projector to repeatedly change the position of the texture pattern within the environment using the MEMS device configured to repeatedly provide the back-and-forth motion that physically alters the position of the texture projector comprises: causing the displacer to repeatedly change the position of the texture pattern within the environment by causing the MEMS device to randomly vibrate along an axis for a predetermined time period. ### Claim 18 (depends on 17) The method of claim 17, wherein causing the displacer to repeatedly change the position of the texture pattern within the environment by causing the MEMS device to randomly vibrate along an axis for a predetermined time period comprises: causing the MEMS device to randomly vibrate along a horizontal axis for the predetermined time period. ### Claim 19 (depends on 17) The method of claim 17, wherein causing the displacer to repeatedly change the position of the texture pattern within the environment by causing the MEMS device to randomly vibrate along an axis for a predetermined time period comprises: causing the MEMS device to randomly vibrate along a vertical axis for the predetermined time period.