# US 10,122,978 B2 · Harmonizing a projected user interface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,978 B2 - Document ID: US10122978B2 - Status: Granted - Kind: B2 - Application number: 14/428,092 - Filed: 2014-04-01 - Published: 2016-09-08 - Granted: 2018-11-06 - Earliest claimed: 2014-04-01 - Assignees: Sony Corporation; Sony Mobile Communications Inc. - Applicants: Sony Corporation - Inventors: David De Leon; Ola Thorn; Magnus Midholt - Web page: https://patentgrep.com/patent/US10122978B2 ## CPC - H04N 9/3182: {Colour adjustment, e.g. white balance, shading or gamut (white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607)} - G03B 17/54: with projector - G06F 1/1673: {Arrangements for projecting a virtual keyboard} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)} - G06F 3/048: Interaction techniques based on graphical user interfaces [GUI] - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/0485: Scrolling or panning - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 9/3194: {including sensor feedback} - G01N 2291/02827: Elastic parameters, strength or force - G06T 2200/24: involving graphical user interfaces [GUIs] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) ## Abstract The present disclosure provides a projection device for adapting a property of a projected user interface. The device determines at least one property of a surface onto which a user interface is projected. The property of the projected user interface is changed based on the at least one property of the surface onto which the projected user interface is projected. ## Claims ### Claim 1 A method for adapting a property of an interactive projected user interface based on at least one property of a surface on to which the projected user interface is projected, the method comprising: determining the at least one property of the surface; determining a relevant user interface animation based on the determined at least one property of the surface, wherein the determined relevant user interface animation appears to match a physics of the surface; and applying the determined relevant user interface animation based on the determined at least one property of the surface, such that the updating of the projected user interface in response to the user interaction with the elements of the projected user interface is changed according to the determined user interface animation. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one property of the surface is at least one of the coloring of the surface, the texture of the surface, the deformability of the surface, the orientation of the surface, the hardness of the surface, or the topography of the surface. ### Claim 3 (depends on 2) The method of claim 2, wherein, if the surface is determined to be deformable, the projected user interface is updated to increase the size of elements of the projected user interface. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one property of the surface is measured during user interaction with the projected user interface. ### Claim 5 (depends on 1) The method of claim 1, wherein the projected user interface is updated such that sound effects of the projected user interface appear to match the at least one property of the surface. ### Claim 6 (depends on 1) The method of claim 1, wherein the determined relevant user interface setting is applied such that the projected user interface appears substantially homogeneous across color variations of the surface or such that the contrast between the projected user interface and the surface is approximately constant across the surface despite color variations of the surface. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one property of the surface is determined using a camera. ### Claim 8 (depends on 7) The method of claim 7, wherein the camera measures at least one of visual distortion of the projected user interface during user input to determine deformability of the surface or coloring of the surface. ### Claim 9 (depends on 8) The method of claim 8, wherein the camera measures visual distortion of the projected user interface during user input by analyzing deformation of the projected user interface during user interaction with the projected user interface. ### Claim 10 (depends on 1) The method of claim 1, wherein the at least one property of the surface is determined using an acoustic sensor and the at least one property of the surface is determined based on the sound generated by the user interacting with the surface. ### Claim 11 (depends on 10) The method of claim 10, wherein the at least one property of the surface measured using the acoustic sensor is at least one of a type of material of the surface or a hardness of the surface. ### Claim 12 (depends on 1) The method of claim 1, wherein the at least one property of the surface is determined using a user worn device. ### Claim 13 (depends on 12) The method of claim 12, wherein the user worn device includes at least one of an accelerometer or a gyroscope configured to capture measurements of user movements during interaction with the projected user interface. ### Claim 14 (depends on 13) The method of claim 13, wherein a processor analyzes the captured measurements to determine at least one of the hardness of the surface or the deformability of the surface. ### Claim 15 (depends on 1) The method of claim 1, wherein the at least one property of the surface is determined based on geo-location coordinates of the projected user interface. ### Claim 16 (depends on 15) The method of claim 15, wherein the geo-location coordinates of the projected user interface are determined using at least one of near field communication, Bluetooth, Geo fencing, indoor Wi-Fi positioning, or GPS. ### Claim 17 (depends on 15) The method of claim 15, wherein the user manually selects the relevant user interface setting for the surface and a current location of the user is tagged such that the selected relevant user interface setting are automatically selected to be the relevant user interface setting for user interfaces projected at the current location. ### Claim 18 (depends on 1) The method of claim 1, wherein the determined at least one property of the surface includes the deformability of the surface. ### Claim 19 (depends on 1) The method of claim 1, wherein the determined at least one property of the surface includes the hardness of the surface. ### Claim 20 A method for adapting a property of a projected user interface based on at least one property of a surface on to which the projected user interface is projected, the method comprising: determining the at least one property of the surface; determining a relevant user interface setting based on the determined at least one property of the surface; and applying the determined relevant user interface setting to the projected user interface, such that the property of the projected user interface is changed according to the determined user interface setting, wherein the projected user interface is updated such that at least one of animations or sound effects of the projected user interface appear to match the at least one property of the surface, wherein: on a less deformable surface, a pressed button results in a hard surface animation; and on a more deformable surface, a pressed button results in a soft surface animation. ### Claim 21 (depends on 20) The method of claim 20, wherein at least one of: the hard surface animation results in an element of the user interface bouncing higher than the soft surface animation; or the hard surface animation is a hard scrolling animation that behaves differently than a soft scrolling animation by at least one of responding quicker, scrolling faster, or scrolling for a longer time. ### Claim 22 A projecting device for projecting an interactive user interface onto a surface and for adapting a property of the projected user interface based on at least one property of the surface, the device comprising: a projector configured to project the user interface onto the surface; a sensor configured to determine the at least one property of the surface; and a processor configured to: determine a relevant user interface animation based on the determined at least one property of the surface, wherein the determined relevant user interface animation appears to match a physics of the surface; and control the user interface projected by the projector such that the updating of the projected user interface in response to the user interaction with the elements of the projected user interface is applied according to the determined user interface animation based on the determined at least one property of the surface. ### Claim 23 (depends on 22) A system for projecting a user interface onto a surface and for adapting a property of the projected user interface based on at least one property of the surface, the system comprising: the projecting device of claim 22; and a computing device in communication with the projecting device. ### Claim 24 (depends on 23) The system of claim 23, wherein the computing device includes the projecting device.