# US 10,482,862 B2 · Computer implemented method for providing augmented reality (AR) function regarding music track > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,862 B2 - Document ID: US10482862B2 - Status: Granted - Kind: B2 - Application number: 15/597,217 - Filed: 2017-05-17 - Published: 2018-11-22 - Granted: 2019-11-19 - Earliest claimed: 2017-05-17 - Assignees: Yousician Oy - Applicants: Yousician Oy - Inventors: Perttu Hämäläinen; Matti Ryynänen - Web page: https://patentgrep.com/patent/US10482862B2 ## CPC - G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/683: using metadata automatically derived from the content - G06T 15/005: {General purpose rendering architectures} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G09B 15/003: {with indication of the keys or strings to be played on instruments} - G10H 2220/005: Non-interactive screen display of musical or status data - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract A computer implemented method for providing an augmented reality (AR) function, comprises receiving input information regarding a music track and an instrument; determining attribute information of the music track based on the received input information; receiving real time content of audiovisual (AV) input signals using at least one capturing device; generating visual information corresponding to visual data of the real time content, wherein the visual information corresponds to a view regarding at least one user limb and an instrument comprising a plurality of user operable elements; generating augmented reality (AR) instruction information based on the attribute information of the music track, the augmented reality (AR) instruction information comprising a plurality of layers; and generating augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a first layer of the augmented reality (AR) instruction information is applied above at least a portion of the visual information. ## Claims ### Claim 1 A computer implemented method for providing an augmented reality (AR) function, comprising: receiving input information of a music track and an instrument; determining attribute information of the music track based on the received input information, the attribute information comprising data for a user to play the music track with the instrument; receiving real time content of audiovisual (AV) input signals using at least one capturing device; the real time content comprising visual information that corresponds to a view regarding at least one user limb and the instrument comprising a plurality of user operable elements; generating augmented reality (AR) instruction information based on the attribute information of the music track, the augmented reality (AR) instruction information comprising a plurality of layers; generating augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a first layer of the augmented reality (AR) instruction information is applied above at least a portion of the visual information; and causing a display of the generated augmented reality visual information. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating the augmented reality (AR) visual information by mixing the augmented reality (AR) instruction information with the visual information by a mixer application; encoding the augmented reality (AR) visual information by the mixer application; and providing the encoded augmented reality (AR) visual information by the mixer application to a user interface of a user device. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving instrument selection information; receiving track selection information; and determining the input information of the music track and the instrument based on the instrument selection information and the track selection information. ### Claim 4 (depends on 3) The method of claim 3, further comprising: segmenting the visual information to generate a plurality of data segments within the visual information, wherein the plurality of data segments comprises at least a user data segment and an instrument data segment. ### Claim 5 (depends on 4) The method of claim 4, wherein the user data segment comprises information of a user limb used to play the instrument. ### Claim 6 (depends on 5) The method of claim 5, wherein the user data segment comprises information of a position or an orientation of the user limb. ### Claim 7 (depends on 4) The method of claim 4, wherein the instrument data segment comprises information of the at least one of the plurality of user operable elements of the instrument. ### Claim 8 (depends on 7) The method of claim 7, wherein the instrument data segment comprises information of a position or an orientation of the at least one of the plurality of user operable elements. ### Claim 9 (depends on 7) The method of claim 7, wherein the at least one of the plurality of user operable elements comprises at least one of the following: a fret; a string; a key; and a button. ### Claim 10 (depends on 4) The method of claim 4, wherein a second layer of the augmented reality (AR) instruction information is applied below the user data segment and above the instrument data segment. ### Claim 11 (depends on 4) The method of claim 4, wherein a second layer of the augmented reality (AR) instruction information is applied below the user data segment and below the instrument data segment. ### Claim 12 (depends on 4) The method of claim 4, comprising: generating the augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a third layer of the augmented reality (AR) instruction information is applied below a second layer of the augmented reality (AR) instruction information. ### Claim 13 (depends on 12) The method of claim 12, wherein the second layer of the augmented reality (AR) instruction information is applied below a first instrument data segment and above a second instrument data segment, and the third layer of the augmented reality (AR) instruction information is applied below the second instrument data segment. ### Claim 14 (depends on 1) The method of claim 1, further comprising performing distance-based segmentation using a depth camera. ### Claim 15 (depends on 14) The method of claim 14, further comprising: determining 3D space information of the instrument based on captured real time content; and adjusting the augmented reality (AR) instruction information based on the 3D space information so that the augmented reality (AR) instruction information changes according to changes in position or orientation of the instrument. ### Claim 16 (depends on 15) The method of claim 15 wherein the method further comprises: segmenting the visual information to generate a plurality of data segments within the visual information, wherein the plurality of data segments comprises at least a user data segment and an instrument data segment; and the first layer of the augmented reality instruction information is applied above the user data segment and the instrument data segment. ### Claim 17 An apparatus for providing an augmented reality (AR) function, comprising: a communication interface; a user interface; at least one capturing device; at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receive input information regarding a music track and an instrument; determine attribute information of the music track based on the received input information, the attribute information comprising data for a user to play the music track with the instrument; receive real time content of audiovisual (AV) input signals using the at least one capturing device; the real time content comprising visual information that corresponds to a view regarding at least one user limb and the instrument comprising a plurality of user operable elements; generate augmented reality (AR) instruction information based on the attribute information of the music track, the augmented reality (AR) instruction information comprising a plurality of layers; generate augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a first layer of the augmented reality (AR) instruction information is applied above at least a portion of the visual information; and cause a display of the generated augmented reality visual information. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein: the apparatus further comprises a depth camera; the at least one capturing device comprises a camera directed at the instrument and at least one of the following: the instrument, and a microphone; and the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: perform distance-based segmentation using the depth camera. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the camera is attached to the instrument so that the camera is configured to capture the real time content of the view regarding the at least one user limb and at least one of the plurality of user operable elements of the instrument. ### Claim 20 A computer program embodied on a computer readable non-transitory medium comprising computer executable program code, which when executed by at least one processor of a master device, causes an apparatus to: receive input information of a music track and an instrument; determine attribute information of the music track based on the received input information, the attribute information comprising data for a user to play the music track with the instrument; receive real time content of audiovisual (AV) input signals using at least one capturing device; the real time content comprising the visual information that corresponds to a view regarding at least one user limb and the instrument comprising a plurality of user operable elements; generate augmented reality (AR) instruction information based on the attribute information of the music track, the augmented reality (AR) instruction information comprising a plurality of layers; generate augmented reality (AR) visual information by applying the augmented reality (AR) instruction information to the visual information so that a first layer of the augmented reality (AR) instruction information is applied above at least a portion of the visual information; and causing a display of the generated augmented reality visual information.