# US 12,112,731 B2 · Method and apparatus for generating music file, and electronic device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,731 B2 - Document ID: US12112731B2 - Status: Granted - Kind: B2 - Application number: 18/545,825 - Filed: 2023-12-19 - Published: 2024-04-18 - Granted: 2024-10-08 - Earliest claimed: 2021-07-23 - Assignees: Beijing Zitiao Network Technology Co., Ltd. - Applicants: Beijing Zitiao Network Technology Co., Ltd. - Inventors: Yufan Xue; Guanjun Guo; Xin Yuan; Yuezhao Chen; Hao Huang; Na Li; Xubin Zhou - Web page: https://patentgrep.com/patent/US12112731B2 ## CPC - G06V 10/82: using neural networks - G10H 1/0066: {using a MIDI interface} - G06T 7/11: Region-based segmentation - G06T 7/73: using feature-based methods - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 10/764: using classification, e.g. of video objects - G10G 1/02: Chord or note indicators, fixed or adjustable, for keyboard of fingerboards - G10H 1/0008: {Associated control or indicating means} - G10H 1/0025: {Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece (automatically producing a series of tones G10H1/26)} - G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part} - G06T 2207/10016: Video; Image sequence - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G10H 2220/441: Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes - G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece - G10H 2250/311: Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10G: REPRESENTATION OF MUSIC; RECORDING MUSIC IN NOTATION FORM; ACCESSORIES FOR MUSIC OR MUSICAL INSTRUMENTS NOT OTHERWISE PROVIDED FOR, e.g. SUPPORTS (music stands A47B; non-musical aspects of musical toy instruments A63H5/00; metronomes G04F5/02; teaching music G09B15/00) ## Abstract The present application relates to the technical field of computers, and discloses a method and apparatus for generating a music file, and an electronic device and a storage medium. The method for generating a music file comprises: obtaining a first image; performing feature extraction on the first image to obtain a salient feature of the first image; mapping the salient feature to a musical instrument digital interface (MIDI) information coordinate system on the basis of the position of the salient feature in the first image, so as to determine MIDI information corresponding to the salient feature, the MIDI information coordinate system being used for indicating a correspondence between MIDI information and time; and generating a music file on the basis of the correspondence between the MIDI information and the time. ## Claims ### Claim 1 A method for generating a music file, comprising: obtaining a first image; performing feature extraction on the first image to obtain a salient feature of the first image; mapping the salient feature into a musical instrument digital interface (MIDI) information coordinate system based on a position of the salient feature in the first image, so as to determine MIDI information corresponding to the salient feature, the MIDI information coordinate system being used for indicating correspondence between the MIDI information and time; and generating a music file based on the correspondence between the MIDI information and time; generating a second image corresponding to the music file; wherein generating a second image corresponding to the music comprises: receiving a second input, wherein the second input is an input for selecting a preset video template; determining an object video template in response to the second input; and generating the second image according to the object video template and a salient object texture map. ### Claim 2 (depends on 1) The method for generating a music file according to claim 1, wherein an image content of the first image comprises a salient object, and the salient feature comprises at least one of: a key point of the salient object or an edge feature point of the salient object. ### Claim 3 (depends on 2) The method for generating a music file according to claim 2, wherein performing feature extraction on the first image to obtain a salient feature of the first image comprises: performing object segmentation on the first image through a convolutional neural network to obtain the salient object in the first image and the edge feature point of the salient object; and performing key point extraction on the salient object to obtain a key point of the salient object. ### Claim 4 (depends on 2) The method for generating a music file according to claim 2, wherein prior to mapping the salient feature into the MIDI information coordinate system based on a position of the salient feature in the first image, the method for generating a music file further comprises: generating the salient object texture map corresponding to the first image according to the salient feature; and determining a position of the salient feature in the first image according to the salient object texture map. ### Claim 5 (depends on 4) The method for generating a music file according to claim 4, wherein generating the salient object texture map corresponding to the first image according to the salient feature comprises: performing edge detection on the first image according to the edge feature point and a Canny edge detection algorithm to obtain an edge image of the salient object; generating a salient object graph corresponding to the salient object according to the key point and the edge feature point; and performing image superposition on the edge image and the salient object graph to obtain the salient object texture map corresponding to the first image. ### Claim 6 (depends on 4) The method for generating a music file according to claim 4, wherein determining a position of the salient feature in the first image according to the salient object texture map comprises: dividing the object texture map into X×Y image units with x rows and y columns, wherein both x and y are integers greater than 1, and the image units comprise at least one of bright pixels and dark pixels, and the bright pixels are pixels with a brightness value of 1 and the dark pixels are pixels with a brightness value of 0; determining an object image unit with a ratio of bright pixels greater than a preset ratio among X×Y image units to obtain N object image units, wherein the number of salient features of the first image is N, and the N object image units are in one-to-one correspondence with the N salient features, and N is a positive integer; determining a first ordinate of the salient feature in the first image according to a number of rows of each of the n object image units in the X×Y image units; determining a first abscissa of the salient feature in the first image according to a number of columns of each of the n object image units in the X×Y image units; and determining a position of the salient feature in the first image according to an abscissa of the salient feature and an abscissa and an ordinate and of the salient feature. ### Claim 7 (depends on 6) The method for generating a music file according to claim 6, wherein mapping the salient feature into a MIDI information coordinate system based on a position of the salient feature in the first image comprises: converting the first ordinate into the MIDI information coordinate system to obtain a second ordinate of the salient feature in the MIDI information coordinate system; converting the first abscissa into the MIDI information coordinate system to obtain a second abscissa of the salient feature in the MIDI information coordinate system; and according to the second ordinate and the second abscissa, mapping the N salient features into the MIDI information coordinate system to obtain N track chunks corresponding to the N salient features one by one. ### Claim 8 (depends on 7) The method for generating the music file according to claim 7, wherein the track chunk contains the MIDI information, and the MIDI information is determined according to a second ordinate corresponding to the track chunk; wherein, the MIDI information comprises at least one of: pitch, timbre or volume. ### Claim 9 (depends on 4) The method for generating a music file according to claim 4, further comprising: receiving a first input, wherein the first input is an input for selecting a preset music feature; determining a target music feature in response to the first input, wherein the target music feature comprises at least one of the following: music style, music mood or music genre; adjusting the music according to the music feature; and playing the music file. ### Claim 10 (depends on 9) The method for generating the music file according to claim 9, wherein playing the music comprises: displaying the second image and playing the music. ### Claim 11 (depends on 1) The method for generating a music file according to claim 1, wherein generating a second image corresponding to the music comprises: generating an object animation through a piano roll graphical interface, wherein the object animation is used for showing a playing progress of the music; and generating the second image according to the object animation and the salient object texture map. ### Claim 12 An electronic device, comprising a processor, a memory and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, performs: obtaining a first image; performing feature extraction on the first image to obtain a salient feature of the first image; mapping the salient feature into a musical instrument digital interface (MIDI) information coordinate system based on a position of the salient feature in the first image, so as to determine MIDI information corresponding to the salient feature; the MIDI information coordinate system being used for indicating correspondence between the MIDI information and time; and generating a music file based on the correspondence between the MIDI information and time; generating a second image corresponding to the music file; wherein generating a second image corresponding to the music comprises: receiving a second input, wherein the second input is an input for selecting a preset video template; determining an object video template in response to the second input; and generating the second image according to the object video template and a salient object texture map. ### Claim 13 (depends on 12) The electronic device for generating a music file according to claim 12, wherein an image content of the first image comprises a salient object, and the salient feature comprises at least one of: a key point of the salient object or an edge feature point of the salient object. ### Claim 14 (depends on 13) The electronic device for generating a music file according to claim 13, wherein performing feature extraction on the first image to obtain a salient feature of the first image comprises: performing object segmentation on the first image through a convolutional neural network to obtain the salient object in the first image and the edge feature point of the salient object; and performing key point extraction on the salient object to obtain a key point of the salient object. ### Claim 15 (depends on 13) The electronic device for generating a music file according to claim 13, wherein prior to mapping the salient feature into the MIDI information coordinate system based on a position of the salient feature in the first image, the program or instructions further performs: generating the salient object texture map corresponding to the first image according to the salient feature; and determining a position of the salient feature in the first image according to the salient object texture map. ### Claim 16 (depends on 15) The electronic device for generating a music file according to claim 15, wherein generating the salient object texture map corresponding to the first image according to the salient feature comprises: performing edge detection on the first image according to the edge feature point and a Canny edge detection algorithm to obtain an edge image of the salient object; generating a salient object graph corresponding to the salient object according to the key point and the edge feature point; and performing image superposition on the edge image and the salient object graph to obtain the salient object texture map corresponding to the first image. ### Claim 17 (depends on 15) The electronic device for generating a music file according to claim 15, wherein determining a position of the salient feature in the first image according to the salient object texture map comprises: dividing the object texture map into X×Y image units with x rows and y columns, wherein both x and y are integers greater than 1, and the image units comprise at least one of bright pixels and dark pixels, and the bright pixels are pixels with a brightness value of 1 and the dark pixels are pixels with a brightness value of 0; determining an object image unit with a ratio of bright pixels greater than a preset ratio among X×Y image units to obtain N object image units, wherein the number of salient features of the first image is N, and the N object image units are in one-to-one correspondence with the N salient features, and N is a positive integer; determining a first ordinate of the salient feature in the first image according to a number of rows of each of the n object image units in the X×Y image units; determining a first abscissa of the salient feature in the first image according to a number of columns of each of the n object image units in the X×Y image units; and determining a position of the salient feature in the first image according to an abscissa of the salient feature and an abscissa and an ordinate and of the salient feature. ### Claim 18 (depends on 17) The electronic device for generating a music file according to claim 17, wherein mapping the salient feature into a MIDI information coordinate system based on a position of the salient feature in the first image comprises: converting the first ordinate into the MIDI information coordinate system to obtain a second ordinate of the salient feature in the MIDI information coordinate system; converting the first abscissa into the MIDI information coordinate system to obtain a second abscissa of the salient feature in the MIDI information coordinate system; and according to the second ordinate and the second abscissa, mapping the N salient features into the MIDI information coordinate system to obtain N track chunks corresponding to the N salient features one by one. ### Claim 19 A readable non-transitory storage medium, wherein a program or instruction is stored on the readable storage medium, and the program or instruction, when executed by a processor, performs: obtaining a first image; performing feature extraction on the first image to obtain a salient feature of the first image; mapping the salient feature into a musical instrument digital interface (MIDI) information coordinate system based on a position of the salient feature in the first image, so as to determine MIDI information corresponding to the salient feature; the MIDI information coordinate system being used for indicating correspondence between the MIDI information and time; and generating a music file based on the correspondence between the MIDI information and time; generating a second image corresponding to the music file; wherein generating a second image corresponding to the music comprises: receiving a second input, wherein the second input is an input for selecting a preset video template; determining an object video template in response to the second input; and generating the second image according to the object video template and a salient object texture map.