# US 2021/0279892 A1 · Image processing method and device, and network training method and device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279892 A1 - Document ID: US20210279892A1 - Status: Published application - Kind: A1 - Application number: 17/329,534 - Filed: 2021-05-25 - Published: 2021-09-09 - Earliest claimed: 2019-01-29 - Applicants: Beijing Sensetime Technology Development Co., Ltd. - Inventors: Xiaohang ZHAN; Xingang PAN; Ziwei LIU; Dahua LIN; Chen Change LOY - Web page: https://patentgrep.com/patent/US20210279892A1 ## CPC - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G06F 18/2413: based on distances to training or reference patterns - G06F 18/253: {of extracted features} - G06F 18/254: {of classification results, e.g. of results related to same input data} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data} - G06V 10/82: using neural networks - G06T 2207/20084: Artificial neural networks [ANN] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing method and a device, and a network training method and a device are provided. The image processing method includes determining a guide group arranged on an image to be processed and directed at a target object, the guide group comprising at least one guide point, and the guide point being used to indicate the position of a sampling pixel, and the magnitude and direction of the motion speed of the sampling pixel; and on the basis of the guide point in the guide group and the image to be processed, performing optical flow prediction to obtain the motion of the target object in the image to be processed. ## Claims ### Claim 1 An image processing method, comprising: determining a guidance group set for a target object in a to-be-processed image, the guidance group comprising at least one guidance point, the guidance point being configured to indicate a position of a sampling pixel and a magnitude and direction of a motion velocity of the sampling pixel, and the sampling pixel being a pixel of the target object in the to-be-processed image; and performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain a motion of the target object in the to-be-processed image. ### Claim 2 (depends on 1) The method of claim 1, wherein performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: performing, according to the magnitude and direction of the motion velocity of the sampling pixel indicated by the guidance point in the guidance group, the position of the sampling pixel indicated by the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image. ### Claim 3 (depends on 1) The method of claim 1, wherein performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: generating, according to the magnitude and direction of the motion velocity of the sampling pixel indicated by the guidance point in the guidance group, a sparse motion corresponding to the target object in the to-be-processed image, the sparse motion being configured to indicate a magnitude and direction of a motion velocity of each sampling pixel of the target object; generating, according to the position of the sampling pixel indicated by the guidance point in the guidance group, a binary mask corresponding to the target object in the to-be-processed image, the binary mask being configured to indicate a position of each sampling pixel of the target object; and performing, according to the sparse motion, the binary mask and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image. ### Claim 4 (depends on 1) The method of claim 1, wherein performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: performing optical flow prediction by inputting the guidance point in the guidance group and the to-be-processed image to a first neural network, to obtain the motion of the target object in the to-be-processed image. ### Claim 5 (depends on 3) The method of claim 3, wherein performing, according to the sparse motion, the binary mask and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: performing feature extraction on the sparse motion corresponding to the target object in the to-be-processed image and the binary mask corresponding to the target object in the to-be-processed image to obtain a first feature; performing feature extraction on the to-be-processed image to obtain a second feature; performing connection processing on the first feature and the second feature to obtain a third feature; and performing optical flow prediction on the third feature to obtain the motion of the target object in the to-be-processed image. ### Claim 6 (depends on 5) The method of claim 5, wherein performing optical flow prediction on the third feature to obtain the motion of the target object in the to-be-processed image comprises: performing full-extent propagation processing by inputting the third feature to at least two propagation networks respectively, to obtain a propagation result corresponding to each of the at least two propagation network; and performing fusion by inputting the propagation result corresponding to each propagation network to a fusion network, to obtain the motion of the target object in the to-be-processed image. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the guidance group set for the target object in the to-be-processed image comprises: determining multiple guidance groups set for the target object in the to-be-processed image, each of the multiple guidance groups comprising at least one guidance point different from guidance points of other guidance groups. ### Claim 8 (depends on 7) The method of claim 7, wherein performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: performing, according to a guidance point in each guidance group and the to-be-processed image, optical flow prediction to obtain a motion, corresponding to a guidance of guidance group, of the target object in the to-be-processed image. ### Claim 9 (depends on 8) The method of claim 8, further comprising: mapping the to-be-processed image according to the motion, corresponding to the guidance of each guidance group, of the target object to obtain a new image corresponding to each guidance group; and generating a video according to the to-be-processed image and the new image corresponding to each guidance group. ### Claim 10 (depends on 1) The method of claim 1, wherein determining the guidance group set for the target object in the to-be-processed image comprises: determining at least one first guidance point set for a first target object in the to-be-processed image; and generating multiple guidance groups according to the at least one first guidance point, directions of first guidance points in a same guidance group being the same and directions of first guidance points in different guidance groups being different. ### Claim 11 (depends on 10) The method of claim 10, wherein performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image comprises: performing, according to the first guidance point in each of the multiple guidance groups and the to-be-processed image, optical flow prediction to obtain a motion, corresponding to a guidance of each guidance group, of the first target object in the to-be-processed image. ### Claim 12 (depends on 11) The method of claim 11, further comprising: fusing the motion, corresponding to the guidance of each guidance group, of the first target object in the to-be-processed image to obtain a mask corresponding to the first target object in the to-be-processed image. ### Claim 13 (depends on 11) The method of claim 11, further comprising: determining at least one second guidance point set in the to-be-processed image, a motion velocity of the second guidance point being 0, wherein performing, according to the first guidance point in each guidance group and the to-be-processed image, optical flow prediction to obtain the motion, corresponding to the guidance of each guidance group, of the first target object in the to-be-processed image comprises: performing, according to the first guidance point in each guidance group, the second guidance point and the to-be-processed image, optical flow prediction to obtain the motion, corresponding to the guidance of each guidance group, of the first target object in the to-be-processed image. ### Claim 14 An electronic device, comprising: a processor; and a memory, configured to store instructions executable for the processor, wherein when the instructions are executed by the processor, the processor is configured to: determine a guidance group set for a target object in an to-be-processed image, the guidance group comprising at least one guidance point, the guidance point being configured to indicate a position of a sampling pixel and a magnitude and direction of a motion velocity of the sampling pixel and the sampling pixel being a pixel of the target object in the to-be-processed image; and perform, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain a motion of the target object in the to-be-processed image. ### Claim 15 (depends on 14) The electronic device of claim 14, wherein the processor is further configured to: perform, according to the magnitude and direction of the motion velocity of the sampling pixel indicated by the guidance point in the guidance group, the position of the sampling pixel indicated by the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image. ### Claim 16 (depends on 14) The electronic device of claim 14, wherein the processor is further configured to: generate, according to the magnitude and direction of the motion velocity of the sampling pixel indicated by the guidance point in the guidance group, a sparse motion corresponding to the target object in the to-be-processed image, the sparse motion being configured to indicate a magnitude and direction of a motion velocity of each sampling pixel of the target object; generate, according to the position of the sampling pixel indicated by the guidance point in the guidance group, a binary mask corresponding to the target object in the to-be-processed image, the binary mask being configured to indicate a position of each sampling pixel of the target object; and perform, according to the sparse motion, the binary mask and the to-be-processed image, optical flow prediction to obtain the motion of the target object in the to-be-processed image. ### Claim 17 (depends on 14) The electronic device of claim 14, wherein the processor is further configured to: perform optical flow prediction by inputting the guidance point in the guidance group and the to-be-processed image to a first neural network, to obtain the motion of the target object in the to-be-processed image. ### Claim 18 (depends on 16) The electronic device of claim 16, wherein the processor is configured to: perform feature extraction on the sparse motion corresponding to the target object in the to-be-processed image and the binary mask corresponding to the target object in the to-be-processed image to obtain a first feature; perform feature extraction on the to-be-processed image to obtain a second feature; perform connection processing on the first feature and the second feature to obtain a third feature; and perform optical flow prediction on the third feature to obtain the motion of the target object in the to-be-processed image. ### Claim 19 (depends on 18) The electronic device of claim 18, wherein the processor is further configured to: perform full-extent propagation processing by inputting the third feature to at least two propagation networks respectively, to obtain a propagation result corresponding to each propagation network; and perform fusion performing by inputting the propagation result corresponding to each propagation network to a fusion network, to obtain the motion of the target object in the to-be-processed image. ### Claim 20 A computer-readable storage medium, in which computer program instructions are stored, the computer program instructions being executed by a processor to perform: determining a guidance group set for a target object in a to-be-processed image, the guidance group comprising at least one guidance point, the guidance point being configured to indicate a position of a sampling pixel and a magnitude and direction of a motion velocity of the sampling pixel, and the sampling pixel being a pixel of the target object in the to-be-processed image; and performing, according to the guidance point in the guidance group and the to-be-processed image, optical flow prediction to obtain a motion of the target object in the to-be-processed image.