Back to results
US 11,386,587 B2Granted
Automatic coloring of line drawing
AssigneePreferred Networks, Inc.
InventorEiichi Matsumoto
Life of this patent
Estimating the term…
Abstract
A line drawing automatic coloring method according to the present disclosure includes: acquiring line drawing data of a target to be colored; receiving at least one local style designation for applying a selected local style to at least one place of the acquired line drawing data; and performing coloring processing reflecting the local style designation on the line drawing data based on a learned model for coloring in which it is learned in advance using the line drawing data and the local style designation as inputs.
Claims36
A method of automatic coloring, comprising:
- acquiring data of a target to be colored;
- receiving at least one local style designation for applying a selected local style to at least a first part of the target to be colored, wherein the selected local style is not applied to another part of the target to be colored;
- and
- performing coloring processing reflecting the local style designation on the first part of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the selected local style applied to the first part of the target, the extracted information being obtained from a predetermined process for extracting a local style based on a user-designated part of a reference image having the selected local style, and (iii) information indicating a position of the first part to which the selected local style is applied, into a neural network system for coloring the target to be colored,
- wherein the selected local style is related to a style different from a color, and
- wherein the neural network system is obtained based on at least a training process using at least (i) image data to be colored, (ii) extracted information that indicates a local style to be applied to a part of the image data to be colored and is obtained from the predetermined process for extracting a local style, (iii) information indicating a position of the part of the image data to be colored, and (iv) answer image data of the image data to be colored.
3 more elements
An automatic coloring device, comprising:
- one or more memories;
- and
- one or more processors configured to:
- acquire data of a target to be colored;
- receive at least one local style designation for applying a selected local style to at least a first part of the acquired data, wherein the selected local style is not applied to another part of the acquired data;
- and
- perform coloring processing reflecting the local style designation on the data of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the selected local style applied to the first part of the target, the extracted information being obtained from a predetermined process for extracting a local style based on a user-designated part of a reference image having the selected local style, and (iii) information indicating a position of the first part to which the selected local style is applied, into a neural network system for coloring the target to be colored, wherein the selected local style is related to a style different from a color, and
- wherein the neural network system is obtained based on at least a training process using at least (i) image data to be colored, (ii) extracted information that indicates a local style to be applied to a part of the image data to be colored and is obtained from the predetermined process for extracting a local style, (iii) information indicating a position of the part of the image data to be colored, and (iv) answer image data of the image data to be colored.
5 more elements
A non-transitory computer readable medium storing a program that causes a computer to execute a method of automatic coloring, the method comprising:
- acquiring a target image;
- receiving a designation of (i) texture to be applied to a first part of the target image and (ii) a position of the first part of the target image, wherein the designation of texture is not applied to another part of the target image;
- coloring at least the first part of the target image so that the texture designated is applied to the first part of the target image, by using a neural network system, wherein the neural network system is a neural network system into which the acquired target image, extracted information of the designated texture that is obtained from a predetermined process based on a user-designated part of a reference image having the texture, and information of the designated position of the first part of the target image are input,
- wherein the neural network system is obtained based on at least a training process using at least (i) image data to be colored, (ii) extracted information of a texture to be applied to a part of the image data to be colored and is obtained from the predetermined process, (iii) information of a position of the part of the image data to be colored, and (iv) answer image data of the image data to be colored.
1 more element
A method of automatic coloring:
- acquiring data of a target to be colored;
- receiving a first local style designation for applying a first selected local style to a first part of the target to be colored and a second local style designation for applying a second selected local style to a second part of the target to be colored;
- and
- performing coloring processing reflecting the first and second local style designations on the data of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the first and second selected local styles applied to the first and second parts of the target respectively, the extracted information being outputted from a predetermined-process for extracting a local style based on a first user-designated part of a first reference image having the first selected local style and a second user-designated part of a second reference image having the second selected local style, and (iii) information indicating positions of the first and second parts to which the first and second selected local styles are applied respectively, into a neural network system for coloring the target to be colored,
- wherein the first and second selected local styles are related, respectively, to first and second styles that are different from a color, and the first and second styles are different from each other, and
- wherein the neural network system is obtained based on at least a training process using at least (i) image data to be colored, (ii) extracted information that indicates a local style to be applied to a part of the image data to be colored and is obtained from the predetermined process for extracting a local style, (iii) information indicating a position of the part of the image data to be colored, and (iv) answer image data of the image data to be colored.
3 more elements
A method of automatic coloring, comprising:
- acquiring data of a target to be colored;
- receiving at least one local style designation for applying a selected local style to at least a first part of the target to be colored, wherein the selected local style is not applied to another part of the target to be colored;
- and
- performing coloring processing reflecting the local style designation on the first part of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the selected local style applied to the first part of the target, the extracted information being extracted from a user-designated part of a reference image having the selected local style, and (iii) information indicating a position of the first part to which the selected local style is applied, into a neural network system for coloring the target to be colored,
- wherein the selected local style is related to a style different from a color, and
- wherein the neural network system is obtained based on at least a training process including:
- generating colored image data by inputting (i) image data to be colored, (ii) extracted information that indicates a local style to be applied to a part of the image data to be colored and is outputted from the predetermined process for extracting a local style, (iii) information indicating a position of the part of the image data to be colored, into the neural network system;
- and
- updating at least a part of the neural network system, based on at least the generated colored image data and answer image data of the image data to be colored.
6 more elements
A method of automatic coloring, comprising:
- acquiring data of a target to be colored;
- receiving at least one local style designation for applying a selected local style to at least a first part of the target to be colored, wherein the selected local style is not applied to another part of the target to be colored;
- and
- performing coloring processing reflecting the local style designation on the first part of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the selected local style applied to the first part of the target, the extracted information being extracted from a user-designated part of a reference image having the selected local style, and (iii) information indicating a position of the first part to which the selected local style is applied, into a neural network system for coloring the target to be colored,
- wherein the selected local style is related to a style different from a color, and
- wherein the neural network system is a decoder obtained based on
- at least:
- a step of executing coloring processing reflecting a local style on image data using the decoder to obtain colored image data, by inputting into the decoder (i) image data to be colored, (ii) extracted information that indicates the local style and is outputted from encoder a predetermined process for extracting a local style and (iii) information indicating a position of a part of the image data to which the local style is applied;
- a step of calculating at least loss of the colored image data with respect to answer image data, using the colored image data obtained by the decoder and the answer image data;
- and
- a step of updating at least a part of parameters of the decoder based on at least the calculated loss.
8 more elements
An automatic coloring device, comprising:
- one or more memories;
- and
- one or more processors configured to:
- acquire data of a target to be colored;
- receive at least one local style designation for applying a selected local style to at least a first part of the acquired data, wherein the selected local style is not applied to another part of the acquired data;
- and
- perform coloring processing reflecting the local style designation on the data of the target to be colored, by inputting (i) the acquired data of the target to be colored, (ii) extracted information indicating the selected local style applied to the first part of the target, the extracted information being obtained from a predetermined process for extracting a local style based on a user-designated part of a reference image having the selected local style, and (iii) information indicating a position of the first part to which the selected local style is applied, into a neural network system for coloring the target to be colored, wherein the selected local style is related to a style different from a color, and
- wherein the neural network system is obtained based on at least a training process including:
- generating colored image data by inputting (i) image data to be colored, (ii) extracted information that indicates a local style to be applied to a part of the image data to be colored and is obtained from the predetermined process for extracting a local style, (iii) information indicating a position of the part of the image data to be colored, into the neural network system;
- and
- updating at least a part of the neural network system, based on at least the generated colored image data and answer image data of the image data to be colored.
8 more elements
A non-transitory computer readable medium storing a program that causes a computer to execute a method of automatic coloring, the method comprising:
- acquiring a target image;
- receiving a designation of (i) texture to be applied to a first part of the target image and (ii) a position of the first part of the target image, wherein the designation of texture is not applied to another part of the target image;
- coloring at least the first part of the target image so that the texture designated is applied to the first part of the target image, by using a neural network system, wherein the neural network system is a neural network system into which the acquired target image, extracted information of the designated texture that is obtained from a predetermined process based on a user-designated part of a reference image having the texture, and information of the designated position of the first part of the target image are input,
- wherein the neural network system is obtained based on at least a training process steps including:
- generating colored image data by inputting (i) image data to be colored, (ii) extracted information of a texture to be applied to a part of the image data to be colored and is obtained from the predetermined process, (iii) information of a position of the part of the image data to be colored, into the neural network system;
- and
- updating at least a part of the neural network system, based on at least the generated colored image data and answer image data of the image data to be colored.
4 more elements
Dependencies are inferred from the claim text.