# US 10,769,495 B2 · Collecting multimodal image editing requests > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,495 B2 - Document ID: US10769495B2 - Status: Granted - Kind: B2 - Application number: 16/052,246 - Filed: 2018-08-01 - Published: 2020-02-06 - Granted: 2020-09-08 - Earliest claimed: 2018-08-01 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Trung Huu Bui; Zhe Lin; Walter Wei-Tuh Chang; Nham Van Le; Franck Dernoncourt - Web page: https://patentgrep.com/patent/US10769495B2 ## CPC - G06N 3/08: Learning methods - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 9/451: Execution arrangements for user interfaces - G06F 16/54: Browsing; Visualisation therefor - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/235: {based on user input or interaction} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G10L 15/063: {Training} - G06T 11/00: Two-dimensional [2D] image generation - G06T 2200/24: involving graphical user interfaces [GUIs] - G10L 2015/0638: {Interactive procedures} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract In implementations of collecting multimodal image editing requests (IERs), a user interface is generated that exposes an image pair including a first image and a second image including at least one edit to the first image. A user simultaneously speaks a voice command and performs a user gesture that describe an edit of the first image used to generate the second image. The user gesture and the voice command are simultaneously recorded and synchronized with timestamps. The voice command is played back, and the user transcribes their voice command based on the play back, creating an exact transcription of their voice command. Audio samples of the voice command with respective timestamps, coordinates of the user gesture with respective timestamps, and a transcription are packaged as a structured data object for use as training data to train a neural network to recognize multimodal IERs in an image editing application. ## Claims ### Claim 1 In a digital medium environment to create multimodal image editing requests, a method implemented by a computing device, the method comprising: displaying a pair of images including a first image and a second image that includes an edit of the first image; presenting an option to skip the pair of images, the option including that the first image and the second image are too similar; recording a multimodal image editing request that includes a user gesture and a voice command that describe the edit of the first image; receiving a user transcription of the voice command; generating a data object in a searchable format, the data object including the voice command, the user gesture, and the user transcription; and training a neural network to recognize the multimodal image editing requests using the data object and the first image as inputs of the neural network and the second image as an output of the neural network. ### Claim 2 (depends on 1) The method as described in claim 1, further comprising playing back the voice command from the recording, wherein the user transcription is generated by a same user who spoke the voice command based on the playing back the voice command. ### Claim 3 (depends on 2) The method as described in claim 2, further comprising displaying a representation of the user gesture simultaneously with the playing back the voice command. ### Claim 4 (depends on 1) The method as described in claim 1, further comprising: displaying a hint for the edit; receiving a user selection of the hint; and indicating in the second image, responsive to the receiving the user selection of the hint, the edit of the first image. ### Claim 5 (depends on 1) The method as described in claim 1, further comprising repeating the displaying, the presenting, the recording, and the receiving the user transcription for a threshold number of image pairs for user compensation for the user gesture, the voice command, and the user transcription of the image pairs. ### Claim 6 (depends on 1) The method as described in claim 1, wherein the recording the user gesture and the voice command includes recording a respective timestamp for each sample of the user gesture generated in the recording and each sample of the voice command generated in the recording. ### Claim 7 (depends on 6) The method as described in claim 6, wherein samples of the user gesture generated in the recording are represented as data tuples including coordinates of the user gesture and the respective timestamp. ### Claim 8 A system for creating multimodal image editing requests implemented by a computing device in a digital medium environment, the system comprising: a memory to maintain a plurality of image pairs, each image pair including a first image and a second image, the second image having been generated from the first image with an edit of the first image, at least one image pair of the plurality of the image pairs being a placebo image pair used to remove user bias, the placebo image pair having the respective second image that is not generated based on an edit of the respective first image; a user interface to sequentially display the plurality of the image pairs by displaying together the respective first image and second image of each image pair; a processor system to implement an editing request creation application at least partially in hardware of the computing device, and for each of the image pairs, the editing request creation application is configured to: record a multimodal image editing request that includes a user gesture and a voice command that describe the edit of the first image; initiate play back of the voice command to generate a user transcription of the voice command; generate a data object in a searchable format, the data object including the voice command, the user gesture, and the user transcription; and train a neural network to recognize the multimodal image editing requests using the data object and the respective first image as inputs of the neural network and the respective second image as an output of the neural network. ### Claim 9 (depends on 8) The system as described in claim 8, wherein the editing request creation application is implemented to generate the data object to include metadata including a user identification, an identification of the computing device, and a length of time the system is in use to create the multimodal image editing requests. ### Claim 10 (depends on 8) The system as described in claim 8, wherein the editing request creation application is implemented to: generate audio samples of the voice command asynchronously from coordinate data of the user gesture; and record timestamps for the audio samples and timestamps for the coordinate data that can be used to synchronize a playing back of the user gesture with a playing back of the voice command. ### Claim 11 (depends on 8) The system as described in claim 8, wherein the editing request creation application is implemented to: compare a threshold time to a time duration between user logins to the system; and automatically play, based on the compare, a video providing user instruction for operating the system, the video including an example image pair, example image editing requests, instructions for skipping the example image pair, and instructions for accessing a hint describing the example image editing requests. ### Claim 12 In a digital medium environment to create multimodal image editing requests, a method implemented by a computing device, the method comprising: displaying a pair of images; a step for recording coordinates of a user gesture and audio samples of a voice command, the user gesture and the voice command representing a multimodal image editing request corresponding to the pair of images; a step for playing back the audio samples of the voice command from the recording; a step for receiving a transcription of the voice command that is user-transcribed from the playing back; a step for enabling, responsive to the receiving the transcription, display of a subsequent pair of images; a step for generating a data object in a searchable format, the data object including the voice command, the user gesture, and the transcription; and a step for training a neural network to recognize the multimodal image editing requests using the data object and a first image of the pair of images as inputs of the neural network and a second image of the pair of images as an output of the neural network. ### Claim 13 (depends on 12) The method as described in claim 12, wherein the step for the recording, the step for the playing back, the step for the receiving, the step for enabling, the step for the generating, and the step for the training are repeated for a plurality of user gestures and voice commands representing a respective plurality of multimodal image editing requests corresponding to the pair of images. ### Claim 14 (depends on 1) The method as described in claim 1, wherein the user gesture is in a shape of an item referenced in the voice command and corresponding to the edit of the first image. ### Claim 15 (depends on 1) The method as described in claim 1, further comprising: receiving a user selection that indicates no more multimodal image editing requests to enter for the pair of images; and displaying, responsive to the receiving the user selection, an additional pair of images including an additional first image and an additional second image that includes an additional edit of the additional first image. ### Claim 16 (depends on 4) The method as described in claim 4, further comprising determining an additional pair of images to display based on the receiving the user selection of the hint, the additional pair of images including an additional first image and an additional second image that includes a same type of edit as the edit of the first image. ### Claim 17 (depends on 8) The system as described in claim 8, wherein the editing request creation application is implemented to receive the user transcription corresponding to a first one of the image pairs to enable the display of a subsequent one of the image pairs in the user interface. ### Claim 18 (depends on 12) The method as described in claim 12, further comprising: a step for initiating a timer once the recording is completed; and wherein the step for playing back the audio samples is responsive to an expiration of the timer. ### Claim 19 (depends on 1) The method as described in claim 1, further comprising displaying a placebo image pair used to remove user bias, the placebo image pair including an additional second image that is not generated based on an edit of an additional first image included in the placebo image pair. ### Claim 20 (depends on 12) The method as described in claim 12, further comprising a step for displaying a placebo image pair used to remove user bias, the placebo image pair including an additional second image that is not generated based on an edit of an additional first image included in the placebo image pair.