# US 10,311,115 B2 · Object search method and apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,115 B2 - Document ID: US10311115B2 - Status: Granted - Kind: B2 - Application number: 14/902,227 - Filed: 2014-05-15 - Published: 2016-05-26 - Granted: 2019-06-04 - Earliest claimed: 2014-05-15 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Yan Li; Xiaojuan Li; Wenmei Gao - Web page: https://patentgrep.com/patent/US10311115B2 ## CPC - G06F 16/9538: Presentation of query results - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/2428: {Query predicate definition using graphical user interfaces, including menus and forms (G06F16/2423 takes precedence)} - G06F 16/50: of still image data - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/951: Indexing; Web crawling techniques - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An object search method and apparatus, where the method includes receiving voice input and gesture input that are of a user; determining, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object; extracting characteristic information of the characteristic category from an image area selected by the user by means of the gesture input; and searching for the target object according to the extracted characteristic information and the name of the target object. The solutions provided in the embodiments of the present disclosure can provide a user with a more flexible search manner, and reduce a restriction on an application scenario during a search. ## Claims ### Claim 1 An object search method implemented by a terminal communicatively coupled to a server, the method comprising: receiving, by the terminal, voice input and gesture input from a user; determining, by the terminal, a name of a target object for which the user expects to search and a characteristic category of the target object according to the voice input; determining, by the terminal, whether the name of the target object and the characteristic category of the target object correspond to a preset category; extracting, by the terminal, extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and sending the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category; sending, by the terminal, the name of the target object, the characteristic category of the target object, and the image corresponding to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; receiving, by the terminal, a search result from the server that corresponds to the extracted characteristic information and the image corresponding to the gesture input; and displaying, by the terminal, the search result. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the voice input and the gesture input that are received simultaneously from the user. ### Claim 3 (depends on 1) The method of claim 1, further comprising acquiring an image area from the gesture input of the user by: acquiring the image area that is selected from a designated image corresponding to the gesture input; and using the image area as the image from the user. ### Claim 4 (depends on 1) The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises at least one of: receiving the voice input and the gesture input that are received simultaneously from the user; receiving the voice input from the user, and when no gesture input is performed by the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user; or receiving the gesture input from the user, and when no voice input is performed by the user, instructing the user to perform a voice input operation, and receiving the voice input from the user. ### Claim 5 (depends on 1) The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the voice input of the user, and when no gesture input is performed by the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user. ### Claim 6 (depends on 1) The method of claim 1, wherein receiving the voice input and the gesture input from the user comprises receiving the gesture input of the user, and when no voice input is performed by the user, instructing the user to perform a voice input operation, and receiving the voice input from the user. ### Claim 7 (depends on 1) The method of claim 1, further comprising acquiring an image area selected by the user as the gesture input, and acquiring the image area comprises at least one of: acquiring an image area that is selected from a designated image by the user as the gesture input, and using the image area as the image area selected by the user; or acquiring an image that is obtained by photographing from the user as the gesture input, and using the obtained image as the image area selected by the user. ### Claim 8 (depends on 1) The method of claim 1, further comprising acquiring an image area selected by the user as the gesture input, and acquiring the image area comprises: acquiring an image that is obtained by photographing from the user as the gesture input; and using the obtained image as the image area selected by the user. ### Claim 9 (depends on 1) The method of claim 1, wherein the gesture input from the user comprises a graph of a profile of an irregular shape input by the user. ### Claim 10 (depends on 1) The method of claim 1, wherein the gesture input from the user comprises a brand logo in a photograph taken by the user. ### Claim 11 (depends on 1) The method of claim 1, wherein the gesture input from the user comprises a pixel value selected by the user from a color histogram displayed on the terminal. ### Claim 12 A terminal communicatively coupled to a server, comprising: a receiver configured to receive voice input and gesture input from a user; a processor coupled to the receiver and configured to: determine, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object; determine whether the name of the target object and the characteristic category of the target object correspond to a preset category; extract extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and send the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category; send the name of the target object, the characteristic category of the target object, and the image corresponding to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; receive a search result from the server that corresponds to the extracted characteristic information and the image corresponding to the gesture input; and display the search result. ### Claim 13 (depends on 12) The terminal of claim 12, wherein the receiver is further configured to perform at least one of: receive the voice input and the gesture input that are received simultaneously from the user; receive the voice input from the user, and when no gesture input is received from the user and no image area is selected, instruct the user to perform an operation of selecting an image area, and receive the gesture input from the user; or receive the gesture input from the user, and when no voice input is received from the user, instruct the user to perform a voice input operation, and receive the voice input from the user. ### Claim 14 (depends on 12) The terminal of claim 12, wherein the processor is further configured to perform at least one of: acquire an image area that is selected from a designated image corresponding to the gesture input, and use the image area as the image from the user; or acquire an image that is obtained by photographing from the user as the gesture input, and use the obtained image as the image from the user. ### Claim 15 An object search method implemented by a terminal communicatively coupled to a server, the method comprising: receiving, by the terminal, voice input and gesture input from a user; determining, by the terminal, a name of a target object for which the user expects to search and a characteristic category of the target object according to the voice input; determining, by the terminal, whether the name of the target object and the characteristic category of the target object correspond to a preset category; extracting, by the terminal, extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and sending the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category; sending, by the terminal, the name of the target object, the characteristic category of the target object, and an image area that corresponds to the gesture input to the server to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; and receiving, by the terminal, a search result from the server, the search result being obtained by the server by searching for the target object represented by the name of the target object, and a characteristic of the characteristic category that is of the image area and represented by the category information is used as a search criterion. ### Claim 16 (depends on 15) The method of claim 15, wherein receiving the voice input and the gesture input from the user comprises at least one of: receiving the voice input and the gesture input that are received-simultaneously from the user; receiving the voice input of the user, and when no gesture input is received from the user and no image area is selected, instructing the user to perform an operation of selecting an image area, and receiving the gesture input from the user; or receiving the gesture input from the user, and when no voice input is received from the user, instructing the user to perform a voice input operation, and receiving the voice input from the user. ### Claim 17 (depends on 15) The method of claim 15, further comprising acquiring an image area selected by the user corresponding to the gesture input by: acquiring the image area that is selected from a designated image by the user corresponding to the gesture input, and using the image area as the image area selected by the user; or acquiring an image that is obtained by photographing from the user as the gesture input, and using the obtained image as the image area selected by the user. ### Claim 18 A terminal communicatively coupled to a server, comprising: a receiver configured to receive voice input and gesture input from a user; a processor coupled to the receiver and configured to: determine, according to the voice input, a name of a target object for which the user expects to search and a characteristic category of the target object; determine whether the name of the target object and the characteristic category of the target object correspond to a preset category; extract extracted characteristic information locally on the terminal according to the name of the target object, the characteristic category of the target object, and an image corresponding to the gesture input and send the extracted characteristic information and the name of the target object to the server when the name of the target object and the characteristic category of the target object correspond to the preset category; send, to the server, the name of the target object, the characteristic category of the target object, and an image area that corresponds to the gesture input to enable the server to extract the extracted characteristic information when the name of the target object and the characteristic category of the target object do not correspond to the preset category; and receive a search result from the server, the search result being obtained by the server by searching for the target object represented by the name of the target object, and a characteristic of the characteristic category that is of the image area and represented by the category information is used as a search criterion. ### Claim 19 (depends on 18) The terminal of claim 18, wherein the receiver is further configured to perform at least one of: receive the voice input and the gesture input that are received simultaneously from the user; receive the voice input from the user, and when no gesture input is performed by the user and no image area is selected, instruct the user to perform an operation of selecting an image area, and receive the gesture input from the user; or receive the gesture input from the user, and when no voice input is performed by the user, instruct the user to perform a voice input operation, and receive the voice input from the user. ### Claim 20 (depends on 18) The terminal of claim 18, wherein the processor is further configured to: acquire an image area that is selected from a designated image by the user corresponding to the gesture input, and use the image area as the image area selected by the user; or acquire an image that is obtained from photographing by the user as the gesture input, and use the obtained image as the image area selected by the user.