# US 12,001,475 B2 · Mobile image search system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,475 B2 - Document ID: US12001475B2 - Status: Granted - Kind: B2 - Application number: 17/127,460 - Filed: 2020-12-18 - Published: 2021-04-08 - Granted: 2024-06-04 - Earliest claimed: 2016-07-05 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Jonathan Brody - Web page: https://patentgrep.com/patent/US12001475B2 ## CPC - G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information} - G06F 16/587: using geographical or spatial information, e.g. location - G06F 16/9038: Presentation of query results - G06F 18/24: Classification techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 1/0007: {Image acquisition} - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - H04N 23/63: by using electronic viewfinders - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06V 2201/09: Recognition of logos - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems, methods, devices, media, and computer-readable instructions are described for local image tagging and processing in a resource-constrained environment such as a mobile device. In some embodiments, characteristics associated with images are used to determine whether to store content (e.g., images and video clips) as ephemeral content or non-ephemeral content. Based on the determination, the image is stored in a non-ephemeral camera roll storage of the mobile device, or an ephemeral local application storage. Additional storage operations such as encryption or backup copying may additionally be determined and performed based on the analysis of the content. In some embodiments, such images may be indexed, sorted, and searched based on the image tagging operations used to characterize the content. ## Claims ### Claim 1 A mobile device comprising: a display; a memory comprising a content database; an image sensor coupled to the memory and configured to capture image data for a first image; and one or more processors coupled to the memory, the display, and the image sensor, and configured to performing operations comprising: accessing the image data for the first image; processing, by the one or more processors, the image data using a filter convolved across a width and a height of the first image in a neural network to determine one or more characteristics of the image data selected from a set of tags and based at least in part on image content of the first image, the neural network comprising a first layer and a second layer in a first subgraph; determining that the first and second layers in the first subgraph comprises an output of the neural network; associating each tag of the set of tags with the output; storing the one or more characteristics of the image data in the content database; processing a user input requesting a user interface output on the display; presenting the user interface with a first plurality of images comprising the first image, wherein the plurality of images are based on a set of default search content characteristics from the content database; and responsive to determining that the first and second layer do not comprise the output, reserving a location in the memory associated with the image data comprising the one or more characteristics. ### Claim 2 (depends on 1) The device of claim 1 wherein the one or more characteristics of the image data comprises a list of objects identified by machine vision processing of the first image as part of the processing of the image data. ### Claim 3 (depends on 2) The device of claim 2 wherein the user interface comprises a plurality of headers, each header associated with a different set of search criteria and one or more content display areas. ### Claim 4 (depends on 3) The device of claim 3 wherein the one or more processors are further configured to initiate display, in each content display area of the user interface, of search result content associated with the corresponding header for the content display area; and initiate display of a crossfade animation between individual search result content elements for each content display area as part of display of each image or video element for each set of search result content associated with each header. ### Claim 5 (depends on 4) The device of claim 4 wherein the search result content comprises one or more images, one or more video clips displayed within the corresponding display area, and one or more content collections. ### Claim 6 (depends on 4) The device of claim 4 wherein the plurality of headers comprises: a first header associated with a time period search criteria; a second header associated with a location search criteria; and a third header associated with an object based search criteria. ### Claim 7 (depends on 6) The device of claim 6 wherein the location search criteria comprises a taken nearby criteria based on content associated with a threshold distance from a current position of the mobile device; and wherein the time period search criteria comprises a start time and an end time for one or more days of the week. ### Claim 8 (depends on 2) The device of claim 2 wherein the one or more characteristics of the image data comprises a set of objects, an image capture location, and an image capture time. ### Claim 9 (depends on 7) The device of claim 7 wherein the one or more processors are further configured to perform operations comprising: processing a user search input after presentation of the user interface; and in response to the user search input, generating a plurality of sets of suggested results. ### Claim 10 (depends on 9) The device of claim 9 wherein the plurality of sets of suggested search results is based, at least in part, on a set of all characteristics within the content database on the mobile device. ### Claim 11 (depends on 10) The device of claim 10 wherein the plurality of sets of suggested search results are further ordered, at least in part, on a number of content elements associated with each suggested search result of the set of suggested search result. ### Claim 12 (depends on 9) The device of claim 9 wherein the one or more processors are further configured to: receive a user interface password input; and in response to the user interface password input, update the plurality of sets of suggested search results. ### Claim 13 (depends on 1) The device of claim 1 wherein the set of default search content characteristics are periodically updated by a communication from a messaging server system. ### Claim 14 A method comprising: capturing, by an image sensor of a mobile device, image data for a first image; processing, by the one or more processors, the image data using a filter convolved across a width and a height of the first image in a neural network to determine one or more characteristics of the image data selected from a set of tags and based at least in part on image content of the first image, the neural network comprising a first layer and a second layer in a first subgraph; determining that the first and second layers in the first subgraph comprises an output of the neural network; associating each tag of the set of tags with the output; storing the one or more characteristics of the image data in the content database; processing a user input requesting a user interface output on the display; presenting the user interface with a first plurality of images comprising the first image, wherein the plurality of images are based on a set of default search content characteristics from the content database; and responsive to determining that the first and second layer do not comprise the output, reserving a location in the memory associated with the image data comprising the one or more characteristics. ### Claim 15 (depends on 14) The method of claim 14 further comprising: in response to the user input, generating a plurality of sets of suggested results and a plurality of headers, each search result associated with a header of the plurality of headers; wherein the plurality of headers comprises a first header associated with a time period search criteria, a second header associated with a location search criteria, and a third header associated with an object based search criteria. ### Claim 16 (depends on 14) The method of claim 14 further comprising, updating a set of all characteristics within the content database in response to processing of the image data; wherein the plurality of sets of suggested search results is based, at least in part, on the set of all characteristics within the content database on the mobile device. ### Claim 17 A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform operations comprising: accessing the image data for a first image; processing, by the one or more processors, the image data using a filter convolved across a width and a height of the first image in a neural network to determine one or more characteristics of the image data selected from a set of tags and based at least in part on image content of the first image, the neural network comprising a first layer and a second layer in a first subgraph; determining that the first and second layers in the first subgraph comprises an output of the neural network; associating each tag of the set of tags with the output; storing the one or more characteristics of the image data in the content database; processing a user input requesting a user interface output on the display; presenting the user interface with a first plurality of images comprising the first image, wherein the plurality of images are based on a set of default search content characteristics from the content database; and responsive to determining that the first and second layer do not comprise the output, reserving a location in the memory associated with the image data comprising the one or more characteristics. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17 wherein the user interface comprises a plurality of headers, each header associated with a different set of search criteria and one or more content display areas. ### Claim 19 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the different sets of search criteria are set by a communication from a messaging server system. ### Claim 20 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the set of default search content characteristics comprises a private characteristic and a non-private characteristic.