# US 12,001,437 B2 · Generating and using a semantic index > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,437 B2 - Document ID: US12001437B2 - Status: Granted - Kind: B2 - Application number: 17/953,048 - Filed: 2022-09-26 - Published: 2024-03-28 - Granted: 2024-06-04 - Earliest claimed: 2022-09-26 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Eric Chris Wolfgang Sommerlade; Vivek Pradeep; Steven N. Bathiche; Nathan Luquetta-Fish - Web page: https://patentgrep.com/patent/US12001437B2 ## CPC - G06F 16/24575: {using context} - G06F 16/2228: {Indexing structures} - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data. ## Claims ### Claim 1 A method for generating a semantic index, the method comprising: receiving content data, the content data comprising a plurality of subsets of content data; labeling each of the plurality of subsets of content data, based on a semantic context corresponding to the content data; generating a feature vector for each of the labels corresponding to a respective one of the plurality of subsets of content data; storing the plurality of subsets of content data and their corresponding labels; grouping the plurality of subsets of content data, based on their labels, thereby generating one or more groups of subsets of content data, wherein the grouping of the plurality of subsets of content data, based on their labels, comprises: determining a distance between each of the feature vectors; determining that one or more of the distances are less than a predetermined threshold; and grouping together the feature vectors with distances therebetween that are less than the predetermined threshold, thereby grouping together the respective subsets of the plurality of subsets of content data to which the feature vectors correspond; and adapting a computing device to perform an action, based on the one or more groups of subsets of content data. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of subsets of content data and their corresponding labels are stored in a database, and wherein the action comprises returning the database comprising the one or more groups of subsets of content data. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of subsets of content data comprise a first type of content data and a second type of content data. ### Claim 4 (depends on 3) The method of claim 3, wherein the first type of content data and the second type of content data are different types of content data from the group of: audio content data, visual content data, gaze content data, weather content data, news content data, time content data, people content data, and location content data. ### Claim 5 (depends on 1) The method of claim 1, wherein the labeling of each of the plurality of subsets of content data comprises: providing each of the plurality of subsets of content data and the semantic context to a machine-learning model, the machine-learning model comprising at least one of a natural language processor or a vision processor; and receiving, from the machine-learning model, a respective label corresponding to each of the plurality of subsets of content data. ### Claim 6 (depends on 1) The method of claim 1, wherein a timestamp is stored with each of the plurality of subsets of content data and their corresponding labels. ### Claim 7 (depends on 1) The method of claim 1, wherein the action comprises annotating one or more elements on a display of a computing device. ### Claim 8 (depends on 1) The method of claim 1, wherein the action comprises generating an email corresponding to the received content data. ### Claim 9 (depends on 1) The method of claim 1, wherein the action comprises generating a calendar entry corresponding to the received content data. ### Claim 10 (depends on 1) The method of claim 1, wherein the action comprises populating a clipboard with a document corresponding to the received content data. ### Claim 11 A method for retrieving a search result from a semantic index, the method comprising: generating a user-interface; receiving, via the user-interface, a query, the query comprising information corresponding to at least two different content types; and receiving, from a semantic index, a search result corresponding to the query, wherein the semantic index is generated using content data that is stored and labelled, based on semantic context, the content data comprising a plurality of subsets of content data corresponding to the at least two different content types, and the storing and labelling comprising grouping the plurality of subsets of content data by: determining a distance between feature vectors generated for labels corresponding to the plurality of subsets of content data; determining that one or more of the distances are less than a predetermined threshold; and grouping together the feature vectors with distances therebetween that are less than the predetermined threshold, thereby grouping together the respective subsets of the plurality of subsets of content data to which the feature vectors correspond. ### Claim 12 (depends on 11) The method of claim 11, further comprising: prior to receiving the search result, providing a suggested query to reduce an expected number of search results for the query. ### Claim 13 (depends on 11) The method of claim 11, wherein the at least two different content types are from the group of: a person, a time, a location, audio content, visual content, weather, and a device. ### Claim 14 A system for generating a semantic index, the system comprising: at least one processor; memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising: receiving content data, the content data comprising a plurality of subsets of content data, the plurality of subsets of content data comprising a first subset of content data corresponding to a virtual type of content and a second subset of content data corresponding to a physical type of content; labeling each of the plurality of subsets of content data, based on a semantic context corresponding to the content data; generating a feature vector for each of the labels corresponding to a respective one of the plurality of subsets of content data; grouping the plurality of subsets of content data, based on their labels, thereby generating one or more groups of subsets of content data, wherein the grouping of the plurality of subsets of content data, based on their labels, comprises: determining a distance between each of the feature vectors; determining that one or more of the distances are less than a predetermined threshold; and grouping together the feature vectors with distances therebetween that are less than the predetermined threshold, thereby grouping together the respective subsets of the plurality of subsets of content data to which the feature vectors correspond; and performing an action, based on the one or more groups of subsets of content data. ### Claim 15 (depends on 14) The system of claim 14, further comprising one or more of a microphone, a camera, or a global positioning system (GPS), and wherein the second subset of content data comprise one or more from the group of: audio content data, visual content data, gaze content data, and location content data. ### Claim 16 (depends on 14) The system of claim 14, wherein the at least one processor comprises a first processor of a first device and a second processor of a second device, wherein the first subset of the plurality of subsets is received via the first device, and wherein the second subset of the plurality of subsets is received via the second device. ### Claim 17 (depends on 14) The system of claim 14, wherein the labeling of each of the plurality of subsets of content data comprises: providing each of the plurality of subsets of content data and the semantic context to a machine-learning model, the machine-learning model comprising at least one of a natural language processor or a vision processor; and receiving, from the machine-learning model, a respective label corresponding to each of the plurality of subsets of content data. ### Claim 18 (depends on 14) The system of claim 14, wherein the set of operations further comprises: generating a user-interface; receiving, via the user-interface, a query, the query comprising information corresponding to two or more elements, the two or more elements being from at least two different content types; and receiving, from a semantic index comprising the plurality of subsets of content data, a search result corresponding to the query.