# US 2021/0278908 A1 · Electronic-Device Interaction Via a Remote User Interface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0278908 A1 - Document ID: US20210278908A1 - Status: Published application - Kind: A1 - Application number: 17/194,319 - Filed: 2021-03-08 - Published: 2021-09-09 - Earliest claimed: 2020-03-09 - Assignees: LayerJot, Inc. - Applicants: LayerJot, Inc. - Inventors: Etay Gafni; Soren Harner - Web page: https://patentgrep.com/patent/US20210278908A1 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/2431: Multiple classes - G06F 18/41: {Interactive pattern learning with a human teacher} - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/235: {based on user input or interaction} - G06V 10/56: relating to colour - G06V 10/764: using classification, e.g. of video objects - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06V 30/10: Character recognition - G06V 2201/034: of medical instruments - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract When an object is detected by an electronic device in a predefined spatial region of a physical mat, the electronic device may perform one or more measurements of the object using the one or more sensors. Note that the physical mat may be on a surface that is separate from the electronic device. Then, the electronic device may identify the object based at least in part on the one or more measurements, where the identification involves a pretrained neural network or a pretrained machine-learning model that uses the one or more measurements as an input and that outputs information specifying the identified object. Moreover, the electronic device may provide classification information associated with the identified object. Next, the electronic device may perform analysis associated with the identified object. For example, the electronic device may increment a count of a number of a type of object that includes the identified object. ## Claims ### Claim 1 An electronic device, comprising: one or more image sensors configured to selectively acquire one or more images of an external environment that comprises an object; an interface circuit configured to communicate with a computer; a processor, coupled to the one or more image sensor, the interface circuit and a memory, configured to execute the program instructions; and the memory configured to store the program instructions, wherein, when executed by the processor, the program instructions cause the electronic device to perform operations comprising: when the object is detected in a predefined spatial region of a physical mat, acquiring the one or more images of the object using the one or more image sensors, wherein the physical mat is disposed on a surface that is separate from the electronic device; identifying the object based at least in part on the one or more images, wherein the identification comprises the use of a pretrained neural network or a pretrained machine-learning model that uses information associated with the one or more images as an input and that outputs information specifying the identified object; providing classification information associated with the identified object; and performing analysis associated with the identified object. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the identifying the object comprises one of: providing the one or more images intended for the computer; and receiving object information associated with the computer, wherein the object information corresponds to the identified object; or determining the object information by performing image analysis on the one or more images. ### Claim 3 (depends on 1) The electronic device of claim 1, wherein the predefined spatial region is specified by multiple predefined markers on the physical mat. ### Claim 4 (depends on 3) The electronic device of claim 3, wherein a given image in the one or more images comprises information associated with at least three of the predefined markers. ### Claim 5 (depends on 3) The electronic device of claim 3, wherein a given predefined marker has a unique spatial pattern relative to remaining predefined markers on the physical mat. ### Claim 6 (depends on 3) The electronic device of claim 3, wherein the predefined markers provide a color reference in different lighting conditions. ### Claim 7 (depends on 1) The electronic device of claim 1, wherein the operations comprise: determining two or more predefined objects that are potential matches for the identified object; providing information specifying the two or more predefined objects; and detecting that a user touched, pointed to or covered up without physical contact one or more of a set of predefined icons in a second predefined spatial region of the physical mat to select one of the two or more predefined objects, wherein a given predefined icon is associated with a given command or instruction, and wherein the classification information is associated with the selected one of the two or more predefined objects. ### Claim 8 (depends on 1) The electronic device of claim 1, wherein the analysis comprises: incrementing a count of a number of a type of object that includes the identified object; determining a state of the identified object; computing an orientation or a spatial arrangement of the identified object relative to one or more other objects in the predefined spatial region; or calculating a numerical value associated with the identified object. ### Claim 9 (depends on 1) The electronic device of claim 1, wherein the identified object comprises a surgical instrument. ### Claim 10 (depends on 1) The electronic device of claim 1, wherein the operations comprise detecting a gesture of a user that specifies at least a portion of the identified object; and wherein the one or more images comprise at least the specified portion of the identified object based at least in part on the detected gesture. ### Claim 11 (depends on 10) The electronic device of claim 10, wherein the gesture corresponds to two fingers on one or both hands of the user. ### Claim 12 (depends on 10) The electronic device of claim 10, wherein the operations comprise performing optical character recognition on at least the specified portion of the identified object. ### Claim 13 (depends on 1) The electronic device of claim 1, wherein the operations comprise: when a second object is detected in a second predefined spatial region of the physical mat, acquiring one or more second images of the second object using the one or more image sensors; identifying the second object based at least in part on the one or more second images; and performing analysis associated with the identified second object. ### Claim 14 (depends on 1) The electronic device of claim 1, wherein the identified object in the predefined spatial region comprises an available surgical instrument, the identified second object in the second predefined spatial region comprises a used surgical instrument, and the analysis comprises maintaining a count of available surgical instruments and used surgical instruments. ### Claim 15 (depends on 1) The electronic device of claim 1, wherein the operations comprise providing a report with results of the analysis, storing information with the results of the analysis, or both. ### Claim 16 A non-transitory computer-readable storage medium for use in conjunction with an electronic device, the computer-readable storage medium storing program instructions that, when executed by the electronic device, causes the electronic device to perform operations comprising: when an object is detected in a predefined spatial region of a physical mat, acquiring one or more images of the object using one or more image sensors in the electronic device, wherein the physical mat is disposed on a surface that is separate from the electronic device; identifying the object based at least in part on the one or more images, wherein the identification comprises the use of a pretrained neural network or a pretrained machine-learning model that uses information associated with the one or more images as an input and that outputs information specifying the identified object; providing classification information associated with the identified object; and performing analysis associated with the identified object. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the predefined spatial region is specified by multiple predefined markers on the physical mat, and a given image in the one or more images comprises information associated with at least three of the predefined markers. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the analysis comprises: incrementing a count of a number of a type of object that includes the identified object; determining a state of the identified object; computing an orientation or a spatial arrangement of the identified object relative to one or more other objects in the predefined spatial region; or calculating a numerical value associated with the identified object. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the operations comprise: when a second object is detected in a second predefined spatial region of the physical mat, acquiring one or more second images of the second object using the one or more image sensors; identifying the second object based at least in part on the one or more second images; and performing analysis associated with the identified second object. ### Claim 20 A method for performing analysis, comprising: by an electronic device: when an object is detected in a predefined spatial region of a physical mat, acquiring one or more images of the object using one or more image sensors in the electronic device, wherein the physical mat is disposed on a surface that is separate from the electronic device; identifying the object based at least in part on the one or more images, wherein the identification comprises the use of a pretrained neural network or a pretrained machine-learning model that uses information associated with the one or more images as an input and that outputs information specifying the identified object; providing classification information associated with the identified object; and performing the analysis associated with the identified object.