# US 12,001,600 B2 · Service glasses with selective data provision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,600 B2
- Document ID: US12001600B2
- Status: Granted
- Kind: B2
- Application number: 17/314,247
- Filed: 2021-05-07
- Published: 2021-11-11
- Granted: 2024-06-04
- Earliest claimed: 2018-11-09
- Assignees: Beckman Coulter, Inc.
- Applicants: Beckman Coulter, Inc.
- Inventors: Darshan Harish Baldev; Rebecca Jean Busacker; Jon Pallage Lindquist, Jr.; Takayuki Mizutani; Benjamin Robert Nelson; Cody Joseph Precord
- Web page: https://patentgrep.com/patent/US12001600B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- A61F 9/04: Eye-masks {; Devices to be worn on the face, not intended for looking through; Eye-pads for sunbathing}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 21/629: {to features or functions of an application}
- G06F 21/84: output devices, e.g. displays or monitors
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G09C 5/00: Ciphering apparatus or methods not provided for in the preceding groups, e.g. involving the concealment or deformation of graphic data such as designs, written or printed messages
- G09G 5/14: Display of multiple viewports
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- H04N 1/4446: {Hiding of documents or document information}
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 23/56: provided with illuminating means
- H04N 23/61: based on recognised objects
- G06F 21/602: {Providing cryptographic facilities or services}
- G06F 2221/2111: Location-sensitive, e.g. geographical location, GPS
- G06F 2221/2149: Restricted operating environment
- G06V 2201/02: Recognising information on displays, dials, clocks
- G08B 13/19686: {Interfaces masking personal details for privacy, e.g. blurring faces, vehicle license plates}
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G 2358/00: Arrangements for display data security
- G09G 2370/02: Networking aspects
- H04B 1/3827: Portable transceivers
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09C: CIPHERING OR DECIPHERING APPARATUS FOR CRYPTOGRAPHIC OR OTHER PURPOSES INVOLVING THE NEED FOR SECRECY
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- H04B: TRANSMISSION

## Abstract

Service for a machine such as a laboratory instrument can be facilitated by capture of images showing the field of view of a user of the machine, and modification of those images to remove confidential information. Modified images can then be provided to service technicians so that the service technicians can have visual information that would allow them to more effectively communicate and interact with the machines' users.

## Claims

### Claim 1

A method comprising:
a) capturing an image of a field of view of a head mounted camera;
b) automatically generating, in near real time, a modified image by performing steps comprising:
i) identifying one or more portions of the image of the field of view of the head mounted camera to be masked, wherein identifying one or more portions of the image of the field of view of the head mounted camera to be masked comprises:
A) identifying one or more display screens in the field of view;
B) for each of the one or more display screens, determining whether an interface is displayed on that display screen that matches an item from a predefined interface library; and
C) for each displayed interface with at least one matching item from the predefined interface library, identifying one or more portions of that interface as portions of the image to be masked based on information from the matching item from the predefined interface library;
and
ii) masking each of the one or more identified portions;
and
c) presenting the modified image, wherein presenting the modified image comprises:
i) displaying the modified image to a remote service technician; or
ii) displaying the modified image on a display of an augmented reality headpiece wherein the head mounted camera is integrated with the augmented reality headpiece;
wherein:
A) the field of view of the head mounted camera is a view of a laboratory comprising one or more laboratory instruments for analyzing patient samples;
B) the predefined interface library is a library comprising interfaces which are adapted to display confidential information related to the patient samples at fixed interface locations; and
C) masking each of the one or more identified portions comprises, for each displayed interface with at least one matching item from the predefined interface library, masking the fixed interface locations at which that interface is adapted to display confidential information related to the patient samples.

### Claim 2 (depends on 1)

The method of claim 1, wherein masking each of the one or more identified portions comprises, for each interface displayed on one of the one or more display screens having at least one matching item from the predefined interface library, overlaying an image corresponding to that interface from the predefined interface library.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
a) the method comprises, prior to capturing the image of the field of view of the head mounted camera, providing one or more notation media exemplar images;
b) identifying one or more portions of the image of the field of view of the head mounted camera to be masked comprises:
i) identifying one or more notation media in the field of view based on information from the notation media exemplar images; and
ii) identifying each of the one or more notation media in the field of view as one of the one or more portions of the image of the field of view of the head mounted camera to be masked.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
a) presenting the modified image comprises transmitting the modified image to the remote service technician;
b) the method comprises:
i) prior to capturing the image of the field of view of the head mounted camera, specifying an imageable area of a laboratory;
ii) determining, after capturing the image of the field of view of the head mounted camera, and based on one or more wireless transceivers located at a border of the imageable area, whether the head mounted camera is located in the imageable area of the laboratory; and
iii) based on a determination that the head mounted camera is not located in the imageable area of the laboratory, automatically deactivating transmission functionality of the head mounted camera.

### Claim 5 (depends on 1)

The method of claim 1, wherein the transmission functionality of the head mounted camera is automatically deactivated by automatically deactivating the head mounted camera.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
a) presenting the modified image comprises displaying the modified image to the remote technician;
b) the method comprises:
i) prior to capturing the image of the field of view of the head mounted camera, specifying an imageable area of a laboratory;
ii) determining, after capturing the image of the field of view of the head mounted camera, and based on distance from a wireless transceiver located inside of the imageable area, whether the head mounted camera is located in the imageable area of the laboratory; and
iii) based on a determination that the head mounted camera is not located in the imageable area of the laboratory, automatically deactivating transmission functionality of the head mounted camera.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
a) presenting the modified image comprises displaying the modified image to the remote service technician; and
b) the method comprises transmitting the modified image to the remote service technician using an internet connection.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
a) presenting the modified image comprises displaying the modified image to the remote service technician;
b) the method comprises, simultaneously with displaying the modified image to the remote service technician, displaying the image of the field of view of the head mounted camera on the display of the augmented reality headpiece wherein the head mounted camera is integrated with the augmented reality headpiece.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
a) the image is captured as part of a video stream;
b) the method comprises transmitting the modified image to the remote service technician; and
c) transmitting the modified image to the remote service technician is performed by transmitting a version of the video stream that includes the modified image rather than the captured image.

### Claim 10 (depends on 1)

The method of claim 1, wherein the image of the field of view of the head mounted camera includes confidential information outside of the one or more portions identified as portions to be masked.

### Claim 11 (depends on 1)

The method of claim 1, wherein the head mounted camera is comprised by a pair of instrumented safety glasses.

### Claim 12 (depends on 1)

The method of claim 1, wherein:
a) presenting the modified image comprises displaying the modified image to the remote technician;
b) the method comprises:
i) the remote technician generating, based on the modified image, a remediation overlay comprising information for addressing a problem in a laboratory instrument in the field of view of the head mounted camera;
and
ii) displaying an image comprising the remediation overlay on a display of the laboratory instrument.

### Claim 13 (depends on 1)

The method of claim 1, wherein masking each of the one or more identified portions comprises, for each interface displayed on one of the one or more display screens having at least one matching item from the predefined interface library, masking portions of that interface at locations indicated by the at least one matching item in the predefined interface library.

### Claim 14

A system comprising:
a) a head mounted camera;
b) a processor;
wherein:
i) the processor is configured with a set of computer instructions operable to, when executed:
A) after the head mounted camera has been activated, determine whether modified images based on images captured by the head mounted camera should be made available for non-immediate viewing;
B) both create a modified image based on an image captured by the head mounted camera and make the modified image available for non-immediate viewing if and only if:
I) a determination is made that the modified image should be made available for non-immediate viewing; and/or
II) both of the following statements are true:
1) a determination was made that modified images based on images captured by the head mounted camera should be made available for non-immediate viewing prior to the image captured by the head mounted camera being captured; and
2) no determination was made that modified images based on images captured by the head mounted camera should not be made available for non-immediate viewing more recently than the most recent determination that modified images based on images captured by the head mounted camera should be made available for non-immediate viewing;
ii) creating the modified image based on the image captured by the head mounted camera comprises:
A) identifying one or more portions of the image captured by the head mounted camera to be masked by performing acts comprising:
I) identifying one ormore display screens in the image captured by the head mounted camera;
II) for each of the one or more display screens, determining whether an interface is displayed on that display screen that matches an item from a predefined interface library; and
III) for each displayed interface with at least one matching item from the predefined interface library, identifying one or more portions of that interface as portions of the image to be masked based on information from the matching item from the predefined interface library;
and
B) masking each of the one or more identified portions;
iii) the field of view of the head mounted camera is a view of a laboratory comprising one or more laboratory instruments for analyzing patient samples;
iv) the predefined interface library is a library comprising interfaces which are adapted to display confidential information related to the patient samples at fixed interface locations; and
v) masking each of the one or more identified portions comprises, for each displayed interface with at least one matching item from the predefined interface library, masking the fixed interface locations at which that interface is adapted to display confidential information related to the patient samples.

### Claim 15 (depends on 14)

The system of claim 14, wherein the computer instructions are operable to, when executed, determine whether images captured by the head mounted camera should be made available for non-immediate viewing by performing a set of steps comprising:
a) determining if the head mounted camera is located within a predefined area; and
b) based on a determination that the head mounted camera is not located within the predefined area, determining that images captured by the head mounted camera should not be made available for non-immediate viewing.

### Claim 16 (depends on 15)

The system of claim 15, wherein:
a) the system comprises one or more wireless transceivers located at a border of the predefined area; and
b) the determination that the head mounted camera is not located within the predefined area is based on detection of the head mounted camera crossing the border of the predefined area.

### Claim 17 (depends on 14)

The system of claim 14, wherein
a) the system comprises a laboratory instrument;
b) the computer instructions are operable to, when executed, determine whether images captured by the head mounted camera should be made available for non-immediate viewing by performing a set of steps comprising:
i) determining an orientation of the head mounted camera relative to the laboratory instrument; and
ii) based on a determination that the orientation of the head mounted camera is offset from the laboratory instrument by 90 degrees or more, determining that images captured by the head mounted camera should not be made available for non-immediate viewing.

### Claim 18 (depends on 14)

The system of claim 14, wherein the system comprises a laboratory instrument configured to encrypt images captured by the head mounted camera.

### Claim 19 (depends on 14)

The system of claim 14, wherein:
a) the head mounted camera comprises an exterior light source;
b) the head mounted camera is configured to activate the exterior light source when the head mounted camera is activated and when a determination is made that images captured by the head mounted camera should be made available for non-immediate viewing; and
c) the head mounted camera is configured to deactivate the exterior light source when the head mounted camera is deactivated and when a determination is made that images captured by the head mounted camera should not be made available for non-immediate viewing.