# US 11,432,723 B1 · Method and apparatus for a compact and high resolution mind-view communicator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,432,723 B1 - Document ID: US11432723B1 - Status: Granted - Kind: B1 - Application number: 16/908,514 - Filed: 2020-06-22 - Granted: 2022-09-06 - Earliest claimed: 2009-06-04 - Applicants: Masoud Vaziri - Inventors: Masoud Vaziri - Web page: https://patentgrep.com/patent/US11432723B1 ## CPC - A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet} - A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)} - A61B 5/163: {by tracking eye movement, gaze, or pupil change} - A61B 5/168: {Evaluating attention deficit, hyperactivity} - A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles} - A61N 1/3606: {adapted for a particular treatment} - G06V 40/19: Sensors therefor - G06V 40/193: {Preprocessing; Feature extraction} - G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) - G16H 40/67: for remote operation - H04N 5/76: Television signal recording - H04N 5/775: between a recording apparatus and a television receiver - H04N 7/22: Adaptations for optical transmission - H04N 9/8205: {involving the multiplexing of an additional signal and the colour video signal} - H04N 21/2187: Live feed - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 23/56: provided with illuminating means - H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices - A61B 5/1107: {Measuring contraction of parts of the body, e.g. organ or muscle} - A61B 5/1112: {Global tracking of patients, e.g. by using GPS} - A61B 2560/0242: adapted to measure environmental factors, e.g. temperature, pollution - H04N 7/181: {for receiving images from a plurality of remote sources} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An embodiment of a mind-view communication apparatus includes a first portable unit and a second portable unit. The first portable unit includes an eyeglass frame, at least one first optical unit disposed on the eyeglass frame for capturing at least one scene image corresponding to a field of view of a user, and at least one second optical unit disposed on the eyeglass frame for capturing at least one eye image corresponding to at least a portion of at least one eye of the user. The second portable unit is in communication with the first portable unit and includes at least one processor configured for receiving the at least one scene image and the at least one eye image, determining a direction within the field of view to which the at least one eye is directed based upon the at least one eye image, and generating a subset of the at least one scene image based on the determined direction. ## Claims ### Claim 1 A multi-camera imaging apparatus comprising: optical components configured to be worn by a user, the optical components comprising: a frame configured to be worn on the head of the user; at least one scene camera module disposed on or in the frame and configured to capture scene images of a scene viewed by the user; at least one eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and electronics components configured to be carried or worn by the user, the electronics components comprising at least one processor configured to: receive the scene images from the at least one scene camera module; recognize an object in the scene images to provide object recognition of the object; receive the eye images from the at least one eye tracking module; determine the user is wearing or is the right user of the multi-camera imaging apparatus with first ones of the eye images by detecting existence of or recognizing the eyes of the user in the first ones of the eye images; follow a gaze point of the eyes of the user with second ones of the eye images and independent of the scene images to provide gaze point following of the eyes of the user; and perform an operation when the object is close to or at the gaze point based on the object recognition and the gaze point following. ### Claim 2 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein the object comprises a hand of the user and the operation comprises interpreting a gesture or a movement of the hand of the user in the scene images. ### Claim 3 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein the at least one processor is further configured to follow the gaze point by determining eye axis angles of the eyes of the user from the second ones of the eye images and triangulating to the gaze point based on the eye axis angles. ### Claim 4 (depends on 3) The multi-camera imaging apparatus of claim 3 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the eyes of the user with infrared light while the second ones of the eye images are captured; the second ones of the eye images comprise reflections of the infrared light from the eyes of the user; and the at least one processor is further configured to determine the eye axis angles based on the reflections of the infrared light in the second ones of the eye images. ### Claim 5 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein: the at least one eye tracking module comprises a left eye tracking camera and a right eye tracking camera; the second ones of the eye images comprise left eye images of a left eye of the user captured by the left eye tracking camera and right eye images of a right eye of the user captured by the right eye tracking camera; and the at least one processor is further configured to follow the gaze point by determining a left eye axis angle of the left eye of the user from the left eye images and a right eye axis angle of the right eye of the user from the right eye images and triangulating to the gaze point based on the left eye axis angle and the right eye axis angle. ### Claim 6 (depends on 5) The multi-camera imaging apparatus of claim 5 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the left eye of the user and the right eye of the user with infrared light while the left eye images and the right eye images are captured; the left eye images comprise reflections of the infrared light from the left eye of the user and the right eye images comprise reflections of the infrared light from the right eye of the user; and the at least one processor is further configured to determine the left eye axis angle based on the reflections of the infrared light from the left eye of the user in the left eye images and determine the right eye axis angle based on the reflections of the infrared light from the right eye of the user in the right eye images. ### Claim 7 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein the at least one processor is further configured to recognize the object by determining distances to the object using the scene images. ### Claim 8 (depends on 1) The multi-camera imaging apparatus of claim 1 further comprising: at least one sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the at least one sensor; detect an event or condition based on the sensor data; and control an aspect of the multi-camera imaging apparatus based on the detection of the event or condition. ### Claim 9 (depends on 8) The multi-camera imaging apparatus of claim 8 wherein the at least one sensor is one of a brainwave sensor, an attention sensor, a temperature sensor, a location sensor, an image sensor, an acceleration sensor, a microphone, and a wireless receiver. ### Claim 10 (depends on 8) The multi-camera imaging apparatus of claim 8 wherein the control the aspect of the multi-camera imaging apparatus comprises increase frame rate of the at least one scene camera in capturing the scene images. ### Claim 11 (depends on 10) The multi-camera imaging apparatus of claim 10 wherein the at least one sensor comprises an acceleration sensor, the sensor data comprises acceleration data, and the event or condition comprises acceleration. ### Claim 12 (depends on 8) The multi-camera imaging apparatus of claim 8 wherein the control the aspect of the multi-camera imaging apparatus comprises at least one of initiate the recognize the object, initiate the follow the gaze point, and initiate the perform the operation. ### Claim 13 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein the at least one scene camera module is at least one of a color camera, a black and white camera, an infrared camera, and a night vision camera. ### Claim 14 (depends on 1) The multi-camera imaging apparatus of claim 1 wherein the object comprises a shape and the operation comprises indexing of the shape. ### Claim 15 A multi-camera imaging apparatus comprising: optical components configured to be worn by a user, the optical components comprising: a frame configured to be worn on the head of the user; at least one scene camera module disposed on or in the frame and configured to capture scene images of a scene viewed by the user; at least one eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and electronics components configured to be carried or worn by the user, the electronics components comprising at least one processor configured to: receive the scene images from the at least one scene camera module; recognize an object in the scene images to provide object recognition of the object; receive the eye images from the at least one eye tracking module; determine the user is wearing or is the right user of the multi-camera imaging apparatus with first ones of the eye images by detecting existence of or recognizing the eyes of the user in the first ones of the eye images; follow the gaze point of the eyes of the user with second ones of the eye images and independent of the scene images to provide gaze point following of the eyes of the user; and provide the object recognition and the gaze point following for use by software that is executed by the at least one processor and performs an operation when the object is close to or at the gaze point based on the object recognition and the gaze point following. ### Claim 16 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the multi-camera imaging apparatus further comprises the software. ### Claim 17 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the object comprises a hand of the user and the operation comprises interpreting a gesture or a movement of the hand of the user in the scene images. ### Claim 18 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the at least one processor is further configured to follow the gaze point by determining eye axis angles of the eyes of the user from the second ones of the eye images and triangulating to the gaze point based on the eye axis angles. ### Claim 19 (depends on 18) The multi-camera imaging apparatus of claim 18 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the eyes of the user with infrared light while the second ones of the eye images are captured; the second ones of the eye images comprise reflections of the infrared light from the eyes of the user; and the at least one processor is further configured to determine the eye axis angles based on the reflections of the infrared light in the second ones of the eye images. ### Claim 20 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein: the at least one eye tracking module comprises a left eye tracking camera and a right eye tracking camera; the second ones of the eye images comprise left eye images of a left eye of the user captured by the left eye tracking camera and right eye images of a right eye of the user captured by the right eye tracking camera; and the at least one processor is further configured to follow the gaze point by determining a left eye axis angle of the left eye of the user from the left eye images and a right eye axis angle of the right eye of the user from the right eye images and triangulating to the gaze point based on the left eye axis angle and the right eye axis angle. ### Claim 21 (depends on 20) The multi-camera imaging apparatus of claim 20 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the left eye of the user and the right eye of the user with infrared light while the left eye images and the right eye images are captured; the left eye images comprise reflections of the infrared light from the left eye of the user and the right eye images comprise reflections of the infrared light from the right eye of the user; and the at least one processor is further configured to determine the left eye axis angle based on the reflections of the infrared light from the left eye of the user in the left eye images and determine the right eye axis angle based on the reflections of the infrared light from the right eye of the user in the right eye images. ### Claim 22 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the at least one processor is further configured to recognize the object by determining distances to the object using the scene images. ### Claim 23 (depends on 15) The multi-camera imaging apparatus of claim 15 further comprising: at least one sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the at least one sensor; detect an event or condition based on the sensor data; and control an aspect of the multi-camera imaging apparatus based on the detection of the event or condition. ### Claim 24 (depends on 23) The multi-camera imaging apparatus of claim 23 wherein the at least one sensor is one of a brainwave sensor, an attention sensor, a temperature sensor, a location sensor, an image sensor, an acceleration sensor, a microphone, and a wireless receiver. ### Claim 25 (depends on 23) The multi-camera imaging apparatus of claim 23 wherein the control the aspect of the multi-camera imaging apparatus comprises increase frame rate of the at least one scene camera in capturing the scene images. ### Claim 26 (depends on 25) The multi-camera imaging apparatus of claim 25 wherein the at least one sensor comprises an acceleration sensor, the sensor data comprises acceleration data, and the event or condition comprises acceleration. ### Claim 27 (depends on 23) The multi-camera imaging apparatus of claim 23 wherein the control the aspect of the multi-camera imaging apparatus comprises at least one of initiate the recognize the object, initiate the follow the gaze point, and execute the software. ### Claim 28 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the at least one scene camera module is at least one of a color camera, a black and white camera, an infrared camera, and a night vision camera. ### Claim 29 (depends on 15) The multi-camera imaging apparatus of claim 15 wherein the object comprises a shape and the operation comprises indexing of the shape. ### Claim 30 A multi-camera imaging apparatus comprising: optical components configured to be worn by a user, the optical components comprising: a frame configured to be worn on the head of the user; at least one scene camera module disposed on or in the frame and configured to capture scene images of a scene viewed by the user; at least one eye tracking module disposed on or in the frame and configured to capture eye images of the eyes of the user; and electronics components configured to be carried or worn by the user, the electronics components comprising at least one processor configured to: receive the scene images from the at least one scene camera module; recognize an object in the scene images to provide object recognition of the object; receive the eye images from the at least one eye tracking module; determine the user is wearing or is the right user of the multi-camera imaging apparatus with first ones of the eye images by detecting existence of the eyes of the user in the first ones of the eye images; follow the gaze point of the eyes of the user with second ones of the eye images and independent of the scene images to provide gaze point following of the eyes of the user; and execute software that uses the object recognition and the gaze point following to perform an operation when the object is close to or at the gaze point based on the object recognition and the gaze point following. ### Claim 31 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the multi-camera imaging apparatus further comprises the software. ### Claim 32 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the object comprises a hand of the user and the operation comprises interpreting a gesture or a movement of the hand of the user in the scene images. ### Claim 33 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the at least one processor is further configured to follow the gaze point by determining eye axis angles of the eyes of the user from the second ones of the eye images and triangulating to the gaze point based on the eye axis angles. ### Claim 34 (depends on 33) The multi-camera imaging apparatus of claim 33 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the eyes of the user with infrared light while the second ones of the eye images are captured; the second ones of the eye images comprise reflections of the infrared light from the eyes of the user; and the at least one processor is further configured to determine the eye axis angles based on the reflections of the infrared light in the second ones of the eye images. ### Claim 35 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein: the at least one eye tracking module comprises a left eye tracking camera and a right eye tracking camera; the second ones of the eye images comprise left eye images of a left eye of the user captured by the left eye tracking camera and right eye images of a right eye of the user captured by the right eye tracking camera; and the at least one processor is further configured to follow the gaze point by determining a left eye axis angle of the left eye of the user from the left eye images and a right eye axis angle of the right eye of the user from the right eye images and triangulating to the gaze point based on the left eye axis angle and the right eye axis angle. ### Claim 36 (depends on 35) The multi-camera imaging apparatus of claim 35 wherein: the at least one eye tracking module comprises at least one infrared light source disposed on the frame and configured to illuminate the left eye of the user and the right eye of the user with infrared light while the left eye images and the right eye images are captured; the left eye images comprise reflections of the infrared light from the left eye of the user and the right eye images comprise reflections of the infrared light from the right eye of the user; and the at least one processor is further configured to determine the left eye axis angle based on the reflections of the infrared light from the left eye of the user in the left eye images and determine the right eye axis angle based on the reflections of the infrared light from the right eye of the user in the right eye images. ### Claim 37 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the at least one processor is further configured to recognize the object by determining distances to the object using the scene images. ### Claim 38 (depends on 30) The multi-camera imaging apparatus of claim 30 further comprising: at least one sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the at least one sensor; detect an event or condition based on the sensor data; and control an aspect of the multi-camera imaging apparatus based on the detection of the event or condition. ### Claim 39 (depends on 38) The multi-camera imaging apparatus of claim 38 wherein the at least one sensor is one of a brainwave sensor, an attention sensor, a temperature sensor, a location sensor, an image sensor, an acceleration sensor, a microphone, and a wireless receiver. ### Claim 40 (depends on 38) The multi-camera imaging apparatus of claim 38 wherein the control the aspect of the multi-camera imaging apparatus comprises increase frame rate of the at least one scene camera in capturing the scene images. ### Claim 41 (depends on 40) The multi-camera imaging apparatus of claim 40 wherein the at least one sensor comprises an acceleration sensor, the sensor data comprises acceleration data, and the event or condition comprises acceleration. ### Claim 42 (depends on 38) The multi-camera imaging apparatus of claim 38 wherein the control the aspect of the multi-camera imaging apparatus comprises at least one of initiate the recognize the object, initiate the follow the gaze point, and execute the software. ### Claim 43 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the at least one scene camera module is at least one of a color camera, a black and white camera, an infrared camera, and a night vision camera. ### Claim 44 (depends on 30) The multi-camera imaging apparatus of claim 30 wherein the object comprises a shape and the operation comprises indexing of the shape. ### Claim 45 (depends on 1) The multi-camera imaging apparatus of claim 1 further comprising: a location sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the location sensor; and store a location for the object. ### Claim 46 (depends on 8) The multi-camera imaging apparatus of claim 8 wherein the at least one processor is further configured to store the sensor data as metadata associated with at least one of the scene images. ### Claim 47 (depends on 46) The multi-camera imaging apparatus of claim 46 wherein the at least one processor is further configured to index the at least one of the scene images using the metadata. ### Claim 48 (depends on 15) The multi-camera imaging apparatus of claim 15 further comprising: a location sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the location sensor; and store a location for the object. ### Claim 49 (depends on 23) The multi-camera imaging apparatus of claim 23 wherein the at least one processor is further configured to store the sensor data as metadata associated with at least one of the scene images. ### Claim 50 (depends on 49) The multi-camera imaging apparatus of claim 49 wherein the at least one processor and is further configured to index the at least one of the scene images using the metadata. ### Claim 51 (depends on 30) The multi-camera imaging apparatus of claim 30 further comprising: a location sensor disposed on the frame and configured to provide sensor data; wherein the at least one processor is further configured to: receive the sensor data from the location sensor; and store a location for the object. ### Claim 52 (depends on 38) The multi-camera imaging apparatus of claim 38 wherein the at least one processor is further configured to store the sensor data as metadata associated with at least one of the scene images. ### Claim 53 (depends on 52) The multi-camera imaging apparatus of claim 52 wherein the at least one processor is further configured to index the at least one of the scene images using the metadata.