# US 2023/0055434 A1 · Multitouch data fusion > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0055434 A1 - Document ID: US20230055434A1 - Status: Published application - Kind: A1 - Application number: 18/046,352 - Filed: 2022-10-13 - Published: 2023-02-23 - Earliest claimed: 2007-01-07 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Wayne Carl WESTERMAN; John Greer ELIAS - Web page: https://patentgrep.com/patent/US20230055434A1 ## CPC - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06V 40/107: {Static hand or arm} - G06V 40/1365: {Matching; Classification} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation - G06F 2203/04808: Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for performing multi-touch (MT) data fusion is disclosed in which multiple touch inputs occurring at about the same time are received to generating first touch data. Secondary sense data can then be combined with the first touch data to perform operations on an electronic device. The first touch data and the secondary sense data can be time-aligned and interpreted in a time-coherent manner. The first touch data can be refined in accordance with the secondary sense data, or alternatively, the secondary sense data can be interpreted in accordance with the first touch data. Additionally, the first touch data and the secondary sense data can be combined to create a new command. ## Claims ### Claim 1 A data processing system, comprising: a multi-touch surface configured for receiving multiple touch inputs at least partially overlapping in time and generating multi-touch data; one or more sensing devices configured for providing secondary data; and a processor communicatively coupled to the multi-touch surface and the one or more sensing devices, the processor configured for time-aligning the multi-touch data and the secondary data to perform one or more operations on an electronic device. ### Claim 2 (depends on 1) The data processing system of claim 1, the processor further configured for time-aligning the multi-touch data and the secondary data by temporally fusing the secondary data with the multi-touch data to connect the secondary data with the multi-touch data in a time domain. ### Claim 3 (depends on 1) The data processing system of claim 1, the processor further configured for receiving both the multi-touch data and the secondary data in a time domain and within a single context. ### Claim 4 (depends on 3) The data processing system of claim 3, wherein the single context is one of a particular application being executed in the data processing system, a particular mode of operation being performed in the data processing system, and a particular computing platform. ### Claim 5 (depends on 1) The data processing system of claim 1, the processor further configured for: determining movement and chording data from the multi-touch data; and time-aligning the movement and chording data and the secondary data by temporally fusing the secondary data with the movement and chording data to connect the secondary data with the movement and chording data in a time domain. ### Claim 6 (depends on 1) The data processing system of claim 1, wherein the multi-touch data and the secondary data represent separate data streams and multiple events in each data stream, the processor further configured for time-aligning the events in each data stream with each other. ### Claim 7 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to expand a multi-touch operation. ### Claim 8 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to interpret the multi-touch data. ### Claim 9 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to take an action corresponding to an assumption about the multi-touch data. ### Claim 10 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to correct the multi-touch data. ### Claim 11 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to filter the multi-touch data. ### Claim 12 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to determine one or more characteristics of one or more contacts associated with the multiple touch inputs. ### Claim 13 (depends on 1) The data processing system of claim 1, the processor further configured for utilizing the secondary data to determine a specific arrangement of one or more contacts associated with the multiple touch inputs. ### Claim 14 (depends on 1) The data processing system of claim 1, the processor further configured for time-aligning the multi-touch data and finger identification secondary data by temporally fusing the finger identification secondary data with the multi-touch data to assist in electronic device access. ### Claim 15 (depends on 1) The data processing system of claim 1, the processor further configured for time-aligning the multi-touch data and gaze vector secondary data by temporally fusing the gaze vector secondary data with the multi-touch data to integrate hands-free input with the multi-touch data. ### Claim 16 (depends on 1) The data processing system of claim 1, the processor further configured for time-aligning the multi-touch data and biometric secondary data by temporally fusing the biometric secondary data with the multi-touch data to perform one of a game activity, a security activity, and fitness activity. ### Claim 17 (depends on 1) An electronic device comprising the data processing system of claim 1. ### Claim 18 A method for performing data processing, comprising: receiving multiple touch inputs at least partially overlapping in time at a multi-touch surface and generating multi-touch data; receiving secondary data from one or more sensing devices; and time-aligning the multi-touch data and the secondary data to perform one or more operations on an electronic device. ### Claim 19 (depends on 18) The method of claim 18, further comprising time-aligning the multi-touch data and the secondary data by temporally fusing the secondary data with the multi-touch data to connect the secondary data with the multi-touch data in a time domain. ### Claim 20 (depends on 18) The method of claim 18, further comprising receiving both the multi-touch data and the secondary data in a time domain and within a single context. ### Claim 21 (depends on 20) The method of claim 20, wherein the single context is one of a particular application being executed in the data processing system, a particular mode of operation being performed in the data processing system, and a particular computing platform. ### Claim 22 (depends on 18) The method of claim 18, further comprising: determining movement and chording data from the multi-touch data; and time-aligning the movement and chording data and the secondary data by temporally fusing the secondary data with the movement and chording data to connect the secondary data with the movement and chording data in a time domain. ### Claim 23 (depends on 18) The method of claim 18, further comprising time-aligning the multi-touch data and finger identification secondary data by temporally fusing the finger identification secondary data with the multi-touch data to assist in electronic device access. ### Claim 24 (depends on 18) The method of claim 18, further comprising time-aligning the multi-touch data and gaze vector secondary data by temporally fusing the gaze vector secondary data with the multi-touch data to integrate hands-free input with the multi-touch data. ### Claim 25 (depends on 18) The method of claim 18, further comprising time-aligning the multi-touch data and biometric secondary data by temporally fusing the biometric secondary data with the multi-touch data to perform one of a game activity, a security activity, and fitness activity.