# US 2019/0377989 A1 · Systems and methods for interfacing between software components > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377989 A1 - Document ID: US20190377989A1 - Status: Published application - Kind: A1 - Application number: 16/000,932 - Filed: 2018-06-06 - Published: 2019-12-12 - Earliest claimed: 2018-06-06 - Assignees: Carbyne Ltd. - Applicants: Carbyne Ltd. - Inventors: Alexander DIZENGOF - Web page: https://patentgrep.com/patent/US20190377989A1 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06F 9/448: Execution paradigms, e.g. implementations of programming paradigms - G06F 9/54: Interprogram communication - G06F 11/302: {where the computing system component is a software system} - G06F 18/25: Fusion techniques - G10L 15/08: Speech classification or search - G06F 11/3079: {the data filtering being achieved by reporting only the changes of the monitored data} - G06F 11/3086: {where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract There is provided a computer implemented method for automatically updating an application programming interface (API) according to a user interface (UI), comprising: providing a UI presented on a display of a computing device, analyzing the visual presentation of the UI on the display to identify at least one key-value pair, defining a dynamic data stream based on the at least one key-value pair, providing the dynamic data stream to an API for access by client code, monitoring the UI presented on the display to detect dynamic updates of the at least one key-value pair, and dynamically updating each respective value of each of the at least one key-value pair of the dynamic data stream of the API according to the detected dynamic updates. ## Claims ### Claim 1 A computer implemented method for automatically updating an application programming interface (API) according to a user interface (UI), comprising: monitoring a visual presentation of a dynamic data object included in a UI, the dynamic data object and the UI are presented on a display of a call center computing device; analyzing the visual presentation of the dynamic data object of the UI on the display to identify a plurality of dynamic updates of at least one key-value pair presented within the dynamic data object, wherein the at least one key-value pair defines at least one of a caller phone number and unique device identifier of a call-in to the call center placed by a caller; for each of the plurality of dynamic updates detect one of a plurality of values of a dynamic part of the at least one key-value pair; for each of the plurality of values: generating a dynamic data stream comprising media data acquired from a user device of the caller, using the respective value of the at least one key-value pair, wherein the media data includes at least one member of a group consisting of: location information of the user device, a real time streaming video captured by the user device and a text based chat session conducted on the user device; providing the dynamic data stream to an API for access by a client code executed by the call center computing device. ### Claim 2 (depends on 1) The method of claim 1, wherein the API includes an object class storing the plurality of values the at least one key-value pair. ### Claim 3 (depends on 1) The method of claim 1, wherein the API is provided to the client code based on callback function. ### Claim 4 (depends on 1) The method of claim 1, wherein an API event is generated in response to each detected dynamic update from the plurality of dynamic updates, and a new data object is created according the respective value of the at least one key-value pair corresponding to the respective detected dynamic update. ### Claim 5 (depends on 1) The method of claim 1, wherein the UI comprises a graphical user interface (GUI) that includes graphical objects, and wherein analyzing comprises performing an optical character recognition (OCR) process of the GUI to identify the plurality of values of the at least one key-value pair. ### Claim 6 (depends on 1) The method of claim 1, wherein analyzing comprises analyzing a screen capture of the display presenting the UI by performing OCR to identify the plurality of values of the at least one key-value pair. ### Claim 7 (depends on 1) The method of claim 1, wherein analyzing comprises: monitoring the display on which the UI is presented over a time interval to create a sequences of screen captures; analyzing the sequence of screen captures to identify the plurality of values over the time interval; analyzing the sequence of screen captures to identify at least one value that remains static during the time interval, wherein each at least one value that remains static is located in proximity to the at least one key-value pair that dynamically changes; and designating the at least one key-value pair according to the at least one value that remains static. ### Claim 8 (depends on 7) The method of claim 7, further comprising: marking at least one region of the UI corresponding to each respective key-value pair; and obtaining at least one of a confirmation and an adaption of at least one marked region from a user. ### Claim 9 (depends on 1) The method of claim 1, wherein the API includes only the respective value of the key-value pair, and the key of the key-value pair is not defined or determined by the client code according to an analysis of the value of the key-value pair. ### Claim 10 (depends on 1) The method of claim 1, wherein the client code computes at least one data object according to the at least one key-value pair of the API, and further comprising injecting the computed data object into the UI for presentation on the display of the computing device. ### Claim 11 (depends on 1) The method of claim 1, wherein the client code dynamically computes at least one data object according to the plurality of values, and the computed data is dynamically injected into the UI for dynamic updating of the presentation on the display of the computing device. ### Claim 12 (depends on 1) The method of claim 1, wherein the user device is a mobile device, wherein the media data further comprises voice data of an analogue telephone call originating from the mobile device, wherein the client code fuses the voice data obtained from the API with the real time streaming video obtained from a video camera of the mobile device to create a fused data feed. ### Claim 13 (depends on 1) The method of claim 1, wherein the client code converts an analogue call from a smartphone associated with the caller phone number into a smart-call, wherein the smart-call comprises at least one multi-media data object, and wherein the at least one multi-media data object is the media data and is injected into the UI for presentation on the display of the computing device, wherein the presentation of at least one multi-media data object is dynamically updated. ### Claim 14 (depends on 11) The method of claim 11, further comprising a smart-call GUI presented as an overlay over the UI, wherein the smart-call GUI is presented according to an inactive state when no value is assigned to the at least one key-value pair indicative of no current analogue call, and the smart-call GUI is presented according to an active state that presents at least one multi-media data object. ### Claim 15 (depends on 14) The method of claim 14, wherein the smart-call GUI of the inactive state includes at least one activation marking for activation of a corresponding multi-media data object, and the smart-call GUI of the active state includes at least one termination marking for termination of a corresponding multi-media data object, wherein the smart-call GUI dynamically expands in response to additional activated multi-media data objects and contracts in response to termination of existing multi-media data objects. ### Claim 16 (depends on 13) The method of claim 13, wherein the at least one multi-media data object is based on a multi-media communication application executing on the smartphone. ### Claim 17 (depends on 16) The method of claim 16, wherein the at least one multi-media data object is selected from the group consisting of: a photo captured by a camera of the smartphone, a video streamed from a camera of the smartphone, an image shared by the smartphone, a message sent by a messaging application executed on the smartphone, a geographic location indicating the location of the smartphone, and one way voice transmitted from the smartphone. ### Claim 18 (depends on 1) The method of claim 1, wherein the client code and the UI are executed on the same computing device. ### Claim 19 A computer implemented method for automatically updating an API according to a UI, comprising: monitoring a visual presentation of a UI presented on a display of a call center computing device; analyzing at least one of a structure and a layout of the visual presentation of the UI to identify a plurality of dynamic updates of at least one key-value pair presented within a dynamic data object of the UI, wherein the at least one key-value pair defines at least one of a caller phone number and unique device identifier of a call-in to the call center placed by a caller; for each of the plurality of dynamic updates detect one of a plurality of values of a dynamic part of the at least one key-value pair; for each of the plurality of values: generating a dynamic data stream comprising media data acquired from a user device of the caller, using the respective value of the at least one key-value pair, wherein the media data includes at least one member of a group consisting of: location information of the user device, a real time streaming video captured by the user device and a text based chat session conducted on the user device; providing the dynamic data stream to an API for access by a client code executed by the call center computing device. ### Claim 20 (depends on 19) The method according to claim 19, wherein the analyzing the code comprises: processing a tree representation of the code of the UI to identify at least one text box input object, wherein a value of a certain key-value pair is designated according to a value stored by a certain text box input object, and a key of the certain key-value pair is designated according to the key of the certain text box input object. ### Claim 21 A computer implemented method for automatically updating an API according to a UI, comprising: providing a UI presented on a display of a computing device; providing a mechanism for a user to manually mark at least one region of the UI on the display; analyzing each manually marked at least one region of the UI on the display to identify at least one key-value pair; analyzing a visual presentation of the least one key-value pair to identify a plurality of dynamic updates of the at least one key-value pair, wherein the at least one key-value pair defines at least one of a caller phone number and unique device identifier of a call-in placed to a call center by a caller; for each of the plurality of dynamic updates detect one of a plurality of values of a dynamic part of the at least one key-value pair; for each of the plurality of values: generating a dynamic data stream comprising media data acquired from a user device of the caller, using the respective value of the at least one key-value pair, wherein the media data includes at least one member of a group consisting of: location information of the user device, a real time streaming video captured by the user device and a text based chat session conducted on the user device; providing the dynamic data stream to an API for access by a client code executed by a call center computing device. ### Claim 22 (depends on 21) The method of claim 21, wherein each region is identified with a single marking act executed by the user via the mechanism. ### Claim 23 (depends on 21) The method of claim 21, wherein analyzing comprises receiving a manual entry by the user indicative of at least one key, wherein a certain key-value pair is identified by pairing each respective manually entered key with a corresponding value identified with a certain manually entered region, and wherein monitoring comprises monitoring each manually entered region of the UI to detect dynamic updates of each respective value of each respective key-value pair. ### Claim 24 (depends on 21) The method of claim 21, wherein the mechanism comprises a marking tool for drawing a geometrical shape defining the at least one region as an overlay of the UI. ### Claim 25 (depends on 21) The method of claim 21, wherein the key-value pair of each respective marked region of the UI is identified by: monitoring the respective marked region over a time interval to identify a dynamic value that dynamically changes over the time interval; receiving a manually entered value; and designating the key of the key-value pair according to the manually entered value and the value of the key-value pair according to the dynamic value. ### Claim 26 A computer implemented method for dynamically updating a UI, comprising: monitoring a visual presentation of at least one dynamic data object associated with a UI presented on a display of a call center computing device to detect a change to the at least one dynamic data object, wherein the at least one dynamic data object defines at least one of a caller phone number and unique device identifier of a call-in to the call center placed by a caller; dynamically extracting a current value of the at least one dynamic data object in response to the detected change; dynamically providing the current value of the at least one dynamic data object to client code so as to allow the client code to acquire media data acquired from a user device of the caller, using the current value, wherein the media data includes at least one member of a group consisting of: location information of the user device, a real time streaming video captured by the user device and a text based chat session conducted on the user device; dynamically computing at least one client data object that includes the media-; and dynamically injecting the at least one client data object for presentation of the media together with the UI on the display of the computing device. ### Claim 27 (depends on 26) The method of claim 26, wherein the at least one dynamic data object is indicative of an interactive user event of a user interacting with the at least one client data object. ### Claim 28 (depends on 26) The method of claim 26, wherein the at least one dynamic data object comprises streaming voice data of an analogue telephone call originating from a phone and received by the computing device, wherein the caller phone number is presented on the UI, wherein the client code executes a natural language processing (NLP) code for analyzing key words identified in the streaming voice data, and wherein the client data object is computed according to the analysis of the key words identified in the streaming voice data. ### Claim 29 (depends on 26) The method of claim 26, further comprising analyzing the UI over a time interval to identify at least one static region where the UI remains constant over the time interval, and wherein the at least one client data object is injected into the UI for presentation over the at least one static region. ### Claim 30 (depends on 26) The method of claim 26, wherein in a next iteration, user interactions with the previously injected client data object are monitored to determine whether the previously injected client data object is selected, wherein another client data object is computed according to the user interactions, and wherein the another client data object is currently injected into the UI in response to the user selection of the previously injected client data object. ### Claim 31 (depends on 1) The method of claim 1, wherein the call center computing device is a public-safety answering point (PSAP).