# US 11,993,283 B2 · System and method for performing graph imputation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,993,283 B2 - Document ID: US11993283B2 - Status: Granted - Kind: B2 - Application number: 17/462,758 - Filed: 2021-08-31 - Published: 2023-03-02 - Granted: 2024-05-28 - Earliest claimed: 2021-08-31 - Assignees: Toyota Motor Engineering & Manufacturing North America, Inc. - Applicants: Toyota Motor Engineering & Manufacturing North America, Inc. - Inventors: Rui Guo; Hongsheng Lu; Prashant Tiwari - Web page: https://patentgrep.com/patent/US11993283B2 ## CPC - B60W 60/001: {Planning or execution of driving tasks} - G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations} - G06F 9/30038: {using a mask} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/09: Supervised learning - G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/4049: Relationship among other objects, e.g. converging dynamic objects - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Described are systems and methods for substituting missing graph information caused during the transmission of graph information from one entity to another. In one example, a system includes a processor and a memory with machine-readable instructions that cause the processor to receive an incoming graph stream comprising a plurality of graphs having detected object information and at least one missing graph, transform the plurality of graphs into incoming embedded vectors having values that represent detected object information of the incoming graph stream, provide the embedded vector that corresponds to the at least one missing graph with a missing graph value, and substitute the missing graph value with a substitute value using a recurrent neural network that interpolates the substitute value using the incoming embedded vectors that are not equal to the missing graph value. ## Claims ### Claim 1 A method comprising steps of: receiving, from an external source, an incoming graph stream comprising a plurality of graphs having detected object information and at least one missing graph; transforming the plurality of graphs into incoming embedded vectors having values that represent detected object information of the incoming graph stream; providing the embedded vector that corresponds to the at least one missing graph with a missing graph value; and substituting the missing graph value with a substitute value using a recurrent neural network that interpolates the substitute value using the incoming embedded vectors that are not equal to the missing graph value. ### Claim 2 (depends on 1) The method of claim 1, wherein the recurrent neural network performs a serial analysis of the incoming embedded vectors that are not equal to the missing graph value to interpolate the substitute value. ### Claim 3 (depends on 2) The method of claim 2, wherein the serial analysis is bi-directional. ### Claim 4 (depends on 1) The method of claim 1, further comprising the step of interpolating the substitute value by copying one of the incoming embedded vectors that is not equal to the missing graph value. ### Claim 5 (depends on 4) The method of claim 4, wherein the one of the incoming embedded vectors copied is sequentially nearest the vector having the missing graph value. ### Claim 6 (depends on 1) The method of claim 1, wherein the external source is a remote vehicle and the detected object information is information regarding one or more objects detected by a perception system of the remote vehicle. ### Claim 7 (depends on 6) The method of claim 6, further comprising the steps: generating a local graph stream comprising a plurality of graphs having detected object information detected by a perception system of an ego vehicle; matching the detected object information from the local graph stream and the incoming graph stream; transforming the plurality of graphs of the local graph stream into local embedded vectors having values that represent detected object information of the local graph stream; concatenating the local embedded vectors and the incoming embedded vectors together to form a global set of embedded vectors; and constructing a global feature space based on the global set of embedded vectors. ### Claim 8 A system comprising: a processor; and a memory in communication with the processor having machine-executable instruction that, when executed by the processor, cause the processor to: receive, from an external source, an incoming graph stream comprising a plurality of graphs having detected object information and at least one missing graph, transform the plurality of graphs into incoming embedded vectors having values that represent detected object information of the incoming graph stream, provide the embedded vector that corresponds to the at least one missing graph with a missing graph value, and substitute the missing graph value with a substitute value using a recurrent neural network that interpolates the substitute value using the incoming embedded vectors that are not equal to the missing graph value. ### Claim 9 (depends on 8) The system of claim 8, wherein the recurrent neural network performs a serial analysis of the incoming embedded vectors that are not equal to the missing graph value to interpolate the substitute value. ### Claim 10 (depends on 9) The system of claim 9, wherein the serial analysis is bi-directional. ### Claim 11 (depends on 8) The system of claim 8, wherein the memory further comprises machine-readable instructions that, when executed by the processor, cause the processor to interpolate the substitute value by copying one of the incoming embedded vectors that is not equal to the missing graph value. ### Claim 12 (depends on 11) The system of claim 11, wherein the one of the incoming embedded vectors copied is sequentially nearest the vector having the missing graph value. ### Claim 13 (depends on 8) The system of claim 8, wherein the external source is a remote vehicle and the detected object information is information regarding one or more objects detected by a perception system of the remote vehicle. ### Claim 14 (depends on 13) The system of claim 13, wherein the memory further comprises machine-readable instructions that, when executed by the processor, cause the processor to: generate local graph stream comprising a plurality of graphs having detected object information detected by a perception system of an ego vehicle; match the detected object information from the local graph stream and the incoming graph stream; transform the plurality of graphs of the local graph stream into local embedded vectors having values that represent detected object information of the local graph stream; concatenate the local embedded vectors and the incoming embedded vectors together to form a global set of embedded vectors; and construct a global feature space based on the global set of embedded vectors. ### Claim 15 A non-transitory computer readable medium having instructions that, when executed by a processor, cause the processor to: receive, from an external source, an incoming graph stream comprising a plurality of graphs having detected object information and at least one missing graph; and transform the plurality of graphs into incoming embedded vectors having values that represent detected object information of the incoming graph stream; provide the embedded vector that corresponds to the at least one missing graph with a missing graph value; and substitute the missing graph value with a substitute value using a recurrent neural network that interpolates the substitute value using the incoming embedded vectors that are not equal to the missing graph value. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the recurrent neural network performs a serial analysis of the incoming embedded vectors that are not equal to the missing graph value to interpolate the substitute value. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising instructions that, when executed by the processor, cause the processor to interpolate the substitute value by copying one of the incoming embedded vectors that is not equal to the missing graph value. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the one of the incoming embedded vectors copied is sequentially nearest the vector having the missing graph value. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the external source is a remote vehicle and the detected object information is information regarding one or more objects detected by a perception system of the remote vehicle. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, further comprising instructions that, when executed by the processor, cause the processor to: generate a local graph stream comprising a plurality of graphs having detected object information detected by a perception system of an ego vehicle; match the detected object information from the local graph stream and the incoming graph stream; transform the plurality of graphs of the local graph stream into local embedded vectors having values that represent detected object information of the local graph stream; concatenate the local embedded vectors and the incoming embedded vectors together to form a global set of embedded vectors; and construct a global feature space based on the global set of embedded vectors.