# US 2022/0044153 A1 · Virtualizing external memory as local to a machine learning accelerator

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044153 A1
- Document ID: US20220044153A1
- Status: Published application
- Kind: A1
- Application number: 17/507,188
- Filed: 2021-10-21
- Published: 2022-02-10
- Earliest claimed: 2019-04-29
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Lawrence J. Madar, III; Temitayo Fadelu; Harshit Khaitan; Ravi Narayanaswami
- Web page: https://patentgrep.com/patent/US20220044153A1

## CPC

- G06F 12/1036: for multiple virtual address spaces, e.g. segmentation (G06F12/1045 takes precedence)
- G06N 20/00: Machine learning
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/045: Combinations of networks
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for virtualizing external memory as local to a machine learning accelerator. One ambient computing system comprises: an ambient machine learning engine; a low-power CPU; and an SRAM that is shared among at least the ambient machine learning engine and the low-power CPU; wherein the ambient machine learning engine comprises virtual address logic to translate from virtual addresses generated by the ambient machine learning engine to physical addresses within the SRAM.

## Claims

### Claim 1

A device comprising:
a main memory shared by multiple client devices, wherein the multiple client devices include an ambient computing device,
wherein the ambient computing device comprises multiple ambient processing devices and a shared local memory that is shared by the multiple ambient processing devices, wherein the multiple ambient processing devices include an ambient machine learning (ML) engine,
wherein the ambient computing device includes virtual address logic configured to translate virtual addresses used by the ambient ML engine into physical addresses of the shared local memory.

### Claim 2 (depends on 1)

The device of claim 1, wherein the ambient computing device is configured to process sensor signals before other client devices sharing the main memory are activated from a low-power state.

### Claim 3 (depends on 1)

The device of claim 1, wherein the multiple client devices include a main ML engine that generates physical addresses in the main memory.

### Claim 4 (depends on 1)

The device of claim 1, wherein upon receiving an interrupt, the device is configured to stream model parameters from the main memory into the shared local memory.

### Claim 5 (depends on 4)

The device of claim 4, wherein streaming the model parameters into the shared local memory overwrites space in the shared local memory used by one or more other ambient processing devices.

### Claim 6 (depends on 4)

The device of claim 4, wherein streaming the model parameters into the shared local memory comprises streaming the model parameters into available space in the shared local memory.

### Claim 7 (depends on 4)

The device of claim 4, wherein the interrupt represents receipt of one or more sensor signals to be processed.

### Claim 8 (depends on 4)

The device of claim 4, wherein the ambient ML engine is configured to perform an inference pass of a machine learning model by using the virtual address logic to access the model parameters that were streamed into the shared local memory.

### Claim 9 (depends on 1)

The device of claim 1, wherein the shared local memory comprises multiple memory banks configured to be individually powered down when entering a low-power state.

### Claim 10 (depends on 1)

The device of claim 1, wherein the multiple ambient processing devices comprise at least one of a direct memory access controller, one or more other ML engines, or one or more processors.

### Claim 11 (depends on 1)

The device of claim 1, wherein the ambient ML engine comprises a single ML compute tile.

### Claim 12

A system comprising:
multiple client devices including an ambient computing device; and
a main memory shared by the multiple client devices,
wherein the ambient computing device comprises multiple ambient processing devices and a shared local memory that is shared by the multiple ambient processing devices, wherein the multiple ambient processing devices include an ambient machine learning (ML) engine,
wherein the ambient computing device includes virtual address logic configured to translate virtual addresses used by the ambient ML engine into physical addresses of the shared local memory.

### Claim 13 (depends on 12)

The system of claim 12, wherein the ambient computing device is configured to process sensor signals before other client devices sharing the main memory are activated from a low-power state.

### Claim 14 (depends on 12)

The system of claim 12, wherein the multiple client devices include a main ML engine that generates physical addresses in the main memory.

### Claim 15 (depends on 12)

The system of claim 12, wherein upon receiving an interrupt, the device is configured to stream model parameters from the main memory into the shared local memory.

### Claim 16 (depends on 15)

The system of claim 15, wherein streaming the model parameters into the shared local memory overwrites space in the shared local memory used by one or more other ambient processing devices.

### Claim 17 (depends on 15)

The system of claim 15, wherein streaming the model parameters into the shared local memory comprises streaming the model parameters into available space in the shared local memory.

### Claim 18 (depends on 15)

The system of claim 15, wherein the interrupt represents receipt of one or more sensor signals to be processed.

### Claim 19 (depends on 15)

The system of claim 15, wherein the ambient ML engine is configured to perform an inference pass of a machine learning model by using the virtual address logic to access the model parameters that were streamed into the shared local memory.

### Claim 20 (depends on 12)

The system of claim 12, wherein the shared local memory comprises multiple memory banks configured to be individually powered down when entering a low-power state.