# US 10,482,029 B1 · Distributed shared memory paging > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,029 B1 - Document ID: US10482029B1 - Status: Granted - Kind: B1 - Application number: 15/664,042 - Filed: 2017-07-31 - Granted: 2019-11-19 - Earliest claimed: 2017-07-31 - Assignees: EMC IP Holding Company LLC - Applicants: EMC IP Holding Company LLC - Inventors: Andrew Chanler; Kevin Tobin - Web page: https://patentgrep.com/patent/US10482029B1 ## CPC - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 12/1009: using page tables, e.g. page table structures - G06F 3/0604: {Improving or facilitating administration, e.g. storage management} - G06F 3/0619: {in relation to data integrity, e.g. data losses, bit errors} - G06F 3/0638: {Organizing or formatting or addressing of data} - G06F 3/0665: {at area level, e.g. provisioning of virtual or logical volumes} - G06F 3/0689: {Disk arrays, e.g. RAID, JBOD} - G06F 12/0868: Data transfer between cache memory and other subsystems, e.g. storage devices or host systems - G06F 12/1063: {the data cache being concurrently virtually addressed} - G06F 12/1475: {in a virtual system, e.g. with translation means} - G06F 2212/222: Non-volatile memory - G06F 2212/657: Virtual address space management - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques for obtaining metadata may include: receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board; querying the local page table for a global memory address of first metadata for the target offset of the logical device; and responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata. The global memory may be a distributed global memory including memory segments from multiple different boards each including its own local page table. Compare and swap operations may be used to perform atomic operations to ensure synchronized access when updating the distributed global memory. ## Claims ### Claim 1 A method of obtaining metadata comprising: receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board; querying the local page table for a global memory address of first metadata for the target offset of the logical device; and responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table. ### Claim 2 (depends on 1) The method of claim 1, wherein the global memory address is an address in a distributed global memory comprising a plurality of global memory segments stored on the plurality of boards, wherein a first of the plurality of global memory segments is included on the board with the director. ### Claim 3 (depends on 2) The method of claim 2, wherein the first indirection layer includes a plurality of pointers to pages of metadata for the logical device, and wherein the first indirection layer is stored in the distributed global memory. ### Claim 4 (depends on 3) The method of claim 3, wherein the pages of metadata for the logical device include the first page and wherein the pages of metadata for the logical device are configured to be paged in and out of the distributed global memory at various points in time. ### Claim 5 (depends on 4) The method of claim 4, wherein each of the plurality of boards includes at least one director that uses a particular one of the plurality of local pages tables, included on said each board, to obtain metadata for at least one logical device. ### Claim 6 (depends on 1) The method of claim 1, wherein the first metadata is used in connection with servicing the I/O operation, and wherein the first metadata includes any of: location information identifying a physical storage area mapped to the target offset of the logical device; cache location information denoting if user data of the target offset of the logical device is stored in cache and if so, identify a location in the cache where the user data is stored; and information used to verify or validate the user data. ### Claim 7 (depends on 5) The method of claim 5, wherein the first page including the first metadata has the corresponding page descriptor stored in the distributed global memory, wherein the page descriptor includes the node mask and further includes: a lock used to control and synchronize access to the corresponding page descriptor; an indicator denoting whether data of the first page stored in the distributed global memory is clean or dirty, wherein dirty indicates that the data of the first page stored in the distributed global memory is a more recent version than another copy of data of the first page stored on non-volatile storage; and a key identifying data stored in the first page, or otherwise denoting that a page of global memory associated with the page descriptor is free. ### Claim 8 (depends on 7) The method of claim 7, wherein the node mask is used in connection with first processing performed to remove the first page from the distributed global memory, wherein the first processing includes selectively sending a request to each of the plurality of boards identified in the node mask, the request requesting that said each board remove the reference to the first global memory address from one of the plurality of local page tables included on said each board. ### Claim 9 (depends on 8) The method of claim 8, wherein a compare and swap operation is performed to write to the distributed global memory in connection with modifying at least one of: the page descriptor, the first indirection layer, and the first metadata of the first page stored at the first global memory address. ### Claim 10 (depends on 1) The method of claim 1, wherein a plurality of indirection layers include the first indirection layer, and wherein the plurality of indirection layers are used to obtain the global memory address of the first metadata. ### Claim 11 (depends on 10) The method of claim 10, wherein the global memory is a distributed global memory, the plurality of indirection layers are stored in the distributed global memory, and wherein at least a second of the plurality of indirection layers includes pages configured to be paged in and out of the distributed global memory. ### Claim 12 (depends on 11) The method of claim 11, wherein the plurality of indirection layers includes global memory addresses of locations in the distributed global memory. ### Claim 13 (depends on 11) The method of claim 11, wherein one of the indirection layers includes global memory addresses storing metadata for the logical device. ### Claim 14 A system comprising: one or more processors; and a memory comprising code stored thereon that, when executed, performs a method of obtaining metadata comprising: receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board; querying the local page table for a global memory address of first metadata for the target offset of the logical device; and responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table. ### Claim 15 A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of obtaining metadata comprising: receiving, by a director, an I/O operation directed to a target offset of a logical device, wherein the director is located on a board including a local page table used by components on the board, wherein the board is one of a plurality of boards, wherein the local page table is one of a plurality of local page tables, and wherein each of the plurality of boards includes a different one of the plurality of local page tables used by components of said each board; querying the local page table for a global memory address of first metadata for the target offset of the logical device; and responsive to the local page table not having the global memory address of the first metadata for the target offset of the logical device, using at least a first indirection layer to obtain the global memory address of the first metadata, wherein the first indirection layer is used to obtain a first page including the first metadata for the logical device, wherein the first page is stored at a first global memory address and wherein the first page has a corresponding page descriptor that includes a node mask identifying which of the plurality of boards includes one of the plurality of local page tables with a reference to the first global memory address stored in the one local page table. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the global memory address is an address in a distributed global memory comprising a plurality of global memory segments stored on the plurality of boards, and wherein a first of the plurality of global memory segments is included on the board with the director. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the first indirection layer includes a plurality of pointers to pages of metadata for the logical device, and wherein the first indirection layer is stored in the distributed global memory. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the pages of metadata for the logical include the first page and wherein the pages of metadata for the logical device are configured to be paged in and out of the distributed global memory at various points in time.