# US 10,769,451 B1 · Sensor mapping to a global coordinate system using a marker grid

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,451 B1
- Document ID: US10769451B1
- Status: Granted
- Kind: B1
- Application number: 16/794,057
- Filed: 2020-02-18
- Granted: 2020-09-08
- Earliest claimed: 2019-10-25
- Assignees: 7-Eleven, Inc.
- Applicants: 7-Eleven, Inc.
- Inventors: Shahmeer Ali Mirza; Sailesh Bharathwaaj Krishnamurthy; Crystal Maung
- Web page: https://patentgrep.com/patent/US10769451B1

## CPC

- G06T 7/292: Multi-camera tracking
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 30/248: {involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" (G06V30/242 takes precedence)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30204: Marker
- G06T 2207/30208: Marker matrix
- G06V 20/44: {Event detection}
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An object tracking system includes a sensor and a tracking system. The sensor is configured to capture a first frame of a global plane for at least a portion of a marker grid in a space. The tracking system is configured to receive a first coordinate in the global plane for a first corner of a marker grid, to determine a second coordinate in the global plane for the first marker on the marker grid, and to determine a third coordinate in the global plane where the second marker on the marker grid. The tracking system is further configured to determine a first pixel location for the first marker, to determine a second pixel location for the second marker, and to generate a homography based on the second coordinate for the first marker, the third coordinate for the second marker, the first pixel location, and the second pixel location.

## Claims

### Claim 1

An object tracking system, comprising:
a first sensor configured to capture a first frame of a global plane for at least a portion of a marker grid in a space, wherein:
the global plane represents (x,y) coordinates for the at least a portion of the space;
the marker grid comprises a first marker and a second marker;
the first marker is a first object that identifies a first location on the marker grid;
the second marker is a second object that identifies a second location on the marker grid;
the first frame comprises a plurality of pixels; and
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column; and
a tracking system operably coupled to the first sensor, comprising:
one or more memories operable to store marker grid information that identifies:
a first offset between a first corner of the marker grid and the first marker; and
a second offset between the first corner of the marker grid and the second marker; and
one or more processors operably coupled to the one or more memories, configured to:
receive a first (x,y) coordinate identifying a first x-value and a first y-value in the global plane where a first corner of a marker grid is located in the space;
determine a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the first marker is located based on the first offset from the first (x,y) coordinate for the first corner of the marker grid;
determine a third (x,y) coordinate identifying a third x-value and a third y-value in the global plane where the second marker is located based on the second offset from the first (x,y) coordinate for the first corner of the marker grid;
receive the first frame;
identify a first pixel within the first frame corresponding with the first marker;
identify a second pixel within the first frame corresponding with the second marker;
determine a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;
determine a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and
generate a first homography based on the second (x,y) coordinate for the first marker, the third (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography maps between pixel locations in the first frame and (x,y) coordinates in the global plane.

### Claim 2 (depends on 1)

The system of claim 1, wherein identifying the first pixel within the first frame comprises identifying a first bounding box for the first marker within the first frame, wherein the first bounding box comprises a first plurality of pixels that contain at least a portion of the first marker.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to:
determine a number of identified markers within the first frame; and
determine that the number of identified markers exceeds a predetermined threshold value; and
wherein identifying the first marker is in response to determining that the number of identified markers exceeds the predetermined threshold value.

### Claim 4 (depends on 1)

The system of claim 1, wherein:
the one or more memories are operable to store a marker dictionary comprising words;
the first marker comprises text; and
identifying the first marker comprises:
identifying text within the first frame;
comparing the identified text to the words in the marker dictionary; and
determining the identified text matches a word in the marker dictionary.

### Claim 5 (depends on 1)

The system of claim 1, wherein:
the one or more memories are operable to store a marker dictionary comprising symbols;
the first marker comprises a symbol; and
identifying the first marker comprises:
identifying a symbol within the first frame;
comparing the identified symbol to the symbols in the marker dictionary; and
determining the identified symbol matches a symbol in the marker dictionary.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to:
receive a fourth (x,y) coordinate identifying a fourth x-value and a fourth y-value in the global plane where a second corner of the marker grid is located in the space;
determine a rotation angle within the global plane based on the first (x,y) coordinate for the first corner of the marker grid and the fourth (x,y) coordinate for the second corner of the marker grid; and
wherein determining the second (x,y) coordinate for the first marker comprises:
applying a translation using the first offset from the first (x,y) coordinate for the first corner of the marker grid; and
applying a rotation using the rotation angle about the first (x,y) coordinate for the first corner of the marker grid.

### Claim 7 (depends on 1)

The system of claim 1, further comprising a second sensor operably coupled to the tracking system, configured to capture a second frame of the at least a portion of the marker grid in the space; and
wherein the one or more processors are further configured to:
determine a third pixel location in the second frame for the first marker;
determine a fourth pixel location in the second frame for the second marker; and
generate a second homography based on the third pixel location, the fourth pixel location, the second (x,y) coordinate for the first marker, and the third (x,y) marker for the second marker, wherein:
the second homography comprises coefficients that translate between the pixel locations in the second frame and (x,y) coordinates in the global plane; and
coefficients of the second homography are different from coefficients of the first homography.

### Claim 8 (depends on 1)

The system of claim 1, further comprising a second sensor operably coupled to the tracking system, configured to capture a second frame of at least a second portion of the space; and wherein:
the first sensor and the second sensor are members of a plurality of sensors configured as a sensor array; and
the sensor array is positioned parallel with the global plane.

### Claim 9 (depends on 1)

The system of claim 1, wherein:
each pixel in the first frame is associated with a pixel value; and
the first homography is further configured to translate between pixel values in the first frame and z-coordinates in the global plane.

### Claim 10 (depends on 1)

The system of claim 1, wherein:
receiving the first (x,y) coordinate comprises receiving a signal identifying the first (x,y) coordinate from a beacon located at the first corner of the marker grid.

### Claim 11 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to:
identify a first pixel marker within the first frame, wherein the first pixel marker is a first light source; and
identify a second pixel marker within the first frame, wherein the second pixel marker is a second light source.

### Claim 12 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to:
identify a first pixel marker within the first frame, wherein the first pixel marker is a first feature of the first marker; and
identify a second pixel marker within the first frame, wherein the second pixel marker is a second feature of the second marker.

### Claim 13

A sensor mapping method, comprising:
receiving a first (x,y) coordinate identifying a first x-value and a first y-value in a global plane where a first corner of a marker grid is located in a space, wherein:
the global plane represents (x,y) coordinates for the at least a portion of the space;
the marker grid comprises a first marker and a second marker;
the first marker is a first object that identifies a first location on the marker grid; and
the second marker is a second object that identifies a second location on the marker grid;
determining a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the first marker is located based on a first offset from the first (x,y) coordinate for the first corner of the marker grid;
determining a third (x,y) coordinate identifying a third x-value and a third y-value in the global plane where the second marker is located based on a second offset from the first (x,y) coordinate for the first corner of the marker grid;
receiving a first frame of the global plane for at least a portion of the marker grid in the space, wherein:
the first frame comprises a plurality of pixels; and
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;
identifying a first pixel within the first frame corresponding with the first marker;
identifying a second pixel within the first frame corresponding with the second marker;
determining a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;
determining a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and
generating a first homography based on the second (x,y) coordinate for the first marker, the third (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography comprises coefficients that translate between pixel locations in the first frame and (x,y) coordinates in the global plane.

### Claim 14 (depends on 13)

The method of claim 13, wherein identifying the first pixel within the first frame comprises identifying a first bounding box for the first marker within the first frame, wherein the first bounding box comprises a first plurality of pixels that contain at least a portion of the first marker.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
determining a number of identified markers within the first frame;
determining that the number of identified markers exceeds a predetermined threshold value; and
identifying the first marker is in response to determining that the number of identified markers exceeds the predetermined threshold value.

### Claim 16 (depends on 13)

The method of claim 13, wherein identifying the first marker comprises:
identifying text within the first frame;
comparing the identified text to words in a marker dictionary; and
determining the identified text matches a word in the marker dictionary.

### Claim 17 (depends on 13)

The method of claim 13, wherein identifying the first marker comprises:
identifying a symbol within the first frame;
comparing the identified symbol to the symbols in a marker dictionary; and
determining the identified symbol matches a symbol in the marker dictionary.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
receiving a fourth (x,y) coordinate identifying a fourth x-value and a fourth y-value in the global plane where a second corner of the marker grid is located in the space;
determining a rotation angle within the global plane based on the first (x,y) coordinate for the first corner of the marker grid and the fourth (x,y) coordinate for the second corner of the marker grid; and
wherein determining the second (x,y) coordinate for the first marker comprises:
applying a translation using the first offset from the first (x,y) coordinate for the first corner of the marker grid; and
applying a rotation using the rotation angle about the first (x,y) coordinate for the first corner of the marker grid.

### Claim 19 (depends on 13)

The method of claim 13, wherein:
each pixel in the first frame is associated with a pixel value; and
the first homography is further configured to translate between pixel values in the first frame and z-coordinates in the global plane.

### Claim 20

A computer program comprising executable instructions stored in a non-transitory computer readable medium that when executed by a processor causes the processor to:
receive a first (x,y) coordinate identifying a first x-value and a first y-value in a global plane where a first corner of a marker grid is located in a space, wherein:
the global plane represents (x,y) coordinates for the at least a portion of the space;
the marker grid comprises a first marker and a second marker;
the first marker is a first object that identifies a first location on the marker grid; and
the second marker is a second object that identifies a second location on the marker grid;
determine a second (x,y) coordinate identifying a second x-value and a second y-value in the global plane where the first marker is located based on a first offset from the first (x,y) coordinate for the first corner of the marker grid;
determine a third (x,y) coordinate identifying a third x-value and a third y-value in the global plane where the second marker is located based on a second offset from the first (x,y) coordinate for the first corner of the marker grid;
receive a first frame of the global plane for at least a portion of the marker grid in the space, wherein:
the first frame comprises a plurality of pixels; and
each pixel from the plurality of pixels is associated with a pixel location comprising a pixel row and a pixel column;
identify a first pixel within the first frame corresponding with the first marker;
identify a second pixel within the first frame corresponding with the second marker;
determine a first pixel location for the first pixel, wherein the first pixel location comprises a first pixel row and a first pixel column of the first frame;
determine a second pixel location for the second pixel, wherein the second pixel location comprises a second pixel row and a second pixel column of the first frame; and
generate a first homography based on the second (x,y) coordinate for the first marker, the third (x,y) coordinate for the second marker, the first pixel location, and the second pixel location, wherein the first homography comprises coefficients that translate between pixel locations in the first frame and (x,y) coordinates in the global plane.