# US 10,373,334 B2 · Computer program, object tracking method, and object tracking device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,334 B2
- Document ID: US10373334B2
- Status: Granted
- Kind: B2
- Application number: 15/785,737
- Filed: 2017-10-17
- Published: 2018-04-19
- Granted: 2019-08-06
- Earliest claimed: 2016-10-19
- Assignees: Seiko Epson Corporation
- Applicants: Seiko Epson Corporation
- Inventors: Alex Levinshtein
- Web page: https://patentgrep.com/patent/US10373334B2

## CPC

- G06T 7/74: {involving reference images or patches}
- G02B 27/0172: {characterised by optical features}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06T 7/75: {involving models}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 2200/04: involving 3D image data
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer program causes an object tracking device to realize functions of: acquiring a first image of a scene including an object captured with a camera positioned at a first position; deriving a 3D pose of the object in a second image captured with the camera positioned at a second position using a 3D model corresponding to the object; deriving 3D scene feature points of the scene based at least on the first image and the second image; obtaining a 3D-2D relationship between 3D points represented in a 3D coordinate system of the 3D model and image feature points on the second image; and updating the derived pose using the 3D-2D relationship, wherein the 3D points include the 3D scene feature points and 3D model points on the 3D model.

## Claims

### Claim 1

A non-transitory computer readable medium that embodies instructions causing one or more processors to perform steps comprising:
acquiring, from a camera, a plurality of image frames of an object and a scene, the scene being a background of the object;
deriving a first 3D pose of the object based on: (i) at least one image in the plurality of image frames, and (ii) at least one 2D template based on a 3D model corresponding to the object without the scene;
deriving 3D scene feature points of the scene based at least on a first image and a second image in the plurality of image frames, the first image being captured with the camera positioned at a first position and the second image being captured with the camera positioned at a second position after the first image is captured;
deriving a second 3D pose of the object based on: (a) the second image, and (b) a most recent 3D pose of the object;
obtaining a 3D-2D relationship between 3D points represented in a 3D coordinate system and image feature points on the second image; and
updating the derived second pose based on the 3D-2D relationship, wherein
the 3D points include the 3D scene feature points and 3D model points on the 3D model.

### Claim 2 (depends on 1)

The non-transitory computer readable medium according to claim 1, wherein deriving the 3D scene feature points is performed based on a triangulation method or a bundle adjustment method.

### Claim 3 (depends on 1)

The non-transitory computer readable medium according to claim 1, wherein the method further comprises:
detecting a 3D pose of the object in the first image.

### Claim 4

A method for tracking an object, comprising:
acquiring, from a camera, a plurality of image frames of an object and a scene, the scene being a background of the object;
deriving, by a processor, a first 3D pose of the object based on: (i) at least one image in the plurality of image frames, and (ii) at least one 2D template based on a 3D model corresponding to the object without the scene;
deriving, by the processor, 3D scene feature points of the scene based at least on a first image and a second image in the plurality of image frames, the first image being captured with the camera positioned at a first position and the second image being captured with the camera positioned at a second position after the first image is captured;
deriving, by the processor, a second 3D pose of the object based on: (a) the second image, and (b) a most recent 3D pose of the object;
obtaining, by the processor, a 3D-2D relationship between 3D points represented in a 3D coordinate system and image feature points on the second image; and
updating, by the processor, the derived second pose based on the 3D-2D relationship, wherein
the 3D points include the 3D scene feature points and 3D model points on the 3D model.

### Claim 5

An object tracking device comprising:
a camera configured to acquire a plurality of image frames of an object and a scene, the scene being a background of the object; and
a processor programmed to:
derive a first 3D pose of the object based on: (i) at least one image in the plurality of image frames, and (ii) at least one 2D template based on a 3D model corresponding to the object without the scene;
derive 3D scene feature points of the scene based at least on the first image and the second image in the plurality of image frames, the first image being captured with the camera positioned at a first position and the second image being captured with the camera positioned at a second position after the first image is captured;
derive a second 3D pose of the object based on: (a) the second image, and (b) a most recent 3D pose of the object;
obtain a 3D-2D relationship between 3D points represented in a 3D coordinate system and image feature points on the second image; and
update the derived second pose based on the 3D-2D relationship, wherein
the 3D points include the 3D scene feature points and 3D model points on the 3D model.