# US 2021/0064907 A1 · Cross-domain image processing for object re-identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0064907 A1
- Document ID: US20210064907A1
- Status: Published application
- Kind: A1
- Application number: 16/998,890
- Filed: 2020-08-20
- Published: 2021-03-04
- Earliest claimed: 2019-08-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Xiaodong Yang; Yang Zou; Zhiding Yu; Jan Kautz
- Web page: https://patentgrep.com/patent/US20210064907A1

## CPC

- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
- G06V 10/40: Extraction of image or video features
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G06N 3/048: Activation functions
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Object re-identification refers to a process by which images that contain an object of interest are retrieved from a set of images captured using disparate cameras or in disparate environments. Object re-identification has many useful applications, particularly as it is applied to people (e.g. person tracking). Current re-identification processes rely on convolutional neural networks (CNNs) that learn re-identification for a particular object class from labeled training data specific to a certain domain (e.g. environment), but that do not apply well in other domains. The present disclosure provides cross-domain disentanglement of id-related and id-unrelated factors. In particular, the disentanglement is performed using a labeled image set and an unlabeled image set, respectively captured from different domains but for a same object class. The identification-related features may then be used to train a neural network to perform re-identification of objects in that object class from images captured from the second domain.

## Claims

### Claim 1

A method, comprising:
accessing a plurality of labeled first images captured from a first domain and a plurality of second images captured from a second domain, wherein the plurality of labeled first images and the plurality of second images are associated with the particular object class; and
disentangling identification-related features and identification-unrelated features from the plurality of labeled first images and the plurality of second images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first domain and the second domain differ in at least one of season, background, viewpoint, illumination, or source camera.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of second images are unlabeled.

### Claim 4 (depends on 1)

The method of claim 1, wherein the identification-related features include appearance features.

### Claim 5 (depends on 1)

The method of claim 1, wherein the identification-unrelated features include at least one of pose, position, viewpoint, illumination, or background.

### Claim 6 (depends on 1)

The method of claim 1, wherein the disentangling is performed by:
extracting, by an appearance encoder shared across the first domain and the second domain, the identification-related features from the plurality of labeled first images and the plurality of second images;
extracting, by a first structure encoder specific to the first domain, a first portion of the identification-unrelated features from the plurality of labeled first images; and
extracting, by a second structure encoder specific to the second domain, a second portion of the identification-unrelated features from the plurality of second images.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first domain is a training domain and the second domain is a test domain.

### Claim 8 (depends on 1)

The method of claim 1, wherein the disentangling is performed by:
extracting appearance or structure features of a select labeled first image of the plurality of labeled first images and a select second image of the plurality of second images, and
swapping the appearance or structure features between the select labeled first image and the select second image to generate a pair of cross-domain synthesized images.

### Claim 9 (depends on 8)

The method of claim 8, wherein the disentangling is further performed by:
extracting the appearance or structure features from the pair of cross-domain synthesized images,
swapping the appearance or structure features between the pair of cross-domain synthesized images to reconstruct the select labeled first image and the select second image.

### Claim 10 (depends on 9)

The method of claim 9, wherein the identification-related features are captured using identity labels of the select labeled first image and at least one loss function.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
performing adaptation using the identification-related features, wherein the adaptation aligns feature distributions across the first domain and the second domain.

### Claim 12 (depends on 11)

The method of claim 11, wherein the adaptation is performed using self-training and adversarial alignment.

### Claim 13 (depends on 12)

The method of claim 12, wherein an appearance encoder shared across the first domain and the second domain extracts the identification-related features from the plurality of labeled first images and the plurality of second images, and
wherein during adversarial alignment training, the appearance encoder learns to produce appearance features of which membership between the first domain and the second domain cannot be differentiated to reduce a distance between cross-domain appearance feature distributions.

### Claim 14 (depends on 13)

The method of claim 13, wherein the self-training supports the disentangling by forcing appearance features of different identities to stay apart while reducing intra-class variations of a same identity.

### Claim 15 (depends on 14)

The method of claim 14, wherein performing the adaptation with self-training causes the appearance encoder to learn both domain-invariant features and discriminative features.

### Claim 16 (depends on 15)

The method of claim 15, wherein learning the domain-invariant features and the discriminative features facilitates re-identification for the particular object class in the second domain.

### Claim 17

A neural network, comprising:
a disentangling module configured to:
access a plurality of labeled first images captured from a first domain and a plurality of second images captured from a second domain, wherein the plurality of labeled first images and the plurality of second images are associated with the particular object class, and
disentangle identification-related features and identification-unrelated features from the plurality of labeled first images and the plurality of second images.

### Claim 18 (depends on 17)

The neural network of claim 17, wherein the neural network further comprises:
an adaptation module configured to:
access the identification-related features disentangled by the disentangling module, and
learn both domain-invariant features and discriminative features for use in re-identification for the particular object class in the second domain.

### Claim 19 (depends on 18)

The neural network of claim 18, wherein the adaptation module performs adaptation using self-training and adversarial alignment.

### Claim 20

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:
accessing a plurality of labeled first images captured from a first domain and a plurality of second images captured from a second domain, wherein the plurality of labeled first images and the plurality of second images are associated with the particular object class; and
disentangling identification-related features and identification-unrelated features from the plurality of labeled first images and the plurality of second images.
