# US 2021/0174145 A1 · Multi-lighting conditions rapid onboarding system for visual item classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174145 A1
- Document ID: US20210174145A1
- Status: Published application
- Kind: A1
- Application number: 17/178,164
- Filed: 2021-02-17
- Published: 2021-06-10
- Earliest claimed: 2019-10-29
- Assignees: Accel Robotics Corporation
- Applicants: Accel Robotics Corporation
- Inventors: Marius BUIBAS; John QUINN
- Web page: https://patentgrep.com/patent/US20210174145A1

## CPC

- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06T 7/50: Depth or shape recovery
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60)
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- H04N 23/10: for generating image signals from different wavelengths
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/56: provided with illuminating means
- H04N 23/60: Control of cameras or camera modules
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/20081: Training; Learning
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying background colors, and that builds a classifier training dataset from these images. Background surfaces may for example be coated with retroreflective tape or film, and variable-color incident light sources may generate the desired background colors. Embodiments may automatically rotate or otherwise reorient the item placed in the onboarding system, so that a relatively small number of cameras can capture views from multiple angles. When an item is placed in the system, a fully automated process may generate a sequence of item orientations and background colors, and may capture and process images from the cameras to create training images. Images of the item from multiple angles, under varying lighting conditions, may be captured without requiring an operator to move or reorient the item.

## Claims

### Claim 1

A multi-lighting conditions rapid onboarding system for visual item classification comprising:
an item imaging system into which each item of a plurality of items is placed, said item imaging system comprising
a rotatable mount onto which said each item is placed;
one or more backgrounds, each configured to display a plurality of colors;
a plurality of lights configured to generate said plurality of colors;
a plurality of cameras in different positions, each camera of said plurality of cameras oriented to view said each item when said each item is placed onto said rotatable mount;
at least one controller coupled to said plurality of cameras, to said plurality of lights, and to said rotatable mount, wherein said at least one controller is configured to
transmit lighting commands to said plurality of lights to successively generate each color of said plurality of colors;
transmit rotation commands to said rotatable mount to successively change an orientation of said each item relative to said plurality of cameras when said each item is placed on said rotatable mount; and,
command said plurality of cameras to capture images from said each camera of said plurality of cameras;
an item classifier training system coupled to said item imaging system, said item classifier comprising a processor configured to
generate a training dataset based on said images associated with said each camera and with said each item, wherein said training dataset comprises training images of said each item labeled with an identifier of said each item;
train an item classifier with said training dataset, wherein said item classifier inputs an image and outputs an identity of an item in said image.

### Claim 2 (depends on 1)

The system of claim 1, wherein said plurality of colors comprises two or more different colors, each having a different hue.

### Claim 3 (depends on 1)

The system of claim 1, wherein said one or more backgrounds comprise a reflective material.

### Claim 4 (depends on 3)

The system of claim 3, wherein said reflective material comprises a retroreflective material.

### Claim 5 (depends on 1)

The system of claim 1, wherein said rotatable mount is configured to rotate at least 360 degrees.

### Claim 6 (depends on 1)

The system of claim 1, wherein said rotatable mount comprises a turntable.

### Claim 7 (depends on 1)

The system of claim 1, wherein said rotatable mount comprises an attachment from which said each item is suspended.

### Claim 8 (depends on 1)

The system of claim 1, wherein said rotatable mount comprises a transparent element through which one or more of said plurality of cameras can view said each item.

### Claim 9 (depends on 1)

The system of claim 1, further comprising:
a weight sensor configured to obtain a weight of said each item when said each item is placed onto said rotatable mount.

### Claim 10 (depends on 9)

The system of claim 9, wherein
said training dataset further comprises said weight of said each item; and,
said item classifier further inputs a weight.

### Claim 11 (depends on 1)

The system of claim 1, wherein
none of said plurality of lights is visible in a field of view of any camera of said plurality of cameras; and,
none of said plurality of cameras is visible in said field of view of said any camera of said plurality of cameras.

### Claim 12 (depends on 1)

The system of claim 1, wherein said generate said training dataset comprises
analyze said images from said each camera of said plurality of cameras to obtain a mask of said each item in said images from said each camera of said plurality of cameras;
apply said mask of said each item to said images from said each camera of said plurality of cameras to obtain foreground images of said each item;
generate said training images of said each item based on said foreground images of said each item.

### Claim 13 (depends on 12)

The system of claim 12, wherein said generate said training images of said each item comprises
modify one or more of a scale, a rotation, a color, and an occlusion of said foreground images to form modified foreground images; and,
add said modified foreground images to background images to form said training images.

### Claim 14 (depends on 12)

The system of claim 12, wherein said analyze said images from said each camera of said plurality of cameras to obtain said mask comprises
calculate a hue difference comprising a difference between
a hue channel of a first image of said images from said each camera of said plurality of cameras corresponding to a first color of said plurality of colors, and
a hue channel of a second image of said images from said each camera of said plurality of cameras corresponding to a second color of said plurality of colors different from said first color;
calculate said mask based on a region in said hue difference comprising values below a threshold value.

### Claim 15 (depends on 1)

The system of claim 1, further comprising
an image processor configured to calculate a 3D model of said each item based on said images from said each camera of said plurality of cameras.

### Claim 16 (depends on 15)

The system of claim 15, wherein said image processor is further configured to calculate one or more of a shape, a size, and a volume of said each item.

### Claim 17

A multi-lighting conditions rapid onboarding system for visual item classification comprising:
an item imaging system into which each item of a plurality of items is placed, said item imaging system comprising
a turntable onto which said each item is placed, said turntable configured to rotate at least 360 degrees;
a plurality of lights configured to generate a plurality of colors comprising two or more different colors each having a different hue;
one or more backgrounds, each configured to display said plurality of colors, wherein
each background of said one or more backgrounds comprises a retroreflective material;
a plurality of cameras in different positions, each camera of said plurality of cameras oriented to view said each item when said each item is placed onto said turntable;
a weight sensor configured to obtain a weight of said each item when said each item is placed onto said turntable.
at least one controller coupled to said plurality of cameras, to said plurality of lights, to said turntable, and to said weight sensor, wherein said at least one controller is configured to
transmit lighting commands to said plurality of lights to successively generate each color of said plurality of colors;
transmit rotation commands to said turntable to successively change an orientation of said each item relative to said plurality of cameras when said each item is placed on said turntable;
capture said weight of said each item from said weight sensor; and,
capture images from said each camera of said plurality of cameras;
an item classifier training system coupled to said item imaging system, said item classifier comprising a processor configured to
generate a training dataset based on said images associated with said each camera and with said each item, wherein said training dataset comprises training images of said each item labeled with an identifier of said each item;
train an item classifier with said training dataset, wherein said item classifier inputs an image and outputs an identity of an item in said image; and,
an image processor configured to calculate a 3D model of said each item based on said images from said each camera of said plurality of cameras.
