# US 2022/0035378 A1 · Image segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035378 A1
- Document ID: US20220035378A1
- Status: Published application
- Kind: A1
- Application number: 17/499,508
- Filed: 2021-10-12
- Published: 2022-02-03
- Earliest claimed: 2019-05-24
- Assignees: Mobileye Vision Technologies Ltd.
- Applicants: Mobileye Vision Technologies Ltd.
- Inventors: Ishay Levi; Asaf Bar Zvi; Yonatan Schwartz; Aaron Siegel
- Web page: https://patentgrep.com/patent/US20220035378A1

## CPC

- G01C 21/26: specially adapted for navigation in a road network
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- B60R 2011/0026: {Windows, e.g. windscreen}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2554/402: Type
- B60W 2554/4026: Cycles
- B60W 2554/4043: Lateral speed
- B60W 2554/801: Lateral distance
- G06T 2207/10012: Stereo images
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 2201/08: Detecting or categorising vehicles
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Systems and methods for navigating a host vehicle are disclosed. In one implementation at least one processor is programmed to receive two or more images captured by a camera of the host vehicle from an environment of the host vehicle; analyze the two or more images to identify a representation of at least a portion of a first object and a representation of at least a portion of a second object; determine a first region of the two or more images associated with the first object and a type of the first object; and determine a second region of the two or more images associated with the second object and type of the second object, wherein the type of the first object is different from the type of the second object.

## Claims

### Claim at position 1

-44. (canceled)

### Claim 45

A navigation system for a host vehicle, the system comprising:
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to:
receive two or more images captured by a camera of the host vehicle from an environment of the host vehicle;
analyze the two or more images to identify a representation of at least a portion of a first object and a representation of at least a portion of a second object;
determine a first region of the two or more images associated with the first object and a type of the first object; and
determine a second region of the two or more images associated with the second object and type of the second object, wherein the type of the first object is different from the type of the second object.

### Claim 46 (depends on 45)

The navigation system of claim 45, wherein the memory further includes instructions that when executed by the circuitry cause the at least one processor to output the type of the first object or the type of the second object.

### Claim 47 (depends on 45)

The navigation system of claim 45, wherein a trained system performs at least a part of the analysis of the two or more images.

### Claim 48 (depends on 47)

The navigation system of claim 47, wherein the trained system includes one or more neural networks.

### Claim 49 (depends on 45)

The navigation system of claim 45, wherein the memory further includes instructions that when executed by the circuitry cause the at least one processor to cause the host vehicle to implement a navigational action based the type of the first object or the type of the second object.

### Claim 50 (depends on 45)

The navigation system of claim 45, wherein the type of the first object is included in a first object class and the type of the second object is included in a second object class, and wherein the first object class and the second object class include mutually exclusive objects.

### Claim 51 (depends on 50)

The navigation system of claim 50, wherein the first object class and the second object class comprise at least one of a vehicle, a pedestrian, an upright immobile object, and road surface.

### Claim 52 (depends on 50)

The navigation system of claim 50, wherein the first object class includes one or more of a car, truck, bus, motorcycle, road, barrier, guardrail, painted road surface, elevated areas relative to road, drivable road surface, or pole.

### Claim 53 (depends on 45)

The navigation system of claim 45, wherein analyzing the two or more images includes analyzing one or more pixels of the two or more images.

### Claim 54 (depends on 53)

The navigation system of claim 53, wherein identifying the representation of the portion of the first object includes identifying at least one pixel in the two or more images associated with the representation of the first object.

### Claim 55 (depends on 54)

The navigation system of claim 54, wherein identifying the representation of the portion of the second object includes identifying at least one pixel in the two or more images associated with the representation of the second object.

### Claim 56 (depends on 55)

The navigation system of claim 55, wherein the at least one pixel in the two or more images associated with the representation of the first object does not include the at least one pixel in the two or more images associated with the representation of the second object.

### Claim 57 (depends on 45)

The navigation system of claim 45, wherein the first object includes a guardrail and the second object incudes a concrete barrier.

### Claim 58 (depends on 45)

The navigation system of claim 45, wherein the first object includes a painted road surface and the second object includes a non-painted road surface.

### Claim 59 (depends on 45)

The navigation system of claim 45, wherein the first region and the second region have different elevations.

### Claim 60 (depends on 59)

The navigation system of claim 59, wherein at least one of the first region or the second region is identified based on the difference in elevations.

### Claim 61 (depends on 59)

The navigation system of claim 59, wherein the two or more images comprises a plurality of images captured at different times and the difference in elevation is determined based on a comparison of pixels between the plurality of images.

### Claim 62 (depends on 45)

The navigation system of claim 45, wherein the first region s a drivable area and the second region is a non-drivable area.

### Claim 63 (depends on 62)

The navigation system of claim 62, wherein the non-drivable area includes an unpaved surface.

### Claim 64 (depends on 62)

The navigation system of claim 62, wherein the non-drivable area includes at least one of a driveway, a sidewalk, or a grassy area.

### Claim 65 (depends on 45)

The navigation system of claim 45, wherein the first object includes a vehicle and the second object includes a pedestrian.

### Claim 66 (depends on 45)

The navigation system of claim 45, wherein identifying the representation of the portion of the first object or the representation of the portion of the second object includes analyzing at least a second image captured by the camera.

### Claim 67 (depends on 66)

The navigation system of claim 66, wherein the second image was captured a predetermined time period after the two or more images.

### Claim 68 (depends on 67)

The navigation system of claim 67, wherein the predetermined me period is based on a frame rate of the camera.

### Claim 69

A navigation system for a host vehicle, the system composing:
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to:
receive at least one image captured by a camera of the host vehicle from an environment of the host vehicle;
analyze the at least one image to identify a representation of at least a portion of a first object and a representation of at least a portion of a second object;
determine, based on the analysis, at least one aspect of a geometry of the first object and at least one aspect of a geometry of the second object;
generate, based on the at least one aspect of the geometry of the first object, a first label associated with a region of the at least one image including the representation of the first object; and
generate, based on the at least one aspect of the geometry of the second object, a second label associated with a region of the at least one image including the representation of the second object.

### Claim 70 (depends on 69)

The navigation system of claim 69, wherein the first label indicates a type of the first object.

### Claim 71 (depends on 69)

The navigation system of claim 69, wherein the first label is associated with an identifier of a toe of the first object.

### Claim 72 (depends on 69)

The navigation system of claim 69, wherein the second label indicates a type of the second object.

### Claim 73 (depends on 69)

The navigation system of claim 69, wherein the second label is associated with an identifier of a type of the second object.

### Claim 74 (depends on 69)

The navigation system of claim 69, wherein the memory further in instructions that when executed by the circuitry cause the at least one processor to output the at least one aspect of the geometry of the first object or the at least one aspect of the geometry of the second object.

### Claim 75 (depends on 69)

The navigation system of claim 69, wherein a trained system performs at least a part of the analysis of the at least one image.

### Claim 76 (depends on 75)

The navigation system of claim 75, wherein the trained system includes one or more neural networks.

### Claim 77 (depends on 69)

The navigation system of claim 69, wherein the first object includes a guardrail and the second object incudes a concrete barrier.

### Claim 78 (depends on 77)

The navigation system of claim 77, wherein the guard rail is identified as a separate object from the concrete barrier based at least in part on a texture of the guard rail determined from the at least one image.

### Claim 79 (depends on 69)

The navigation system of claim 69, wherein the at least one aspect of the geometry of the first object includes a height of the first object and the first label is generated at least in part based on the height.

### Claim 80 (depends on 69)

The navigation system of claim 69, wherein the at least one aspect of the geometry of the second object includes a height of the second object and the second label is generated at least in part based on the height.

### Claim 81 (depends on 69)

The navigation system of claim 69, wherein the type of the first object is included in a first object class and the type of the second object is included in a second object class, and wherein the first object class and the second object class include mutually exclusive objects.

### Claim 82 (depends on 81)

The navigation system of claim 81, wherein each of the first object class and the second object class includes at least one of a lane mark, a curb, or a wall.

### Claim 83 (depends on 69)

The navigation system of claim 69, wherein the memory further includes instructions that when executed b the circuit cause the at least one processor to receive an output from at least one sensor of the host vehicle, and wherein the at least one aspect of the geometry of the first object or the at least one aspect of the geometry of the second object is determined based on the output.

### Claim 84 (depends on 69)

The navigation system of claim 69, wherein the at least one aspect of the geometry of the first object is determined based on a texture of the first object.

### Claim 85 (depends on 69)

The navigation system of claim 69, wherein the at least one aspect of the geometry of the second object is determined based on a texture of the second object.

### Claim 86 (depends on 69)

The navigation system of claim 69, wherein the memory further includes in that when executed by the circuitry cause the at least one processor to determine that at least one of the first object or the second object is a road edge.

### Claim 87 (depends on 86)

The navigation system of claim 86, wherein the memory further includes instructions that when executed by the circuitry cause the at least one processor to determine a type of the road edge.

### Claim 88 (depends on 86)

The navigation system of claim 86, wherein the memory further includes instructions that when executed by the circuitry cause the at least one processor to determine whether the road edge is traversable by the host vehicle.

### Claim 89

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, are configured to cause at least one processor to:
receive two or more images captured by a camera of e host vehicle from an environment of the host vehicle;
analyze the two or more images to identify a representation of at least a portion of a first object and a representation of at least a portion of a second object;
determine a first region of the two or more images associated with the first object and a type of the first object; and
determine a second region of the two or more images associated with the second object and type of the second object, wherein the type of the first object is different from the type of the second object.

### Claim 90

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, are configured to cause at least one processor to:
receive at least one image captured by a camera of the host vehicle an environment of the host vehicle;
analyze the at least one image to identify a representation of at least a portion of a first object and a representation of at least a portion of a second object;
determine, based on the analysis, at least one aspect of a geometry of the first object and at least one aspect of a geometry of the second object;
generate, based on the at least one aspect of the geometry of the first object, a first label associated with a region of the at least one image including the representation of the first object; and
generate, based on the at least one aspect of the geometry of the second object, a second label associated with a region of the at least one image including the representation of the second object.