# US 2023/0055978 A1 · Method for controlling vehicle, vehicle and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0055978 A1
- Document ID: US20230055978A1
- Status: Published application
- Kind: A1
- Application number: 17/588,258
- Filed: 2022-01-29
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: Beijing Xiaomi Mobile Software Co., Ltd.
- Applicants: Beijing Xiaomi Mobile Software Co., Ltd.
- Inventors: Rui CAI; He ZHANG; Zhiwei LI; Hao LIU
- Web page: https://patentgrep.com/patent/US20230055978A1

## CPC

- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings
- B60Q 1/1415: {Dimming circuits}
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/141: Control of illumination
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2552/15: Road slope, i.e. the inclination of a road segment in the longitudinal direction
- B60W 2552/35: Road bumpiness, e.g. potholes
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- G06T 2207/30256: Lane; Road marking
- G06T 2207/30261: Obstacle
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for controlling a vehicle is provided. The vehicle includes an image capturing device. The method includes: controlling the image capturing device to collect an image of a scene where the vehicle is located; acquiring projection information of a projection pattern in the image; determining image-altering information corresponding to the projection pattern according to the projection information; and controlling movement of the vehicle according to the image-altering information.

## Claims

### Claim 1

A method for controlling a vehicle comprising an image capturing device, the method comprising:
controlling the image capturing device to collect an image of a scene where the vehicle is located;
acquiring projection information of a projection pattern in the image;
determining image-altering information corresponding to the projection pattern according to the projection information; and
controlling movement of the vehicle according to the image-altering information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the vehicle comprises a plurality of the image capturing devices, and acquiring projection information of a projection pattern in the image comprises:
determining information of a reference pixel point in a reference image, wherein the reference image belongs to a plurality of the images; and
parsing the information of the reference pixel point, to obtain reference projection information of the projection pattern in the reference image, wherein the reference projection information is projection information obtained when the image capturing device for collecting the reference image captures the projection pattern.

### Claim 3 (depends on 2)

The method according to claim 2, wherein determining image-altering information corresponding to the projection pattern according to the projection information comprises:
determining target projection information according to the reference projection information, wherein the target projection information is projection information obtained when the image capturing device for collecting a target image captures the projection pattern, the target image belongs to a plurality of the images, and the reference image is different from the target image; and
determining the image-altering information corresponding to the projection pattern according to the reference projection information and the target projection information.

### Claim 4 (depends on 3)

The method according to claim 3, wherein determining target projection information according to the reference projection information comprises:
determining the target projection information according to the reference projection information and a preset corresponding relationship,
wherein the preset corresponding relationship comprises target projection information corresponding to the reference projection information,
wherein the preset corresponding relationship is obtained in advance by calibrating the projection pattern according to a reference image capturing parameter corresponding to the image capturing device for collecting the reference image and a target image capturing parameter of the image capturing device for collecting the target image.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the vehicle further comprises a projection device, and the method further comprises:
controlling the projection device to project the projection pattern to the scene, prior to controlling the image capturing device to collect an image of a scene where the vehicle is located.

### Claim 6 (depends on 4)

The method according to claim 4, wherein
the reference projection information is a reference pixel coordinate based on a first camera coordinate system, wherein the first camera coordinate system is a coordinate system of the image capturing device for collecting the reference image; and
the target projection information is a target pixel coordinate based on a second camera coordinate system, wherein the second camera coordinate system is a coordinate system of the image capturing device for collecting the target image.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the reference pixel point and a target pixel point respectively correspond to a position point in a space of the scene,
wherein determining the image-altering information corresponding to the projection pattern according to the reference projection information and the target projection information comprises:
performing coordinate conversion on the reference pixel coordinate according to the reference image capturing parameter, to obtain a first point coordinate of the position point based on a vehicle coordinate system, wherein the vehicle coordinate system is a coordinate system where the vehicle is located;
performing coordinate conversion on the target pixel coordinate according to the target image capturing parameter, to obtain a second point coordinate of the position point based on the vehicle coordinate system; and
determining the image-altering information corresponding to the projection pattern according to the first point coordinate and a plurality of the second point coordinates.

### Claim 8 (depends on 7)

The method according to claim 7, wherein determining the image-altering information corresponding to the projection pattern according to the first point coordinate and a plurality of the second point coordinates comprises:
generating point cloud data corresponding to the position point according to the first point coordinate and the plurality of the second point coordinates; and
determining the image-altering information corresponding to the projection pattern according to a plurality of the point cloud data respectively corresponding to a plurality of the position points in the scene.

### Claim 9 (depends on 7)

The method according to claim 7, wherein the position point is located on ground in the scene,
wherein controlling movement of the vehicle according to the image-altering information comprises:
determining terrain information of the ground according to the image-altering information; and
controlling the movement of the vehicle according to the terrain information.

### Claim 10 (depends on 9)

The method according to claim 9, wherein determining terrain information of the ground according to the image-altering information comprises at least one of:
determining ground flatness of the ground according to the image-altering information;
determining a ground slope of the ground according to the image-altering information; and
determining traffic sign information on the ground according to the image-altering information,
wherein at least one of the ground flatness, the ground slope and the traffic sign information is taken as the terrain information.

### Claim 11 (depends on 7)

The method according to claim 7, wherein the position point is located at a target object in the scene,
wherein controlling movement of the vehicle according to the image-altering information comprises:
determining shape information and position information of the target object according to the image-altering information; and
controlling the vehicle to avoid an obstacle according to the shape information and the position information.

### Claim 12

A vehicle, comprising:
a body;
a plurality of image capturing devices, configured to collect an image of a scene where the vehicle is located; and
a control apparatus, connected to each of the plurality of image capturing devices, and configured to execute the method for controlling a vehicle, wherein the method comprises:
controlling the image capturing device to collect an image of a scene where the vehicle is located;
acquiring projection information of a projection pattern in the image;
determining image-altering information corresponding to the projection pattern according to the projection information; and
controlling movement of the vehicle according to the image-altering information.

### Claim 13 (depends on 12)

The vehicle according to claim 12, further comprising:
a projection device, connected to the control apparatus,
wherein the control apparatus is further configured to control the projection device to project a projection pattern to the scene, wherein the projection pattern is used to control movement of the vehicle.

### Claim 14 (depends on 13)

The vehicle according to claim 13, further comprising:
a vehicle lamp for illumination,
wherein the projection device is integrated with the vehicle lamp.

### Claim 15

An electronic device, comprising:
a memory;
a processor; and
a computer program stored in the memory and executable on the processor,
wherein the processor when executing the program achieves a method for controlling a vehicle, wherein the vehicle comprises an image capturing device, wherein the processor is configured to:
control the image capturing device to collect an image of a scene where the vehicle is located;
acquire projection information of a projection pattern in the image;
determine image-altering information corresponding to the projection pattern according to the projection information; and
control movement of the vehicle according to the image-altering information.

### Claim 16 (depends on 15)

The electronic device according to claim 15, wherein the vehicle comprises a plurality of the image capturing devices, and the processor is configured to acquire projection information of a projection pattern in the image by:
determining information of a reference pixel point in a reference image, wherein the reference image belongs to a plurality of the images; and
parsing the information of the reference pixel point, to obtain reference projection information of the projection pattern in the reference image, wherein the reference projection information is projection information obtained when the image capturing device for collecting the reference image captures the projection pattern.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein the processor is configured to:
determine target projection information according to the reference projection information, wherein the target projection information is projection information obtained when the image capturing device for collecting a target image captures the projection pattern, the target image belongs to a plurality of the images, and the reference image is different from the target image; and
determine the image-altering information corresponding to the projection pattern according to the reference projection information and the target projection information.

### Claim 18 (depends on 17)

The electronic device according to claim 17, wherein the processor is configured to:
determine the target projection information according to the reference projection information and a preset corresponding relationship,
wherein the preset corresponding relationship comprises target projection information corresponding to the reference projection information,
wherein the preset corresponding relationship is obtained in advance by calibrating the projection pattern according to a reference image capturing parameter corresponding to the image capturing device for collecting the reference image and a target image capturing parameter of the image capturing device for collecting the target image.

### Claim 19 (depends on 15)

The electronic device according to claim 15, wherein the vehicle further comprises a projection device, and the processor is further configured to
control the projection device to project the projection pattern to the scene, prior to control the image capturing device to collect an image of a scene where the vehicle is located.

### Claim 20 (depends on 19)

The electronic device according to claim 19, wherein
the reference projection information is a reference pixel coordinate based on a first camera coordinate system, wherein the first camera coordinate system is a coordinate system of the image capturing device for collecting the reference image; and
the target projection information is a target pixel coordinate based on a second camera coordinate system, wherein the second camera coordinate system is a coordinate system of the image capturing device for collecting the target image.
