# US 2019/0318482 A1 · Sidewalk edge finder system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318482 A1 - Document ID: US20190318482A1 - Status: Published application - Kind: A1 - Application number: 16/388,581 - Filed: 2019-04-18 - Published: 2019-10-17 - Earliest claimed: 2016-10-24 - Assignees: Starship Technologies Oü - Applicants: Starship Technologies Oü - Inventors: Rasmus RAAG - Web page: https://patentgrep.com/patent/US20190318482A1 ## CPC - G06T 7/12: Edge-based segmentation - G06T 7/73: using feature-based methods - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06T 2207/10012: Stereo images - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method includes acquiring at least one image with at least one camera associated with at least one mobile robot; and extracting a plurality of straight lines from the at least one image; creating at least one dataset comprising data related to the plurality of straight lines extracted from the at least one image; forming a plurality of hypotheses for a walkway boundary based on the at least one dataset and determining at least one hypothesis with the highest likelihood of representing a walkway boundary; and using the at least one hypothesis to adjust a direction and/or speed of motion of the at least one mobile robot. ## Claims ### Claim 1 A method comprising: (a) acquiring at least one image with at least one camera associated with at least one mobile robot; and (b) extracting a plurality of lines from the at least one image acquired in (a); (c) creating at least one dataset comprising data related to the plurality of lines extracted from the at least one image in (b); (d) forming a plurality of hypotheses for a walkway boundary based on the at least one dataset and determining at least one hypothesis with the highest likelihood of representing a walkway boundary; and (e) using the at least one hypothesis determined in (d) to navigate the at least one mobile robot. ### Claim 2 (depends on 1) The method of claim 1, wherein said using in (e) comprises using the at least one hypothesis to adjust a direction and/or speed of motion of the at least one mobile robot. ### Claim 3 (depends on 1) The method of claim 1, wherein the dataset is formed by projecting the extracted lines on a certain projection plane. ### Claim 4 (depends on 3) The method of claim 3, wherein the certain projection plane comprises a ground plane defined with respect to the at least one mobile robot. ### Claim 5 (depends on 4) The method of claim 4, wherein the ground plane was determined as part of the creating in (c). ### Claim 6 (depends on 1) The method of claim 1, wherein hypotheses for the walkway boundary are based on data related to different subsets of lines extracted from the at least one image. ### Claim 7 (depends on 1) The method of claim 1, wherein a line is categorized as either a left line or a right line with respect to the at least one mobile robot. ### Claim 8 (depends on 1) The method of claim 1, further comprising making a left partial hypothesis on lines to the left of the at least one mobile robot and a right partial hypothesis on lines to the right of the at least one mobile robot, combining partial hypotheses to make a full hypothesis, and assigning a likelihood value to said full hypothesis. ### Claim 9 (depends on 8) The method of claim 8, wherein the dataset is formed by projecting the extracted lines on a certain projection plane, and wherein the certain projection plane comprises a ground plane defined with respect to the at least one mobile robot, and wherein for each line projected onto the ground plane, two points are extracted, preferably corresponding to endpoints of the line, and each partial hypothesis is based on a different collection of such points. ### Claim 10 (depends on 9) The method of claim 9, wherein a partial hypothesis is formulated by applying a linear regression algorithm to a different collection of points. ### Claim 11 (depends on 10) The method of claim 10, further comprising assigning weights to the points for the purpose of partial and full hypotheses likelihood, the weights differing for at least two points. ### Claim 12 (depends on 11) The method of claim 11, wherein the weights depend on a distance of the points from the at least one mobile robot. ### Claim 13 (depends on 1) The method of claim 1, wherein the hypotheses are assigned likelihood values based on a discrete and finite set of parameters. ### Claim 14 (depends on 13) The method of claim 13, wherein said parameters comprise one or more of: (i) a number of points on which a particular hypothesis is based, (ii) a distance of points on which a hypothesis is based from the at least one mobile robot, (iii) a distance between a left partial hypothesis and a right partial hypothesis, (iv) an angle between a left partial hypothesis and a right partial hypothesis, and/or, when the at least one mobile robot comprises at least two cameras, (v) a number of cameras agreeing on the same lines on which a hypothesis is based. ### Claim 15 (depends on 1) The method of claim 1, wherein the at least one camera comprises at least two cameras, and wherein a first at least one camera is forward-facing with respect to a direction of motion of the at least one mobile robot, and wherein a second at least one camera is backward-facing with respect to said direction of motion of the at least one mobile robot. ### Claim 16 (depends on 1) The method of claim 1, wherein the at least one camera comprises a stereo camera system comprising at least two lenses, and wherein the stereo camera system is forward-facing with respect to a direction of motion of the at least one mobile robot. ### Claim 17 (depends on 1) The method of claim 1, wherein the at least one camera comprises at least two cameras, and wherein at least two of said cameras are sideways-facing with respect to a direction of motion of the at least one mobile robot. ### Claim 18 (depends on 1) The method of claim 1, further comprising the at least one mobile robot navigating on a walkway at least partially using the at least one hypothesis with the highest likelihood of representing the walkway boundary. ### Claim 19 (depends on 1) The method of claim 1, wherein the at least one hypothesis with the highest likelihood of representing the walkway boundary comprises an input of a navigation procedure of the at least one mobile robot. ### Claim 20 (depends on 1) The method of claim 1, wherein the at least one camera is adapted to acquire 1 to 10 images per second, more preferably 2 to 8 images per second, even more preferably 3 to 6 images per second. ### Claim 21 (depends on 1) The method of claim 1, wherein the extraction of the lines from at least one image is performed by applying an edge detector algorithm to the at least one image followed by a line extractor algorithm ### Claim 22 (depends on 1) The method of claim 1, wherein the at least one mobile robot further comprises a processing component adapted to implement at least part of at least one of (b), (c) and (d), preferably all of (b), (c) and (d). ### Claim at position 23 23-24. (canceled) ### Claim 25 (depends on 4) The method of claim 4, wherein the at least one mobile robot comprises at least one light-based range sensor and wherein the method further comprises the at least one light-based range sensor obtaining data and the ground plane being obtained using the data obtained by the at least one light-based range sensor. ### Claim at position 25 26-27. (canceled) ### Claim 28 (depends on 1) The method of claim 1, wherein (i) the at least one mobile robot has a width in the range of 20 to 100 cm, preferably 40 to 70 cm, such as about 55 cm.; and/or (ii) the at least one mobile robot has a height in the range of 20 to 100 cm, preferably 40 to 70 cm, such as about 60 cm.; and/or (iii) the at least one mobile robot has a length in the range of 30 to 120 cm, preferably 50 to 80 cm, such as about 65 cm.; and/or (iv) the at least one mobile robot has a weight in the range of 2 to 50 kg, preferably 5 to 40 kg, more preferably 7 to 25 kg, such as 10 to 20 kg.; and/or (v) the at least one mobile robot is adapted not to travel with a speed of more than 10 km/h, or more than 8 km/h, or more than 6 km/h. ### Claim 29 A system comprising: (a) at least one camera associated with at least one mobile robot, the at least one camera adapted to acquire at least one image; and (b) at least one processing component constructed and adapted to at least extract a plurality of lines from the at least one image, create at least one dataset comprising data related to the plurality of lines extracted from the at least one image, and form a plurality of hypotheses for a walkway boundary based on the at least one dataset and determine at least one hypothesis with the highest likelihood of representing a walkway boundary. ### Claim 30 (depends on 29) The system of claim 29 wherein the mobile robot is constructed and adapted to use the most likely hypothesis as part of autonomous navigation on walkways. ### Claim 31 (depends on 29) The system of claim 29, wherein said at least one camera attached to the at least one mobile robot, and wherein at least one of said cameras is forward-facing with respect to a direction of motion of the at least one mobile robot and at least one of said cameras is backward-facing with respect to the direction of motion of the at least one mobile robot. ### Claim 32 (depends on 29) The system of claim 29, wherein said at least one camera attached to the at least one mobile robot, and wherein said at least one camera comprises a stereo camera system comprising at least two lenses, and wherein the stereo camera system is forward-facing with respect to a direction of motion of the at least one mobile robot. ### Claim at position 31 33-34. (canceled) ### Claim 35 (depends on 29) The system of claim 29, wherein the at least one mobile robot comprises at least one light-based range sensor. ### Claim at position 33 36-37. (canceled) ### Claim 38 (depends on 29) The system of claim 29, wherein (i) the at least one mobile robot has a width in the range of 20 to 100 cm, preferably 40 to 70 cm, such as about 55 cm.; and/or (ii) the at least one mobile robot has a height in the range of 20 to 100 cm, preferably 40 to 70 cm, such as about 60 cm.; and/or (iii) the at least one mobile robot has a length in the range of 30 to 120 cm, preferably 50 to 80 cm, such as about 65 cm.; and/or (iv) the at least one mobile robot has a weight in the range of 2 to 50 kg, preferably 5 to 40 kg, more preferably 7 to 25 kg, such as 10 to 20 kg.; and/or (v) the at least one mobile robot is adapted not to travel with a speed of more than 10 km/h, or more than 8 km/h, or more than 6 km/h. ### Claim 39 A non-transitory computer-readable medium with one or more computer programs stored therein that, when executed by one or more processors, cause the one or more processors to perform at least the operations of: (a) acquiring at least one image with at least one camera associated with at least one mobile robot; and (b) extracting a plurality of lines from the at least one image acquired in (a); (c) creating at least one dataset comprising data related to the plurality of lines extracted from the at least one image in (b); (d) forming a plurality of hypotheses for a walkway boundary based on the at least one dataset and determining at least one hypothesis with the highest likelihood of representing a walkway boundary; and (e) using the at least one hypothesis determined in (d) to navigate the at least one mobile robot. ### Claim 40 (depends on 39) The computer-readable medium of claim 39, wherein the one or more processors comprises at least one CPU and at least one GPU.