# US 10,769,453 B2 · Electronic device and method of controlling operation of vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,453 B2 - Document ID: US10769453B2 - Status: Granted - Kind: B2 - Application number: 15/964,779 - Filed: 2018-04-27 - Published: 2018-11-22 - Granted: 2020-09-08 - Earliest claimed: 2017-05-16 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Dae-hyun Ban; Seo-woo Jang; Un-kyu Park - Web page: https://patentgrep.com/patent/US10769453B2 ## CPC - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01) - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60W 40/04: Traffic conditions - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60K 2360/175: Autonomous driving - B60K 2360/178: Warnings - B60K 2360/179: Distances to obstacles or vehicles - B60R 2300/205: using a head-up display - B60R 2300/301: combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision - B60R 2300/302: combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data - B60R 2300/305: merging camera image with lines or icons - B60R 2300/308: by overlaying the real scene, e.g. through a head-up display on the windscreen - B60W 2050/0088: {Adaptive recalibration} - B60W 2050/143: {Alarm means} - B60W 2050/146: {Display means} - B60W 2420/403: Image sensing, e.g. optical camera - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract An electronic device and method of controlling an operation of a vehicle are provided. The method includes obtaining a video sequence including a plurality of frames from a camera installed on the vehicle; detecting, from the plurality of frames, an object included in the plurality of frames; obtaining position information regarding the object with respect to each of the plurality of frames; determining whether an event related to driving of the vehicle has occurred by analyzing time-series changes in positions of the object in the plurality of frames; generating a notification message about the event based on a result of the determining; and outputting the generated notification message, wherein the detecting of the object, the obtaining of the position information, the determining of whether an event has occurred, and the generating of the notification message are performed using a plurality of learning models. ## Claims ### Claim 1 An electronic device configured to control an operation of a vehicle, the electronic device comprising: a memory configured to store at least one program; and at least one processor configured to provide a notification message by executing the at least one program, wherein the at least one program includes instructions, which when executed by the processor, cause the electronic device to perform at least one operation comprising: obtaining a video sequence comprising a plurality of frames from a camera installed on the vehicle; detecting, from the plurality of frames, an object included in the plurality of frames via a first learning model; obtaining position information regarding the detected object with respect to each of the plurality of frames via the first learning model; determining whether an event related to driving of the vehicle has occurred by analyzing time-series changes in positions of the detected object in the plurality of frames via a second learning model; generating a notification message about the event based on a result of the determining via the second learning model; and outputting the generated notification message, wherein an input to the second learning model includes an output of the first learning model, the output of the first learning model including the position information regarding the detected object with respect to each of the plurality of frames. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the at least one program includes instructions, which when executed by the processor, cause the electronic device to perform at least one operation comprising the plurality of frames being input into the first learning model, and the position information output by the first learning model includes a group of pixels in each of the plurality of frames indicating the position of the detected object. ### Claim 3 (depends on 1) The electronic device of claim 1, wherein the determining of whether an event has occurred comprises: determining a type of the event; and determining a risk level of driving, and the generating of the notification message comprises generating the notification message based on the type of the event and the risk level of driving. ### Claim 4 (depends on 3) The electronic device of claim 3, wherein a command for controlling an operation of a module installed on the vehicle is transmitted to the module installed on the vehicle based on the type of the event and the risk level of driving. ### Claim 5 (depends on 1) The electronic device of claim 1, wherein the outputting of the notification message comprises displaying the notification message on a head-up display (HUD) of the vehicle. ### Claim 6 (depends on 1) The electronic device of claim 1, wherein the obtaining of the positional information regarding the object comprises determining positions of the object in the plurality of frames on a pixel-by-pixel basis. ### Claim 7 (depends on 1) The electronic device of claim 1, wherein the first learning model is generated using a fully convolutional network (FCN), by learning criteria for determining a type of the object from a plurality of different types of objects and learning criteria for determining positions of the object in the plurality of frames, the type of the object is determined using the first learning model in the detecting of the object, and positions of the object in the plurality of frames are determined with respect to the plurality of frames using the first learning model in the obtaining of the positional information regarding the object. ### Claim 8 (depends on 1) The electronic device of claim 1, wherein the input to the second learning model is obtained by reducing dimensions of the output of the first learning model. ### Claim 9 (depends on 1) The electronic device of claim 1, wherein the second learning model is generated using a recurrent neural network (RNN), by learning criteria for determining whether an event related to driving of a vehicle has occurred and learning criteria for determining the content of a notification message, and the generating of the notification message comprises, determining the content of the notification message using the second learning model. ### Claim 10 (depends on 1) The electronic device of claim 1, wherein the at least one program further comprises instructions for applying, to the plurality of frames, a filter for flattening lightening degrees of the plurality of frames to input the plurality of frames to the first learning model. ### Claim 11 A method of controlling an operation of a vehicle, the method comprising: obtaining a video sequence comprising a plurality of frames from a camera installed on the vehicle; detecting, from the plurality of frames, an object included in the plurality of frames via a first learning model; obtaining position information regarding the detected object with respect to each of the plurality of frames via the first learning model; determining whether an event related to driving of the vehicle has occurred by analyzing time-series changes in positions of the detected object in the plurality of frames via a second learning model; generating a notification message about the event based on a result of the determining via the second learning model; and outputting the generated notification message, wherein an input to the second learning model includes an output of the first learning model, the output of the first learning model including the position information regarding the detected object with respect to each of the plurality of frames. ### Claim 12 (depends on 11) The method of claim 11, wherein the plurality of frames are input into the first learning model, and the position information output by the first learning model includes a group of pixels in each of the plurality of frames indicating the position of the detected object. ### Claim 13 (depends on 11) The method of claim 11, wherein the determining of whether an event has occurred comprises: determining a type of the event; and determining a risk level of driving, and the generating of the notification message comprises generating the notification message based on the type of the event and the risk level of driving. ### Claim 14 (depends on 13) The method of claim 13, wherein a command for controlling an operation of a module installed on the vehicle is transmitted to the module installed on the vehicle based on the type of the event and the risk level of driving. ### Claim 15 (depends on 11) The method of claim 11, wherein the obtaining of the positional information regarding the object comprises determining positions of the object in the plurality of frames on a pixel-by-pixel basis. ### Claim 16 (depends on 11) The method of claim 11, wherein the first learning model is generated using a fully convolutional network (FCN), by learning criteria for determining a type of the object from a plurality of different types of objects and learning criteria for determining positions of the object in the plurality of frames, the type of the object is determined using the first learning model in the detecting of the object, and positions of the object in the plurality of frames are determined with respect to the plurality of frames using the first learning model in the obtaining of the positional information regarding the object. ### Claim 17 (depends on 11) The method of claim 11, wherein the input to the second learning model is obtained by reducing dimensions of the output of the first learning model. ### Claim 18 (depends on 11) The method of claim 11, wherein the second learning model is generated using a recurrent neural network (RNN), by learning criteria for determining whether an event related to driving of a vehicle has occurred and learning criteria for determining the content of a notification message, and the generating of the notification message comprises, determining the content of the notification message using the second learning model. ### Claim 19 (depends on 11) The method of claim 11, further comprising: applying, to the plurality of frames, a filter for flattening lightening degrees of the plurality of frames to input the plurality of frames to the first learning model. ### Claim 20 A non-transitory computer readable recording medium having recorded thereon a computer program which, when executed by a processor, causes the processor to: obtain a video sequence comprising a plurality of frames from a camera installed on a vehicle; detect, from the plurality of frames, an object included in the plurality of frames using a first learning model; obtain position information regarding the detected object with respect to each of the plurality of frames using the first learning model; determine whether an event related to driving of the vehicle has occurred by analyzing time-series changes in positions of the detected object in the plurality of frames using a second learning model; generate a notification message about the event based on a result of the determining using the second learning model; and output the generated notification message, wherein an input to the second learning model includes an output of the first learning model, the output of the first learning model including the position information regarding the detected object with respect to each of the plurality of frames.