# US 12,001,519 B2 · Object classification and related applications based on frame and event camera processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,519 B2 - Document ID: US12001519B2 - Status: Granted - Kind: B2 - Application number: 17/177,933 - Filed: 2021-02-17 - Published: 2022-04-14 - Granted: 2024-06-04 - Earliest claimed: 2020-10-08 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Ted Yamazaki; Tonni Larsen; Leigh Anderson; Tatsushi Nashida; Naomasa Takahashi - Web page: https://patentgrep.com/patent/US12001519B2 ## CPC - G06F 18/2413: based on distances to training or reference patterns - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/22: Display screens - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions - B60K 35/85: Arrangements for transferring vehicle- or driver-related data - B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering - B60W 40/09: Driving style or behaviour - G06F 18/256: {of results relating to different input data, e.g. multimodal recognition} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) - G06V 10/443: {by matching or filtering} - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60K 2360/146: Instrument input by gesture - B60K 2360/176: Camera images - B60K 2360/191: Highlight information - B60K 2360/334: Projection means - B60W 2050/0075: {Automatic parameter input, automatic initialising or calibrating means} - B60W 2400/00: Indexing codes relating to detected, measured or calculated conditions or factors - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2540/30: Driving style - B60W 2554/402: Type - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system and method for object classification and related applications based on frame and event camera processing is provided. The system acquires an output of image sensor circuitry and determines a first object classification result based on feature points associated with first objects in a first frame of the acquired output. The system executes an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame. The generated event frame includes pixel-level movement information associated with second objects. The system determines a second object classification result based on the pixel-level movement information and determines one or more object classification tags based on the determined first and the determined second object classification result. The determined one or more object classification tags correspond to one or more objects, included in at least the first objects and the second objects. ## Claims ### Claim 1 A system, comprising: circuitry configured to: acquire an output of image sensor circuitry; determine a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output; execute an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein the generated event frame comprises pixel-level movement information associated with one or more second objects; determine a second object classification result based on the pixel-level movement information; determine whether there is a difference between the determined first object classification result and the determined second object classification result; determine one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects; determine, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified; execute a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and classify the determined errored object into a first object class based on the execution of the pattern matching operation. ### Claim 2 (depends on 1) The system according to claim 1, wherein the one or more first objects are in one of a stationary state or are about to reach the stationary state, and the one or more second objects are in motion. ### Claim 3 (depends on 1) The system according to claim 1, wherein the circuitry is further configured to: detect the feature points in the first frame of the acquired output; and determine a first pattern match between the detected feature points with first reference features associated with known object classes, wherein the first object classification result is determined based on the determined first pattern match. ### Claim 4 (depends on 1) The system according to claim 1, wherein the circuitry is further configured to determine the second object classification result based on pattern matching the pixel-level movement information with second reference features associated with known object classes. ### Claim 5 (depends on 1) The system according to claim 1, wherein the first object classification result comprises the first frame overlaid with first bounding box information associated with the one or more first objects, and the second object classification result comprises the event frame overlaid with second bounding box information associated with the one or more second objects. ### Claim 6 (depends on 1) The system according to claim 1, wherein the circuitry is further configured to determine, for the errored object, a first object classification tag of the one or more object classification tags based on a determination that the first object class is correct. ### Claim 7 (depends on 1) The system according to claim 1, wherein the pattern matching operation is one of a machine learning-based pattern matching operation or a deep learning-based pattern matching operation. ### Claim 8 (depends on 1) The system according to claim 1, wherein the circuitry is further configured to: transmit one of the event frame or the first frame to a server, based on a determination that the first object class is incorrect; receive a second object class for the determined errored object from the server based on the transmission; and determine, for the errored object, a first object classification tag of the one or more object classification tags based on a determination that the second object class is correct. ### Claim 9 (depends on 1) The system according to claim 1, wherein each of the first frame and the event frame capture scene information of an environment outside a vehicle. ### Claim 10 (depends on 9) The system according to claim 9, wherein the circuitry is further configured to: control an electronic control system of the vehicle to detect an event comprising one of an accident or a near miss in the environment; and determine a behavior pattern of a driver of the vehicle based on the determined one or more object classification tags and the detected event. ### Claim 11 (depends on 10) The system according to claim 10, wherein the electronic control system is one of Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD) system. ### Claim 12 (depends on 10) The system according to claim 10, wherein the circuitry is further configured to control one or more operations of the vehicle based on the determined behavior pattern. ### Claim 13 (depends on 10) The system according to claim 10, wherein the circuitry is further configured to: generate notification information based on the determined behavior pattern; and transmit the generated notification information to an electronic device which is associated with one of the vehicle or the driver of the vehicle. ### Claim 14 (depends on 1) The system according to claim 1, wherein the circuitry is further configured to: control, inside a vehicle, an image projecting device to project an image of a user interface (UI) onto a projection surface inside the vehicle, wherein each of the first frame and the event frame capture scene information of an in-vehicle environment inside the vehicle; detect a finger movement over a virtual UI element of the UI included in the projected image, based on a second object classification tag of the determined one or more object classification tags; select the virtual UI element based on the detection; and control one or more vehicle functions based on the selection. ### Claim 15 A method, comprising: acquiring an output of image sensor circuitry; determining a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output; executing an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein the generated event frame comprises pixel-level movement information associated with one or more second objects; determining a second object classification result based on the pixel-level movement information; determining whether there is a difference between the determined first object classification result and the determined second object classification result; determining one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects; determining, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified; executing a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and classifying the determined errored object into a first object class based on the execution of the pattern matching operation. ### Claim 16 (depends on 15) The method according to claim 15, further comprising: detecting the feature points in the first frame of the acquired output; and determining a first pattern match between the detected feature points with first reference features associated with known object classes, wherein the first object classification result is determined based on the determined first pattern match. ### Claim 17 (depends on 15) The method according to claim 15, further comprising determining the second object classification result based on pattern matching the pixel-level movement information with second reference features associated with known object classes. ### Claim 18 A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a system, causes the system to execute operations, the operations comprising: acquiring an output of image sensor circuitry; determining a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output; executing an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein the generated event frame comprises pixel-level movement information associated with one or more second objects; determining a second object classification result based on the pixel-level movement information; determining whether there is a difference between the determined first object classification result and the determined second object classification result; determining one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects; determining, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified; executing a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and classifying the determined errored object into a first object class based on the execution of the pattern matching operation.