# US 2019/0318491 A1 · System and method for gathering data related to quality service in a customer service environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318491 A1 - Document ID: US20190318491A1 - Status: Published application - Kind: A1 - Application number: 15/780,349 - Filed: 2017-07-13 - Published: 2019-10-17 - Earliest claimed: 2017-07-13 - Applicants: Tempo Analytics Inc. - Inventors: Robert LAGANIERE; Luis E. GURRIERI - Web page: https://patentgrep.com/patent/US20190318491A1 ## CPC - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06Q 30/01: Customer relationship services - G06T 7/593: from stereo images - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole} - G06V 40/176: {Dynamic expression} - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G06V 40/174: {Facial expression recognition} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods relating to gathering data which may be used to produce metrics to analyze customer quality of service. Color-plus-depth sensors are placed facing a service area where customers are located. Images and depth maps of the service area are gathered by the sensors. These data sets are then processed by an on-site processor to isolate each customer and/or each item of interest in each data set. Each detected customer is identified using customer descriptors and the location of each item of interest and of each detected customer is determined relative to a reference point. The extracted data from each data set is uploaded to an off-site server for further processing. The off-site server analyzes the uploaded data to determine customer behavior, customer demographics, customer biometrics, and emotional state. The server can also analyze the uploaded data to determine quality of service (including service time). ## Claims ### Claim 1 A system for gathering data relating to a service area, the system comprising: at least one color-plus-depth sensor for placement behind said service area and facing customers; a processor for: receiving time indexed images from said at least one color-plus-depth sensor processing said images; extracting sets of customer descriptors from each of said time indexed images, each set of customer descriptors being for uniquely identifying one of said customers; extracting sets of item descriptors from each of said time indexed images, each set of item descriptors being for identifying items in said service area; and uploading said sets of customer descriptors and said sets of item descriptors to a server for further processing; wherein said sets of customer descriptors preserve customer anonymity. ### Claim 2 (depends on 1) The system according to claim 1, wherein said color-plus-depth sensor is of a type that is at least one of: RGB+D sensors; stereoscopic cameras; structured light sensors; and time of flight sensors. ### Claim 3 (depends on 1) The system according to claim 1, wherein said set of customer descriptors is based on at least one of: a 3D structure and color of a customer's clothes; a 3D structure and color of a customer's body parts; local image-based color histograms; local image-based histograms of oriented gradients; local image-based binary patterns; local image-based oriented filter responses; at least one local object-based facial feature; at least one object-based arm at least one object-based hand; at least one interest point on a customer's body; a customer's face; a customer's head shape; a customer's head size; a customer's torso shape; a customer's torso size; a customer's skin tonality; a customer's facial expression; a customer's location relative to a predetermined reference point; color descriptors of different parts of a customer in said images; a customer's height; and a customer's location in said service area. ### Claim 4 (depends on 1) The system according to claim 1, wherein said at least one color-plus-depth sensors has a field of view which overlaps a field of view of at least one other color-plus-depth sensor. ### Claim 5 (depends on 1) The system according to claim 1, wherein said at least one color-plus-depth sensors has a field of view which fully overlaps a field of view of at least one other color-plus-depth sensor. ### Claim 6 (depends on 3) The system according to claim 3, wherein said predetermined reference point is a service counter in said service area. ### Claim 7 (depends on 1) The system according to claim 1, wherein said service area is divided into two sub-areas, one of said two sub-areas being closer to said at least one sensor. ### Claim 8 (depends on 1) The system according to claim 1, wherein said server processes data received from said processor to determine at least one of: customer demographics; customer transaction state; customer satisfaction; average service time to serve a customer; customer biometrics; customer actions; and customer emotional state. ### Claim 9 (depends on 1) The system according to claim 1, wherein said color-plus-depth sensors are behind and above a service counter in said service area. ### Claim 10 (depends on 1) The system according to claim 1, wherein said color-plus-depth sensors produce at least one depth map. ### Claim 11 A method for gathering data relating to a service area, the method comprising: a) placing at least one color-plus-depth sensor at a location behind said service area and facing customers in said service area; b) producing at least one color image and at least one depth map of said service area using said at least one color-plus-depth sensor; c) using a processor to process said at least one color image to extract a first set of customer descriptors for at least one customer in said service area, said first set of customer descriptors being for describing said at least one customer based on at least one color descriptor for said at least one customer; d) using a processor to process said at least one depth map to extract a second set of customer descriptors for said at least one customer in said service area, said second set of customer descriptor being for describing said at least one customer based on at least one 3D descriptor for said at least one customer; e) uploading said first and second sets of customer descriptors to a server for further processing. ### Claim 12 (depends on 11) The method according to claim 11, wherein said sets of customer descriptors is based on: a 3D structure and color of a customer's clothes; a 3D structure and color of a customer's body parts; local image-based color histograms; local image-based histograms of oriented gradients; local image-based binary patterns; local image-based oriented filter responses; at least one local object-based facial feature; at least one object-based arm at least one object-based hand; at least one interest point on a customer's body; a customer's face; a customer's head shape; a customer's head size; a customer's torso shape; a customer's torso size; a customer's skin tonality; a customer's facial expression; a customer's location relative to a predetermined reference point; color descriptors of different parts of a customer in said images; a customer's height; and a customer's location in said service area. ### Claim 13 (depends on 11) The method according to claim 11, wherein said at least one color image is segmented to isolate said at least one customer in said color image. ### Claim 14 (depends on 11) The method according to claim 11, wherein said server is remote from said at least one color-plus-depth sensor. ### Claim 15 (depends on 11) The method according to claim 11, further including a step of using said processor to process said at least one color image to extract item descriptors for at least one item of interest in said service area, said item descriptor being for describing at least one item for use by said at least one customer. ### Claim 16 (depends on 11) The method according to claim 11 wherein said server tracks said at least one customer across multiple color images using said customer descriptor. ### Claim 17 Non-transitory computer-readable media having encoded thereon computer-readable computer instructions which, when executed, implement a method for gathering data relating to a service area, the method comprising: a) placing at least one color-plus-depth sensor at a location behind said service area and facing customers in said service area; b) producing at least one color image and at least one depth map of said service area using said at least one color-plus-depth sensor; c) using a processor to process said at least one color image to extract a first set of customer descriptors for at least one customer in said service area, said first set of customer descriptors being for describing said at least one customer based on at least one color descriptor for said at least one customer; d) using a processor to process said at least one depth map to extract a second set of customer descriptors for said at least one customer in said service area, said second set of customer descriptor being for describing said at least one customer based on at least one 3D descriptor for said at least one customer; e) uploading said first and second sets of customer descriptors to a server for further processing.