Computer-readable recording medium, fraud detection method, and fraud detection apparatus

An information processing method comprising:

• detecting a person and commodity products sold in a store by analyzing consecutive image frames captured by a first camera installed inside of the store;
• generating by inputting each of the consecutive image frames including the person into a machine learning model, skeleton information on the person on each of the consecutive image frames, the skeleton information on the person including a position of fingers of the person;
• determining whether or not the person holds a commodity product associated with a motion of extending a hand of the person by using partial images including the position of fingers within the consecutive image frames where a motion of the extending a hand detected from the skeleton information is detected and an object detection algorithm that identifies an object presence;
• determining whether or not the person holds a commodity product associated with a motion of returning the hand of the person by using partial images including the position of fingers within the consecutive image frames where a motion of the extending a hand detected from the skeleton information is detected and the object detection algorithm;
• determining an occurrence of a first motion of the person acquiring the commodity product, when determining that the person does not hold the commodity product at a time of the motion of extending the hand and determining that the person holds the commodity product at a time of the motion of returning the hand on each of the consecutive image frames;
• generating, at a time of determining the occurrence of the first motion, first information where an identification of the person and the first motion are associated, and storing the first information in a motion history, wherein the motion history stores feature information on the person extracted from the consecutive image frames captured by the first camera;
• determining an occurrence of a second motion of the person returning the commodity product, when determining that the person holds the commodity product at a time of the motion of extending the hand and determining that the person does not hold the commodity product at a time of the motion of returning the hand on each of the consecutive image frames;
• generating at a time of determining the occurrence of the second motion, second information where the identification of the person and the second motion are associated, and storing the second information in the motion history by appending the second information to the first information;
• counting an acquisition count by referring to the motion history and incrementing the acquisition count for the first motion of the person or decrementing the acquisition count for the second motion of the person;
• identifying the person who is captured in consecutive image frames captured by a second camera capturing therein an area that includes a self-service checkout terminal and is a same person whose identification is stored in the motion history, by performing person re-identification that compares the feature information stored in the motion history with feature information on the person extracted from the captured consecutive image frames from the second camera;
• receiving commodity product information via a network from the self-service checkout terminal that the identified person uses, wherein the commodity product information is information transmitted from a first terminal that is used by the person and stores therein a registered commodity product by scanning a code of a commodity product, and detecting a number of registrations indicating a number of scanned commodity products based on the received commodity product information;
• generating, based on a difference between the detected number of registrations and the counted acquisition count, an alert that indicates a commodity product put into a shopping basket by the person is not registered in the first terminal;
• and
• transmitting the generated alert to a second terminal via the network, wherein the first terminal is a terminal that stores therein information on the commodity product targeted for the purchase by scanning a bar code or a QR code attached to the commodity product.

An information processing apparatus, comprising:

• a memory;
• and
• a processor coupled to the memory and the processor configured to:
• detect a person and commodity products sold in a store by analyzing consecutive image frames captured by a first camera installed inside of the store;
• generate by inputting each of the consecutive image frames including the person into a machine learning model, skeleton information on the person on each of the consecutive image frames, the skeleton information on the person including a position of fingers of the person;
• determine whether or not the person holds a commodity product associated with a motion of extending a hand of the person by using partial images including the position of fingers within the consecutive image frames where a motion of the extending a hand detected from the skeleton information is detected and an object detection algorithm that identifies an object presence;
• determine whether or not the person holds a commodity product associated with a motion of returning the hand of the person by using partial images including the position of fingers within the consecutive image frames where a motion of the extending a hand detected from the skeleton information is detected and the object detection algorithm;
• determining an occurrence of a first motion of the person acquiring the commodity product, when determining that the person does not hold the commodity product at a time of the motion of extending the hand and determining that the person holds the commodity product at a time of the motion of returning the hand on each of the consecutive image frames;
• generating, at a time of determining the occurrence of the first motion, first information where an identification of the person and the first motion are associated, and storing the first information in a motion history, wherein the motion history stores feature information on the person extracted from the consecutive image frames captured by the first camera;
• determining an occurrence of a second motion of the person returning the commodity product, when determining that the person holds the commodity product at a time of the motion of extending the hand and determining that the person does not hold the commodity product at a time of the motion of returning the hand on each of the consecutive image frames;
• generating at a time of determining the occurrence of the second motion, second information where the identification of the person and the second motion are associated, and storing the second information in the motion history by appending the second information to the first information;
• counting an acquisition count by referring to the motion history and incrementing the acquisition count for the first motion of the person or decrementing the acquisition count for the second motion of the person;
• identifying the person who is captured in consecutive image frames captured by a second camera capturing therein an area that includes a self-service checkout terminal and is a same person whose identification is stored in the motion history, by performing person re-identification that compares the feature information stored in the motion history with feature information on the person extracted from the captured consecutive image frames from the second camera;
• receiving commodity product information via a network from the self-service checkout terminal that the identified person uses, wherein the commodity product information is information transmitted from a first terminal that is used by the person and stores therein a registered commodity product by scanning a code of a commodity product, and detecting a number of registrations indicating a number of scanned commodity products based on the received commodity product information;
• generating, based on a difference between the detected number of registrations and the counted acquisition count, an alert that indicates a commodity product put into a shopping basket by the person is not registered in the first terminal;
• and
• transmitting the generated alert to a second terminal via the network, wherein the first terminal is a terminal that stores therein information on the commodity product targeted for the purchase by scanning a bar code or a QR code attached to the commodity product.