# US 2013/0254772 A1 · Verification of complex workflows through internal assessment or community based assessment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0254772 A1
- Document ID: US20130254772A1
- Status: Published application
- Kind: A1
- Application number: 13/426,104
- Filed: 2012-03-21
- Published: 2013-09-26
- Earliest claimed: 2012-03-21
- Assignees: Phillip Morris International; International Business Machines Corporation
- Inventors: Stephanie M. Corthesy; Pablo Meyer Rojas; Raquel Norel; John J. Rice; Ajay K. Royyuru; Joerg J. Sprengel; Gustavo A. Stolovitzky; Thomas Bonk; Julia Hoeng; Manuel Peitsch; Katrin Stolle
- Web page: https://patentgrep.com/patent/US20130254772A1

## CPC

- G06Q 10/0633: Workflow analysis
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A method of implementing verification of a complex workflow includes partitioning the workflow into modules, wherein the modules have inputs, processing steps and outputs; selecting, from the workflow, one of the partitioned modules for independent verification by challenge thereof; running, with a computing device, a challenge of the selected module, the challenge comprising comparing reference outputs to outputs of the selected module, wherein reference inputs are received by the selected module and the reference outputs are generated using the reference inputs and one of an ideal performing module or a well-established module; determining whether outputs of the selected module meet verification criteria with respect to the reference outputs, and based on the determining, implementing one of: declaring the selected module verified; subdividing the selected module into smaller modules and repeating the challenge on the smaller modules; or declaring the selected module not verified.

## Claims

### Claim 1

A method of implementing verification of a complex workflow, the method comprising:
partitioning the workflow into modules, wherein the modules have inputs, processing steps and outputs;
selecting, from the workflow, one or more of the partitioned modules for independent verification by challenge thereof;
running, with a computing device, a challenge of the selected module, the challenge comprising comparing reference outputs to outputs of the selected module, wherein reference inputs are received by the selected module and the reference outputs are generated using the reference inputs and one of an ideal performing module or a well-established module;
determining whether outputs of the selected module meet verification criteria with respect to the reference outputs, and based on the determining, implementing one of:
declaring the selected module verified;
subdividing the selected module into smaller modules and repeating the challenge on the smaller modules; or
declaring the selected module not verified.

### Claim 2 (depends on 1)

The method of claim 1, wherein the challenge comprises an internal challenge with respect to an organization such that the outputs of the selected module are compared only with the reference outputs of the ideal performing module or well-established module.

### Claim 3 (depends on 1)

The method of claim 1, wherein the challenge comprises a community-based challenge in which outputs from modules of external participants are compared with the outputs of the selected module and the reference outputs of the ideal performing module or well-established module.

### Claim 4 (depends on 3)

The method of claim 3, wherein the community-based challenge comprises at least two external participants.

### Claim 5 (depends on 3)

The method of claim 3, further comprising determining whether a best performer of the outputs of the external participant modules and the outputs of the selected module meet the verification criteria with respect to the reference outputs.

### Claim 6 (depends on 4)

The method of claim 4, wherein, in the event a best performer is determined to result from an external participant, the best performer module is declared verified and replaces the selected module in the workflow.

### Claim 7 (depends on 4)

The method of claim 4, wherein information regarding the best performer is received by a third party entity with respect to the computing device and any of the external participants.

### Claim 8 (depends on 7)

The method of claim 7, wherein the identities of the external parties are unknown to the third party entity, the third party entity performing a ranking of the outputs of the external participant modules, the outputs of the selected module and the reference outputs.

### Claim 9 (depends on 1)

The method of claim 1, wherein the verification criteria comprise a quantitative metric of proximity to a predetermined threshold.

### Claim 10 (depends on 1)

The method of claim 1, wherein the ideal performing module is determined by one or more of theoretical modeling, statistical averaging over large datasets, and the use of a priori knowledge.

### Claim 11 (depends on 1)

The method of claim 1, wherein the complex workflow comprises one of the following domains: industrial manufacturing, industrial research and development, academic scientific research, and information technology.

### Claim 12

A computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method of verification of a complex workflow, the method comprising:
partitioning the workflow into modules, wherein the modules have inputs, processing steps and outputs;
selecting, from the workflow, one of the partitioned modules for independent verification by challenge thereof;
running, with a computing device, a challenge of the selected module, the challenge comprising comparing reference outputs to outputs of the selected module, wherein reference inputs are received by the selected module and the reference outputs are generated using the reference inputs and one of an ideal performing module or a well-established module;
determining whether outputs of the selected module meet verification criteria with respect to the reference outputs, and based on the determining, implementing one of:
declaring the selected module verified;
subdividing the selected module into smaller modules and repeating the challenge on the smaller modules; or
declaring the selected module not verified.

### Claim 13 (depends on 12)

The computer readable storage medium of claim 12, wherein the challenge comprises an internal challenge with respect to an organization such that the outputs of the selected module are compared only with the reference outputs of the ideal performing module or well-established module.

### Claim 14 (depends on 12)

The computer readable storage medium of claim 12, wherein the challenge comprises a community-based challenge in which outputs from modules of external participants are compared with the outputs of the selected module and the reference outputs of the ideal performing module or well-established module.

### Claim 15 (depends on 14)

The computer readable storage medium of claim 14, wherein the community-based challenge comprises at least two external participants.

### Claim 16 (depends on 14)

The computer readable storage medium of claim 14, wherein the method further comprises determining whether a best performer of the outputs of the external participant modules and the outputs of the selected module meet the verification criteria with respect to the reference outputs.

### Claim 17 (depends on 15)

The computer readable storage medium of claim 15, wherein, in the event a best performer is determined to result from an external participant, the best performer module is declared verified and replaces the selected module in the workflow.

### Claim 18 (depends on 15)

The computer readable storage medium of claim 15, wherein information regarding the best performer is received by a third party entity with respect to the computing device and any of the external participants.

### Claim 19 (depends on 18)

The computer readable storage medium of claim 18, wherein the identities of the external parties are unknown to the third party entity, the third party entity performing a ranking of the outputs of the external participant modules, the outputs of the selected module and the reference outputs.

### Claim 20 (depends on 12)

The computer readable storage medium of claim 12, wherein the verification criteria comprise a quantitative metric of proximity to a predetermined threshold, the metric of proximity computed using computer readable instructions stored in the readable storage medium.

### Claim 21 (depends on 12)

The computer readable storage medium of claim 12, wherein the ideal performing module is determined by one or more of theoretical modeling, statistical averaging over large datasets, and the use of a priori knowledge.

### Claim 22 (depends on 12)

The computer readable storage medium of claim 12, wherein the complex workflow comprises one of the following domains: industrial manufacturing, industrial research and development, academic scientific research, and information technology.
