# US 11,308,699 B2 · Method and system for data generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,699 B2 - Document ID: US11308699B2 - Status: Granted - Kind: B2 - Application number: 16/823,003 - Filed: 2020-03-18 - Published: 2021-09-09 - Granted: 2022-04-19 - Earliest claimed: 2020-03-03 - Assignees: Arm Limited - Applicants: Arm Limited - Inventors: Vasileios Laganakos; Irenéus Johannes De Jong; Gary Dale Carpenter - Web page: https://patentgrep.com/patent/US11308699B2 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06F 18/2193: {based on specific statistical tests} - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/0499: Feedforward networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/094: Adversarial learning - G06N 20/00: Machine learning - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06T 2210/61: Scene description - G06T 2219/004: Annotating, labelling - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computer-implemented method includes generating, using a scene generator, first candidate scene data; obtaining reference scene data corresponding to a predetermined reference scene; processing the first candidate scene data and the reference scene data, using a scene discriminator, to generate first discrimination data for estimating whether each of the first candidate scene data and the reference scene data corresponds to a predetermined reference scene; updating a set of parameter values for the scene discriminator using the first discrimination data; generating, using the scene generator, second candidate scene data; processing the second candidate scene data, using the scene discriminator with the updated set of parameter values for the scene discriminator, to generate second discrimination data for estimating whether the second candidate scene data corresponds to a predetermined reference scene; and updating a set of parameter values for the scene generator using the second discrimination data. ## Claims ### Claim 1 A computer-implemented method comprising: generating, using a scene generator, first candidate scene data indicating a position and orientation for each of a first candidate set of three-dimensional objects with respect to a given co-ordinate system; obtaining reference scene data corresponding to a first predetermined reference scene of a set of predetermined reference scenes and indicating a position and orientation for each of a reference set of three-dimensional objects with respect to the given co-ordinate system; processing the first candidate scene data and the reference scene data, using a scene discriminator, to generate first discrimination data for estimating whether each of the first candidate scene data and the reference scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; updating a set of parameter values for the scene discriminator using the first discrimination data; generating, using the scene generator, second candidate scene data indicating a position and orientation for each of a second candidate set of three-dimensional objects with respect to the given co-ordinate system; processing the second candidate scene data, using the scene discriminator with the updated set of parameter values for the scene discriminator, to generate second discrimination data for estimating whether the second candidate scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; and updating a set of parameter values for the scene generator using the second discrimination data. ### Claim 2 (depends on 1) The method of claim 1, comprising: generating, using the scene generator with the updated set of parameter values, output scene data indicating a position and orientation for each of an output set of three-dimensional objects with respect to the given co-ordinate system; and processing the output scene data to generate a corresponding synthetic three-dimensional scene. ### Claim 3 (depends on 2) The method of claim 2, comprising rendering the generated synthetic three-dimensional scene to generate an image of the generated synthetic three-dimensional scene. ### Claim 4 (depends on 3) The method of claim 3, wherein the generated image comprises depth information. ### Claim 5 (depends on 3) The method of claim 3, wherein the generated image comprises annotations indicative of the positions and orientation of the three-dimensional objects in the generated synthetic three-dimensional scene. ### Claim 6 (depends on 1) The method of claim 1, wherein: the scene generator comprises an artificial neural network; and generating the each of the first candidate scene data and the second candidate scene data comprises: sampling a respective noise variable from a predetermined noise distribution; and processing the sampled respective noise variable using the artificial neural network to generate the candidate scene data. ### Claim 7 (depends on 1) The method of claim 1, wherein the scene discriminator comprises an artificial neural network. ### Claim 8 (depends on 1) The method of claim 1, wherein: the first discrimination data indicates whether each of the first candidate scene data and the reference scene data corresponds to a real-life scene of a given scene type; and the second discrimination data comprises an estimated probability that the second candidate scene data corresponds to a real-life scene of a given scene type. ### Claim 9 (depends on 8) The method of any claim 8, wherein obtaining the reference scene data comprises processing an image of a predetermined three-dimensional scene using a computer vision system. ### Claim 10 (depends on 1) The method of claim 1, wherein: the first discrimination data comprises an estimated probability that each of the first candidate scene data and the reference scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; and the second discrimination data comprises an estimated probability that the second candidate scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes. ### Claim 11 (depends on 1) The method of claim 1, wherein: the first discrimination data is indicative of a distance between the first candidate scene data and the reference scene data. ### Claim 12 (depends on 1) The method of claim 1, wherein each of the first candidate scene data, the second candidate scene data, and the reference scene data indicates one or more transformations applied to three-dimensional objects in the respective set of three-dimensional objects. ### Claim 13 (depends on 12) The method of claim 12, wherein each of the first candidate scene data, the second candidate scene data, and the reference scene data indicates one or more object parameters for three-dimensional objects in the respective candidate set of three-dimensional objects. ### Claim 14 (depends on 1) The method of claim 1, wherein each of the first candidate scene data, the second candidate scene data, and the reference scene data indicates one or more lighting conditions and/or one or more weather conditions. ### Claim 15 A data processing system comprising: an object database arranged to store a plurality of labelled three-dimensional objects; a scene generator operable to generate first candidate scene data indicating a position and orientation for each of a first candidate set of the labelled three-dimensional objects with respect to a given co-ordinate system; a reference scene database arranged to store reference scene data corresponding to a first predetermined reference scene of a set of predetermined reference scenes and indicating a position and orientation for each of a reference set of labelled three-dimensional objects with respect to the given co-ordinate system; a scene discriminator operable to process the first candidate scene data and the reference scene data to generate first discrimination data for estimating whether each of the first candidate scene data and the reference scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; and a learner operable to update a set of parameter values for the scene discriminator using the first discrimination data, wherein: the scene generator is further operable to generate second candidate scene data indicating a position and orientation for each of a second candidate set of labelled three-dimensional objects with respect to the given co-ordinate system; the scene discriminator is further operable to process the second candidate scene data, using the updated set of parameter values for the scene discriminator, to generate second discrimination data for estimating whether the second candidate scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; and the learner is further operable to update a set of parameter values for the scene generator using the second discrimination data. ### Claim 16 (depends on 15) The data processing system of claim 15, comprising: a three-dimensional engine operable to process scene data generated by the scene generator to generate a corresponding synthetic three-dimensional scene. ### Claim 17 A non-transient storage medium comprising machine-readable instructions which, when executed by a computer, cause the computer to perform the steps of: generating, using a scene generator, first candidate scene data indicating a position and orientation for each of a first candidate set of three-dimensional objects with respect to a given co-ordinate system; obtaining reference scene data corresponding to a first predetermined reference scene of a set of predetermined reference scenes and indicating a position and orientation for each of a reference set of three-dimensional objects with respect to the given co-ordinate system; processing the first candidate scene data and the reference scene data, using a scene discriminator, to generate first discrimination data for estimating whether each of the first candidate scene data and the reference scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; updating a set of parameter values for the scene discriminator using the first discrimination data; generating, using the scene generator, second candidate scene data indicating a position and orientation for each of a second candidate set of three-dimensional objects with respect to the given co-ordinate system; processing the second candidate scene data, using the scene discriminator with the updated set of parameter values for the scene discriminator, to generate second discrimination data for estimating whether the second candidate scene data corresponds to any predetermined reference scene of the set of predetermined reference scenes; and updating a set of parameter values for the scene generator using the second discrimination data.