Capturing objects in an unstructured video stream

A first electronic device comprising:

• one or more processors;
• one or more image sensors;
• a non-transitory memory;
• and
• one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
• obtaining, using the one or more image sensors, a first unstructured video stream that provides pixel values for a plurality of pixels for rendering in an extended reality (XR) environment, wherein the one or more image sensors capture pass-through image data including a portion of a second unstructured video stream being displayed on a secondary display of a second electronic device that is different from the first electronic device;
• generating respective pixel characterization vectors for a portion of the plurality of pixels, wherein each of the respective pixel characterization vectors is associated with a corresponding pixel in the portion of the plurality of pixels, wherein generating each of the respective pixel characterization vectors includes determining a respective instance label value and adding the respective instance label value to each of the respective pixel characterization vectors indicating separate objects in one or more images of the first unstructured video stream;
• identifying a first object within the portion of the plurality of pixels based on satisfying an object confidence threshold that indicates that pixels are associated with pixel characterization vectors that each include a first instance label value that is associated with an object represented in the portion of the plurality of pixels;
• generating respective semantic label values corresponding to pixels associated with the first object, wherein the respective semantic label values are added to pixel characterization vectors associated with the first object and characterize the first object;
• and
• providing a first XR affordance corresponding to the identified first object to instantiate an objective-effectuator, wherein the objective-effectuator performing actions in the XR environment is characterized by the respective semantic label values.

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a first electronic device including one or more processors and one or more image sensors, cause the first electronic device to:

• obtain, using the one or more image sensors, a first unstructured video stream that provides pixel values for a plurality of pixels for rendering in an extended reality (XR) environment, wherein the one or more image sensors capture pass-through image data including a portion of a second unstructured video stream being displayed on a secondary display of a second electronic device that is different from the first electronic device;
• generate respective pixel characterization vectors for a portion of the plurality of pixels, wherein each of the respective pixel characterization vectors is associated with a corresponding pixel in the portion of the plurality of pixels, wherein generating each of the respective pixel characterization vectors includes determining a respective instance label value and adding the respective instance label value to each of the respective pixel characterization vectors indicating separate objects in one or more images of the first unstructured video stream;
• identifying a first object within the portion of the plurality of pixels based on satisfying an object confidence threshold that indicates that pixels are associated with pixel characterization vectors that each include a first instance label value that is associated with an object represented in the portion of the plurality of pixels;
• generate respective semantic label values corresponding to pixels associated with the first object, wherein the respective semantic label values are added to pixel characterization vectors associated with the first object and characterize the first object;
• and
• provide a first XR affordance corresponding to the identified first object to instantiate an objective-effectuator, wherein the objective-effectuator performing actions in the XR environment is characterized by the respective semantic label values.