# US 2020/0287953 A1 · Pluggable components for augmenting device streams

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0287953 A1
- Document ID: US20200287953A1
- Status: Published application
- Kind: A1
- Application number: 16/881,496
- Filed: 2020-05-22
- Published: 2020-09-10
- Earliest claimed: 2016-11-28
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Mei L. WILSON; Christian P. LARSON; Sang K. CHOE; Yin LI
- Web page: https://patentgrep.com/patent/US20200287953A1

## CPC

- H04L 65/762: {at the source (reformatting of additional data in video distribution servers H04N21/2355)}
- G06F 3/002: {Specific input/output arrangements not covered by G06F3/01 - G06F3/16 (other optical apparatus G02B27/00)}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 9/4411: {Configuring for operating with peripheral devices; Loading of device drivers}
- G06F 9/44526: {Plug-ins; Add-ons}
- G06F 13/00: Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units (interface circuits for specific input/output devices G06F3/00 {; multiprogram control therefor G06F9/46}; multiprocessor systems G06F15/16 )
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- G06F 9/544: {Buffers; Shared memory; Pipes}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described are examples for providing stream augmenting components. A stream augmenting component having an augmentation function indicating a plurality of interfaces to one or more devices and an output stream can be initialized. At least a portion of the plurality of interfaces for the one or more devices can be accessed to at least one of obtain stream output from the one or more devices or control the one or more devices. A plurality of stream outputs from the one or more devices can be virtualized to the output stream at least in part by modifying or correlating data received over the plurality of stream outputs. The stream augmenting component can output the output stream to one or more other stream augmenting components or applications via at least another portion of the one or more interfaces.

## Claims

### Claim 1

A computing device for providing stream augmenting components, comprising:
a memory storing one or more parameters or instructions for executing a stream server, wherein the stream server exposes one or more interfaces related to one or more stream augmenting components; and
at least one processor coupled to the memory, wherein the at least one processor is configured to:
initialize a stream augmenting component having an augmentation function indicating a plurality of interfaces to one or more devices and an output stream;
access at least a portion of the plurality of interfaces for the one or more devices to at least one of obtain stream output from the one or more devices or control the one or more devices;
virtualize a plurality of stream outputs from the one or more devices to the output stream at least in part by modifying or correlating data received over the plurality of stream outputs; and
output, by the stream augmenting component, the output stream to one or more other stream augmenting components or applications via at least another portion of the one or more interfaces.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein the at least one processor is configured to access at least the portion of the one or more interfaces for at least one device of the one or more devices to control the at least one device to modify the stream output obtained from the at least one device.

### Claim 3 (depends on 2)

The computing device of claim 2, wherein the at least one processor is configured to control the at least one device based at least in part on the stream output received for at least another device of the one or more devices.

### Claim 4 (depends on 3)

The computing device of claim 3, wherein the stream output received for the at least another device is received from another augmentation function of another stream augmenting component that indicates an interface to the another device.

### Claim 5 (depends on 3)

The computing device of claim 3, wherein the at least one device is a camera, wherein the at least another device is a depth camera, and wherein the at least one processor is configured to control the camera based at least in part on depth information in the stream output received from the depth camera.

### Claim 6 (depends on 1)

The computing device of claim 1, wherein the at least one processor is further configured to initialize, based on initializing the stream augmenting component, another stream augmenting component having another augmentation function with another interface that accesses the output stream.

### Claim 7 (depends on 1)

The computing device of claim 1, wherein the at least one processor is configured to virtualize the stream output at least in part by adding metadata from other devices to framed data in the stream output.

### Claim 8 (depends on 1)

The computing device of claim 1, wherein the at least one processor is configured to access at least the portion of the one or more interfaces for multiple devices of the one or more devices to control, based on an operating parameter, the multiple devices to modify the stream output from the multiple devices.

### Claim 9 (depends on 8)

The computing device of claim 8, wherein the operating parameter is a time offset.

### Claim 10

A method for providing stream augmenting components, comprising:
initializing a stream augmenting component having an augmentation function indicating a plurality of interfaces to one or more devices and an output stream;
accessing at least a portion of the plurality of interfaces for the one or more devices to at least one of obtain stream output from the one or more devices or control the one or more devices;
virtualizing a plurality of stream outputs from the one or more devices to the output stream at least in part by modifying or correlating data received over the plurality of stream outputs; and
outputting, by the stream augmenting component, the output stream to one or more other stream augmenting components or applications via at least another portion of the one or more interfaces.

### Claim 11 (depends on 10)

The method of claim 10, wherein accessing at least the portion of the one or more interfaces includes accessing for at least one device of the one or more devices to control the at least one device to modify the stream output from the at least one device.

### Claim 12 (depends on 11)

The method of claim 11, wherein controlling the at least one device is based at least in part on the stream output received for at least another device of the one or more devices.

### Claim 13 (depends on 12)

The method of claim 12, wherein the stream output received for the at least another device is received from another augmentation function of another stream augmenting component that indicates an interface to the another device.

### Claim 14 (depends on 12)

The method of claim 12, wherein the at least one device is a camera, wherein the at least another device is a depth camera, and wherein controlling the camera is based at least in part on depth information in the stream output received from the depth camera.

### Claim 15 (depends on 10)

The method of claim 10, further comprising initializing, based on initializing the stream augmenting component, another stream augmenting component having another augmentation function with another interface that accesses the output stream.

### Claim 16 (depends on 10)

The method of claim 10, wherein virtualizing the stream output includes adding metadata from other devices to framed data in the stream output.

### Claim 17 (depends on 10)

The method of claim 10, wherein accessing at least the portion of the one or more interfaces includes accessing for multiple devices of the one or more devices to control, based on an operating parameter, the multiple devices to modify the stream output from the multiple devices.

### Claim 18 (depends on 17)

The method of claim 17, wherein the operating parameter is a time offset.

### Claim 19

A non-transitory computer-readable medium, comprising code executable by one or more processors for providing stream augmenting components, the code comprising code for:
initializing a stream augmenting component having an augmentation function indicating a plurality of interfaces to one or more devices and an output stream;
accessing at least a portion of the plurality of interfaces for the one or more devices to at least one of obtain stream output from the one or more devices or control the one or more devices;
virtualizing a plurality of stream outputs from the one or more devices to the output stream at least in part by modifying or correlating data received over the plurality of stream outputs; and
outputting, by the stream augmenting component, the output stream to one or more other stream augmenting components or applications via at least another portion of the one or more interfaces.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the code for accessing at least the portion of the one or more interfaces for at least one device of the one or more devices accesses to control the at least one device to modify the stream output from the at least one device.
