# US 2021/0243491 A1 · Source devices, sink devices, methods and computer programs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243491 A1
- Document ID: US20210243491A1
- Status: Published application
- Kind: A1
- Application number: 17/239,217
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2018-10-24
- Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Applicants: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Inventors: Stefan KRAEGELOH; Thomas HELLER; Rinat ZEH; Herbert THOMA
- Web page: https://patentgrep.com/patent/US20210243491A1

## CPC

- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/43635: {HDMI}
- H04N 21/485: End-user interface for client configuration
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A source device for providing audio data and video data on one or more digital media interfaces is described, wherein the source device is configured to automatically adjust a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path.

## Claims

### Claim 1

A source device for providing audio data and video data on one or more digital media interfaces,
wherein the source device is configured to automatically adjust a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path,
wherein the source device is configured to acquire an encoded video representation,
to decode the encoded video representation, to acquire a decoded video representation, and
to provide the video data, such that the video data represent the decoded video representation, and
wherein the source device is configured to selectively delay the decoding of the encoded video representation based on the information about the latency of the audio path and the information about the latency of the video path.
wherein the source device is configured to acquire the information about the latency of the audio path using an audio delay information which is acquired from an audio playback device via one of the one or more digital interfaces, and/or
wherein the source device is configured to acquire the information about the latency of the video path using a video delay information which is acquired from a video rendering device via one of the one or more digital interfaces.

### Claim 2 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to negotiate a decoding latency with an audio playback device, and to adjust the time alignment in dependence on the negotiated decoding latency.

### Claim 3 (depends on 2)

The source device according to claim 2,
wherein the source device is configured to acquire a latency span information describing a span of possible latency values from the audio playback device, or to acquire a latency list information describing one or more possible latency values from the audio playback device, and
wherein the source device is configured to select a desired latency value on the basis of the latency span information or the latency list information; and
wherein the source device is configured to instruct the audio playback device to use the desired latency value; and
wherein the source device is configured to adjust the time alignment in dependence on the selected desired audio playback latency value.

### Claim 4 (depends on 3)

The source device according to claim 3,
wherein the source device is configured to verify whether the audio playback device uses the selected desired latency value, as instructed, and
wherein the source device is configured to consider a default latency value if the audio playback device fails to use the selected desired latency value.

### Claim 5 (depends on 3)

The source device according to claim 3,
wherein the source device is configured to receive a latency negotiation request comprising a latency span information describing a span of possible latency values from the audio playback device, or comprising a latency list information describing one or more possible latency values from the audio playback device, and
wherein the source device is configured to select a desired latency value on the basis of the latency span information or the latency list information.

### Claim 6 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to discover a topology of a connection between the source device and an audio playback device and also of a connection between the source device and a video rendering device, in order to acquire a topology information, and
wherein the source device is configured to adjust the time alignment in dependence on the topology information.

### Claim 7 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to acquire a total audio latency information describing a total latency of an audio path for a first audio mode or describing an average total latency of the audio path over a plurality of audio modes, and/or
wherein the source device is configured to acquire a total video latency information describing a total latency of a video path for a first video mode or describing an average total latency of the video path over a plurality of video modes.

### Claim 8 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to query a plurality of individual devices for device-individual latency information.

### Claim 9 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to determine the information about the latency of the video path
using a summation of device-individual current video latency information, if device-individual current video latency information is available to the source device for the video rendering device and all devices between the source device and the video rendering device and/or
using
a total video latency information describing a total latency of a video path for a first video mode or describing an average total latency of the video path over a plurality of video modes,
a latency information describing a contribution of the video rendering device to the total video latency information, and
a current video latency information of the video rendering device,
if the total video latency information, the latency information describing a contribution of the video rendering device to the total video latency information and the current video latency information of the video rendering device is available to the source device but some device-individual current video latency information is unavailable to the source device; and/or
using the total video rendering information if the total video rendering information is available to the source device but the current video latency information of the video rendering device is unavailable to the source device.

### Claim 10 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to determine the information about the latency of the audio path
using a summation of device-individual passthrough latency information associated with a passthrough of audio information for the second audio mode, if device-individual passthrough latency information is available for all devices between the source device and the audio playback device; and/or
using a total audio latency information describing a total latency of an audio path for a first audio mode or describing an average total latency of the audio path over a plurality of audio modes; and/or
using a summation of any known or estimated audio latency information of devices in an audio path between the source device and the audio playback device.

### Claim 11 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to correct an information about the latency of the audio path acquired using the total audio latency information using latency information associated with a playback using the first audio mode and using latency information about a latency associated with a passthrough of audio information for the second audio mode.

### Claim 12 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to select a determination rule for a determination of the information about the latency of the video path in dependence on an amount of information available to the source.

### Claim 13 (depends on 12)

The source device according to claim 12,
wherein the source device is configured to determine a delay to be applied in the provision of audio data and/or a delay to be applied in the provision of the video data in dependence on the information about the latency of the video path and in dependence on the information about the latency of the audio path.

### Claim 14 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to also allow for an adjustment of the time alignment using a user interface.

### Claim 15 (depends on 1)

The source device according to claim 1,
wherein the source device is configured to communicate with the audio playback device and/or the video rendering device and/or one or more devices between the source device and the audio playback device and/or one or more devices between the source device and the video rendering device using an additional link, which is separate from the one or more digital media interfaces, to acquire latency information from the one or more devices.

### Claim 16

A sink device,
wherein the sink device is configured to receive audio data via a digital media interface,
wherein the sink device is configured to decode the received audio data, to acquire decoded audio data, and
wherein the sink device is configured to output the decoded output data;
wherein the sink device is configured to report a device-individual latency information which comprises latency information associated with a playback using the first audio mode and latency information associated with a playback using a second audio mode, in response to a request from a source device, and
wherein the sink device is configured to negotiate a decoding latency or a playback latency associated with the second audio mode with a source device.

### Claim 17 (depends on 16)

The sink device according to claim 16,
wherein the sink device is configured
to provide a latency span information describing a span of possible decoding latency values or playback latency values to the source device, or
to provide a latency list information describing one or more possible decoding latency values or playback latency values to the source device; and
wherein the sink device is configured to receive a latency selection message from the source device and to set a decoding latency or playback latency in response to the latency selection message.

### Claim 18 (depends on 16)

The sink device according to claim 16,
wherein the sink device is configured to provide a contribution to a total audio latency information describing a total latency of an audio path for a first audio mode or describing an average total latency of the audio path over a plurality of audio modes,
wherein the contribution to the total audio latency information is different from the latency information associated with a playback using a second audio mode.

### Claim 19 (depends on 16)

The sink device according to claim 16,
wherein the contribution to a total audio latency information provided by the sink device is equal to the latency information associated with a playback using the first audio mode.

### Claim 20

A sink device,
wherein the sink device is configured to receive video data via a digital media interface,
wherein the sink device is configured to render the received video data;
wherein the sink device is configured to report a device-individual latency information which comprises
static latency information representing a contribution of the sink device to a total video latency information describing a total latency of a video path or describing an average total latency of the video path over a plurality of video modes; and
latency information representing an actual latency of a rendering of video data,
wherein the contribution to the total video latency information is different from the latency information representing the actual latency of a rendering of video data.

### Claim 21 (depends on 20)

The sink device according to claim 20,
wherein the static latency information representing the contribution to the total video latency information is different from the latency information representing the actual latency of a rendering of video data in a way that
the static latency information representing the contribution to the total video latency information represents a video latency of the sink device when using a standard video mode for rendering the video data, and
the latency information representing the actual latency of a rendering of video data represents a video latency of the sink device for a currently used video mode for rendering the video data.

### Claim 22 (depends on 20)

The sink device according to claim 20,
wherein the device-individual latency information reported by the sink device also comprises
latency information associated with an audio playback using a first audio mode; and/or
average latency information associated with an audio playback using a plurality of different modes; and/or
latency information associated with a passthrough of audio information for the second audio mode.

### Claim 23 (depends on 20)

The sink device according to claim 20,
wherein the sink device is configured to provide a contribution to a total audio latency information describing a total latency of an audio path for a first audio mode or describing an average total latency of the audio path over a plurality of audio modes,
wherein the contribution to the total audio latency information is different from the latency information associated with a passthrough of audio information for the second audio mode.

### Claim 24 (depends on 20)

The sink device according to claim 20,
wherein the sink device is configured to provide the device-individual latency information such that the device-individual latency information reported by the sink device comprises a latency information associated with a forwarding of audio data 160;260;660;760;860;960 from a first digital media interface to a second digital media interface.

### Claim 25

A method for providing audio data and video data on one or more digital media interfaces,
wherein the method comprises automatically adjusting a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path,
wherein the method comprises:
acquiring an encoded video representation,
decoding the encoded video representation, to acquire a decoded video representation, and
providing the video data, such that the video data represent the decoded video representation, and
wherein the method comprises selectively delaying the decoding of the encoded video representation based on the information about the latency of the audio path and the information about the latency of the video path, and
wherein the method comprises
acquiring the information about the latency of the audio path using an audio delay information which is acquired from an audio playback device via one of the one or more digital interfaces, and/or
acquiring the information about the latency of the video path using a video delay information which is acquired from a video rendering device via one of the one or more digital interfaces.

### Claim 26

A method for operating a sink device,
wherein the method comprises receiving audio data via a digital media interface,
wherein the method comprises decoding the received audio data, to acquire decoded audio data, and
wherein the method comprises outputting the decoded output data;
wherein the method comprises reporting a device-individual latency information which comprises latency information associated with a playback using the first audio mode and latency information associated with a playback using a second audio mode, in response to a request from a source device, and
wherein the method comprises negotiating a decoding latency or a playback latency associated with the second audio mode with a source device.

### Claim 27

A method for operating a sink device,
wherein the method comprises receiving video data via a digital media interface,
wherein the method comprises rendering the received video data
wherein the method comprises reporting a device-individual latency information which comprises
latency information representing a contribution of the sink device to a total video latency information describing a total latency of a video path or describing an average total latency of the video path over a plurality of video modes; and
latency information representing an actual latency of a rendering of video data,
wherein the contribution to the total audio latency information is different from the latency information representing the actual latency of a rendering of video data.

### Claim 28

A source device for providing audio data and video data on one or more digital media interfaces,
wherein the source device is configured to automatically adjust a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path,
wherein the source device is configured to negotiate a decoding latency with an audio playback device, and to adjust the time alignment in dependence on the negotiated decoding latency.

### Claim 29

A source device for providing audio data and video data on one or more digital media interfaces, wherein the source device is configured
to automatically adjust a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path; and
to negotiate a decoding latency with an audio playback device, and to adjust the time alignment in dependence on the negotiated decoding latency; and
to acquire a latency span information describing a span of possible latency values from the audio playback device, or to acquire a latency list information describing one or more possible latency values from the audio playback device; and
to select a desired latency value on the basis of the latency span information or the latency list information such that the desired latency value fits an information about the latency of the video path; and
to instruct the audio playback device to use the desired latency value; and
to adjust the time alignment in dependence on the selected desired audio playback latency value.

### Claim 30

A method for providing audio data and video data on one or more digital media interfaces, comprising
automatically adjusting a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path;
negotiating a decoding latency with an audio playback device, and to adjust the time alignment in dependence on the negotiated decoding latency.

### Claim 31

A method for providing audio data and video data on one or more digital media interfaces, comprising
automatically adjusting a time alignment between a provision of audio data and a provision of video data based on an information about a latency of an audio path and based on an information about a latency of a video path;
negotiating a decoding latency with an audio playback device, and to adjust the time alignment in dependence on the negotiated decoding latency;
acquiring a latency span information describing a span of possible latency values from the audio playback device, or acquiring a latency list information describing one or more possible latency values from the audio playback device; and
selecting a desired latency value on the basis of the latency span information or the latency list information such that the desired latency value fits an information about the latency of the video path; and
instructing the audio playback device to use the desired latency value; and
adjusting the time alignment in dependence on the selected desired audio playback latency value.

### Claim 32 (depends on 25)

A non-transitory digital storage medium having stored thereon a computer program for performing the method for providing audio data and video data on one or more digital media interfaces according to claim 25 when said computer medium is run by a computer.

### Claim 33 (depends on 26)

A non-transitory digital storage medium having stored thereon a computer program for performing the method for operating a sink device according to claim 26 when said computer medium is run by a computer.

### Claim 34 (depends on 27)

A non-transitory digital storage medium having stored thereon a computer program for performing the method for operating a sink device according to claim 27 when said computer medium is run by a computer.

### Claim 35 (depends on 30)

A non-transitory digital storage medium having stored thereon a computer program for performing the method for providing audio data and video data on one or more digital media interfaces according to claim 30 when said computer medium is run by a computer.

### Claim 36 (depends on 31)

A non-transitory digital storage medium having stored thereon a computer program for performing the method for providing audio data and video data on one or more digital media interfaces according to claim 31 when said computer medium is run by a computer.
