# US 10,504,247 B2 · Transmission apparatus, mixed reality system, transmission method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,247 B2
- Document ID: US10504247B2
- Status: Granted
- Kind: B2
- Application number: 15/548,176
- Filed: 2016-03-30
- Published: 2018-02-01
- Granted: 2019-12-10
- Earliest claimed: 2015-04-23
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Shun Sugimoto
- Web page: https://patentgrep.com/patent/US10504247B2

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G02B 27/0172: {characterised by optical features}
- G06F 3/147: using display panels
- G06T 7/74: {involving reference images or patches}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- H04L 65/80: Responding to QoS
- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/179: the unit being a scene or a shot
- H04N 21/2408: {Monitoring of the upstream path of the transmission network, e.g. client requests (monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/2662: Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
- H04N 21/4122: {additional display device, e.g. video projector (digital output for controlling a plurality of local displays G06F3/1423)}
- H04N 21/4143: {embedded in a} Personal Computer [PC]
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4318: {by altering the content in the rendering process, e.g. blanking, blurring or masking an image region (image enhancement or restoration in general G06T5/00)}
- H04N 21/6332: directed to client
- H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2207/30244: Camera pose
- G09G 2340/02: Handling of images in compressed format, e.g. JPEG, MPEG
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2370/10: Use of a protocol of communication by packets in interfaces along the display data pipeline
- G09G 2370/16: Use of wireless transmission of display information
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A transmission apparatus transmits, to a reception apparatus, first video captured by an imaging apparatus and second video captured to detect a position and orientation of the imaging apparatus when capturing the first video, and includes a memory containing instructions, and at least one processor for executing the instructions. The instructions are executed to operate as a control unit to control transmission rates of the first and second videos so that a transmission rate ratio between the first video and the second video changes, and a transmission unit to transmit the first video and the second video to the reception apparatus in accordance with the controlled transmission rates. The control unit further controls the transmission rates of the first video and the second video so that the transmission rate of the second video decreases if a communication bandwidth between the transmission apparatus and the reception apparatus decreases.

## Claims

### Claim 1

A transmission apparatus for transmitting, to a reception apparatus, first video captured by an imaging apparatus and second video captured to detect a position and orientation of the imaging apparatus when capturing the first video, comprising:
a memory containing instructions; and
at least one processor for executing the instructions, the instructions being executed to operate as
a control unit configured to control a transmission rate of the first video and a transmission rate of the second video so that a transmission rate ratio between the first video and the second video changes; and
a transmission unit configured to transmit the first video and the second video to the reception apparatus in accordance with the controlled transmission rates,
wherein:
if a communication bandwidth between the transmission apparatus and the reception apparatus is smaller than a sum of a first setting value and a second setting value, the control unit controls the transmission rate of the first video to be smaller than the first setting value, and controls the transmission rate of the second video to be equal to the second setting value; and
if the communication bandwidth exceeds the sum of the first setting value and the second setting value, the control unit controls the transmission rate of the first video to be equal to the first setting value, and controls the transmission rate of the second video to exceed the second setting value.

### Claim 2 (depends on 1)

The transmission apparatus according to claim 1, further executing the instructions to operate as a bandwidth determination unit configured to determine the communication bandwidth in accordance with condition information which has been received from the reception apparatus,
wherein the condition information includes at least one of:
a communication error rate and delay information in data transmission from the transmission apparatus to the reception apparatus; or
information about a bandwidth usable for data transmission from the transmission apparatus to the reception apparatus.

### Claim 3 (depends on 1)

The transmission apparatus according to claim 1, wherein the control unit is further configured to control the transmission rates of the first video and the second video so that the transmission rate ratio between the first video and the second video changes in accordance with information which has been transmitted from the reception apparatus and indicates estimation accuracy of the position and orientation of the imaging apparatus using the second video.

### Claim 4 (depends on 3)

The transmission apparatus according to claim 3, wherein the information indicating the estimation accuracy of the position and orientation of the imaging apparatus includes the number of indices which are arranged in a physical space and have been detected from the second video to detect the position and orientation of the imaging apparatus.

### Claim 5 (depends on 3)

The transmission apparatus according to claim 3, wherein the information indicating the estimation accuracy of the position and orientation of the imaging apparatus includes a difference between the number of indices detected from the current second video and the number of indices detected from the second video a predetermined time before.

### Claim 6 (depends on 1)

The transmission apparatus according to claim 1, wherein the control unit is further configured to control the transmission rates of the first video and the second video so that the transmission rate ratio between the first video and the second video changes in accordance with a moving amount of the imaging apparatus.

### Claim 7 (depends on 1)

The transmission apparatus according to claim 1, wherein:
the reception apparatus generates a synthesized video by synthesizing the first video and a superimposing image generated in accordance with the position and orientation of the imaging apparatus detected using the second video; and
the control unit is further configured to control the transmission rates of the first video and the second video so that the transmission rate ratio between the first video and the second video changes in accordance with a size of the superimposing image occupying the synthesized video.

### Claim 8 (depends on 1)

The transmission apparatus according to claim 1, wherein:
the reception apparatus generates a synthesized video by synthesizing the first video and a superimposing image generated in accordance with the position and orientation of the imaging apparatus detected using the second video; and
the control unit is further configured to control the transmission rates of the first video and the second video in accordance with a weight of the second video determined through at least one of:
increasing the weight of the second video if estimation accuracy of the position and orientation of the imaging apparatus using the second video, which is obtained in accordance with information transmitted from the reception apparatus, is lower than a threshold;
increasing the weight of the second video if a moving amount of the imaging apparatus is larger than a threshold; or
increasing the weight of the second video if a size of the superimposing image occupying the synthesized video is larger than the threshold.

### Claim 9 (depends on 8)

The transmission apparatus according to claim 8, wherein the control unit is further configured to determine the weight of the second video further through at least one of:
increasing the weight of the second video in response to user designation; or
increasing the weight of the second video if the transmission rate of the first video has reached a target.

### Claim 10

A transmission apparatus for transmitting, to a reception apparatus, first video captured by an imaging apparatus and second video captured to detect a position and orientation of the imaging apparatus when capturing the first video, comprising:
a memory containing instructions; and
at least one processor for executing the instructions, the instructions being executed to operate as
a bandwidth determination unit configured to determine a communication bandwidth between the transmission apparatus and the reception apparatus;
an accuracy determination unit configured to determine, using condition information received from the reception apparatus, estimation accuracy of the position and orientation of the imaging apparatus using the second video;
a control unit configured to determine transmission rates of the first video and the second video in response to determining that the communication bandwidth has become narrow, and decrease the transmission rate of the second video in response to determining that the estimation accuracy exceeds a threshold and decrease the transmission rate of the first video in response to determining that the estimation accuracy is lower than the threshold;
a first compression unit configured to compress the first video in accordance with the determined transmission rate;
a second compression unit configured to compress the second video in accordance with the determined transmission rate; and
a transmission unit configured to transmit the compressed first video and the compressed second video to the reception apparatus.

### Claim 11

A transmission apparatus for transmitting, to a reception apparatus, first video captured by an imaging apparatus and second video captured to detect a position and orientation of the imaging apparatus when capturing the first video, comprising:
a memory containing instructions; and
at least one processor for executing the instructions, the instructions being executed to operate as
a bandwidth determination unit configured to determine a communication bandwidth between the transmission apparatus and the reception apparatus;
a control unit configured to determine, in response to determining that the communication bandwidth has become narrow, transmission rates of the first video and the second video, and decrease the transmission rate of the second video in response to determining that the current transmission rate of the second video exceeds a threshold and decrease the transmission rate of the first video in response to determining that the current transmission rate of the second video is not higher than the threshold;
a first compression unit configured to compress the first video in accordance with the determined transmission rate;
a second compression unit configured to compress the second video in accordance with the determined transmission rate; and
a transmission unit configured to transmit the compressed first video and the compressed second video to the reception apparatus.

### Claim 12 (depends on 1)

The transmission apparatus according to claim 1, wherein the transmission apparatus is a head mounted display, and the transmission apparatus further comprises:
an imaging apparatus including a first camera which captures the first video and a second camera which captures the second video; and
the instructions are further executed to operate as
a reception unit configured to receive a synthesized video generated by synthesizing the first video and a superimposing image generated in accordance with the position and orientation of the imaging apparatus detected using the second video; and
a presentation unit configured to present the synthesized video to a user.

### Claim 13 (depends on 12)

A mixed reality system comprising:
a transmission apparatus according to claim 12; and
an external apparatus configured to receive first video and second video, generate a synthesized video by synthesizing the first video and a superimposing image generated in accordance with a position and orientation of an imaging apparatus detected using the second video, and transmit the synthesized video to the transmission apparatus.

### Claim 14

A transmission method comprising:
controlling a transmission rate of first video and a transmission rate of second video so that a transmission rate ratio between the first video and the second video changes, wherein the first video is captured by an imaging apparatus and the second video is captured to detect a position and orientation of the imaging apparatus when capturing the first video; and
transmitting the first video and the second video to a reception apparatus in accordance with the controlled transmission rates,
wherein:
if a communication bandwidth between a transmission apparatus and the reception apparatus is smaller than a sum of a first setting value and a second setting value, the transmission rate of the first video is controlled to be smaller than the first setting value, and the transmission rate of the second video is controlled to be equal to the second setting value; and
if the communication bandwidth exceeds the sum of the first setting value and the second setting value, the transmission rate of the first video is controlled to be equal to the first setting value, and the transmission rate of the second video is controlled to exceed the second setting value.

### Claim 15

A non-transitory computer-readable medium storing a program comprising instructions for causing a computer to perform a method comprising:
controlling a transmission rate of first video and a transmission rate of second video so that a transmission rate ratio between the first video and the second video changes, wherein the first video is captured by an imaging apparatus and the second video is captured to detect a position and orientation of the imaging apparatus when capturing the first video; and
transmitting the first video and the second video to a reception apparatus in accordance with the controlled transmission rates,
wherein:
if a communication bandwidth between a transmission apparatus and the reception apparatus is smaller than a sum of a first setting value and a second setting value, the transmission rate of the first video is controlled to be smaller than the first setting value, and the transmission rate of the second video is controlled to be equal to the second setting value; and
if the communication bandwidth exceeds the sum of the first setting value and the second setting value, the transmission rate of the first video is controlled to be equal to the first setting value, and the transmission rate of the second video is controlled to exceed the second setting value.