# US 11,113,849 B2 · Method of controlling virtual content, terminal device and computer readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,849 B2 - Document ID: US11113849B2 - Status: Granted - Kind: B2 - Application number: 16/720,015 - Filed: 2019-12-19 - Published: 2020-04-23 - Granted: 2021-09-07 - Earliest claimed: 2018-08-10 - Assignees: Guangdong Virtual Reality Technology Co., Ltd. - Applicants: Guangdong Virtual Reality Technology Co., Ltd. - Inventors: Yiqun Wu; Yongtao Hu; Jingwen Dai; Jie He - Web page: https://patentgrep.com/patent/US11113849B2 ## CPC - G06V 20/20: in augmented reality scenes - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/00: Two-dimensional [2D] image generation - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06T 2207/30204: Marker - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of controlling virtual content, a terminal device and a computer readable medium are provided. The terminal device includes a display configured to display a virtual object, a camera configured to capture a target image including a marker, at least one processor, and a memory. The memory stores one or more programs configured to be executed by the at least one processor. The one or more programs includes instructions of acquiring the target image including the marker, where a plurality of sub-markers is distributed on the marker; determining a blocked target, wherein the blocked target is a blocked sub-marker of the plurality of sub-markers in the target image; generating a control instruction according to the blocked target; and controlling the displayed virtual object based on the control instruction. ## Claims ### Claim 1 A terminal device, comprising: a display configured to display a virtual object; a camera configured to capture a target image including a marker; at least one processor; and a memory storing one or more programs configured to be executed by the at least one processor, the one or more programs including instructions of: acquiring the target image including the marker, wherein a plurality of sub-markers are distributed on the marker, features of the plurality of sub-makers are different, each of the plurality of sub-markers comprises at least one feature point, the feature comprises information of the at least one feature point of the sub-marker; determining a blocked target according to a blocked feature, wherein the blocked target is a blocked sub-marker of the plurality of sub-markers in the target image; acquiring, as a reference, at least one blocked sub-marker determined within a preset period before a target time point, wherein the target time point is a time point at which the blocked target is determined to be blocked, determining blocking change information according to the reference and the blocked target, the blocking change information comprises target change information indicating a change trend from the reference to the blocked target, generating a control instruction for the virtual object according to the blocking change information; generating a sub-control instruction for the virtual object according to blocked information of the blocked target; and controlling the displayed virtual object based on the control instruction and the sub-control instruction. ### Claim 2 (depends on 1) The terminal device of claim 1, wherein the determining the blocked target comprises: determining the blocked sub-marker in the target image; and determining, as the blocked target, a blocked feature point from the at least one feature point in the blocked sub-marker, wherein the blocked information is obtained based on pixel points of a blocked portion of the blocked feature point. ### Claim 3 (depends on 2) The terminal device of claim 2, wherein the sub-control instruction corresponding to the blocked feature point and the control instruction corresponding to the blocked sub-marker is a same type of control instructions. ### Claim 4 (depends on 3) The terminal device of claim 3, wherein the control instruction is configured to enlarge the virtual object, and the sub-control instruction is configured to enlarge the virtual object with a preset multiple. ### Claim 5 (depends on 2) The terminal device of claim 2, wherein the sub-control instruction corresponding to the blocked feature point and the control instruction corresponding to the blocked sub-marker are different types of control instructions. ### Claim 6 (depends on 5) The terminal device of claim 5, wherein the control instruction is configured to enlarge the virtual object, and the sub-control instruction is configured to rotate the virtual object by preset degrees. ### Claim 7 (depends on 2) The terminal device of claim 2, wherein the marker is a pattern comprises a topology structure, referring to a communication relationship between the sub-markers and the at least one feature point of the marker. ### Claim 8 (depends on 1) The terminal device of claim 1, wherein the blocked information comprises a ratio of a number of pixel points occupied by a blocked portion of the blocked target in the target image to a number of pixel points occupied by an unblocked portion of the blocked target in the target image. ### Claim 9 (depends on 1) The terminal device of claim 1, wherein the generating the control instruction for the virtual object comprises: acquiring a position of the blocked target on the marker; determining a virtual button that matches the position; and generating the control instruction for pressing the virtual button. ### Claim 10 (depends on 1) The terminal device of claim 1, wherein the determining the blocked target comprises: receiving a start-up instruction in response to capturing an image of a start-up marker; and determining the blocked target in response to the start-up instruction. ### Claim 11 (depends on 1) The terminal device of claim 1, wherein the determining the blocked target comprises: detecting one blocked sub-marker in the target image; acquiring a blocking duration in which the one blocked sub-marker is blocked; generating a start-up instruction when the blocking duration is greater than a preset value; and determining the blocked target in response to the start-up instruction. ### Claim 12 (depends on 1) The terminal device of claim 1, wherein the control instruction is configured to enlarge or reduce the virtual object, and the sub-control instruction is configured to enlarge or reduce the virtual object with a preset multiple. ### Claim 13 (depends on 1) The terminal device of claim 1, wherein the generating a sub-control instruction according to blocked information of the blocked target comprises: determining a change duration indicating a duration between a time point at which the reference is determined to be blocked and the target time point, and generating the sub-control instruction according to the change duration; and wherein the controlling the displayed virtual object based on the control instruction and the sub-control instruction comprises: controlling the displayed virtual object based on the blocking change information and the change duration. ### Claim 14 (depends on 1) The terminal device of claim 1, wherein an imaging parameter of the marker is adjusted, before the acquiring of the target image including the marker. ### Claim 15 A method of controlling virtual content, comprising: acquiring a target image including a marker, wherein a plurality of sub-markers are distributed on the marker, features of the plurality of sub-markers are different, and there is a preset correspondence between the features and identifications of the sub-markers, each of the plurality of sub-markers comprises at least one feature point, the feature comprises information of the at least one feature point of the sub-marker; determining a blocked target, wherein the blocked target is a blocked sub-marker of the plurality of sub-markers in the target image; acquiring, as a reference, at least one blocked sub-marker determined within a preset period before a target time point, wherein the target time point is a time point at which the blocked target is determined to be blocked, determining blocking change information according to the reference and the blocked target, the blocking change information comprises target change information indicating a change trend from the reference to the blocked target, generating a control instruction for the virtual object according to the blocking change information; determining a change duration indicating a duration between a time point at which the reference is determined to be blocked and the target time point, generating a sub-control instruction for the virtual object according to the change duration; and controlling a displayed virtual object based on the control instruction and the sub-control instruction. ### Claim 16 (depends on 15) The method of claim 15, wherein the determining the blocked target comprises: determining the blocked sub-marker in the target image; and determining, as the blocked target, a blocked feature point from the at least one feature point in the blocked sub-marker, wherein the blocked information is obtained based on pixel points of a blocked portion of the blocked feature point. ### Claim 17 (depends on 15) The method of claim 15, wherein the blocked information comprises a ratio of a number of pixel points occupied by a blocked portion of the blocked target in the target image to a number of pixel points occupied by an unblocked portion of the blocked target in the target image. ### Claim 18 (depends on 15) The method of claim 15, wherein the generating the control instruction for the virtual object comprises: acquiring a position of the blocked target on the marker; determining a virtual button that matches the position; and generating the control instruction for pressing the virtual button. ### Claim 19 (depends on 15) The method of claim 15, wherein the determining the blocked target comprises: receiving a start-up instruction in response to capturing an image of a start-up marker; and determining the blocked target in response to the start-up instruction. ### Claim 20 A non-transitory computer readable medium storing one or more programs configured to be executed by one or more processors of a terminal device, the one or more programs comprising instructions for: adjusting an imaging parameter of the marker; acquiring a target image including a marker according to the adjusted imaging parameter, wherein a plurality of sub-markers are distributed on the marker, features of the plurality of sub-markers are different, each of the plurality of sub-markers comprises at least one feature point, the feature comprises information of the at least one feature point of the sub-marker; determining a blocked target, wherein the blocked target is a blocked sub-marker of the plurality of sub-markers in the target image; acquiring, as a reference, at least one blocked sub-marker determined within a preset period before a target time point, wherein the target time point is a time point at which the blocked target is determined to be blocked, determining blocking change information according to the reference and the blocked target, the blocking change information comprises target change information indicating a change trend from the reference to the blocked target, generating a control instruction for the virtual object according to the blocking change information; generating a sub-control instruction for the virtual object according to blocked information of the blocked target; and controlling the displayed virtual object based on the control instruction and the sub-control instruction.