# US 10,503,812 B2 · Method and apparatus for materials in different screen sizes using an imaging sensor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,812 B2 - Document ID: US10503812B2 - Status: Granted - Kind: B2 - Application number: 15/079,911 - Filed: 2016-03-24 - Published: 2016-07-14 - Granted: 2019-12-10 - Earliest claimed: 2009-10-14 - Assignees: IPLContent, LLC - Applicants: Chi Fai Ho; Peter P. Tong - Inventors: Chi Fai Ho; Peter P. Tong - Web page: https://patentgrep.com/patent/US10503812B2 ## CPC - G06F 40/114: Pagination - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor - G06F 15/0291: {for reading, e.g. e-books (constructional details of portable computers G06F1/1613)} - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/106: Display of layout of documents; Previewing - G06F 40/123: Storage facilities - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract One embodiment includes a mobile device, having at least an imaging sensor, applicable for reading materials, with at least a section including text linked to an illustration. Annotations could be added to the materials. The device can layout the section by keeping the text with the illustration to be displayed in one screen. Another embodiment includes materials with a text sub file with texts, an illustration sub file with illustrations, and a logic sub file with rules on displaying the materials. Either the text or the illustration sub file includes position information linking at least an illustration to a corresponding piece of text. Yet another embodiment includes materials with a logic sub file that can analyze an attribute of, and provide a response to, a user. ## Claims ### Claim 1 A mobile device applicable for a user to read materials, the mobile device comprising: a controller; a display; an imaging sensor; and a storage medium to store at least a piece of materials, with the piece of materials comprising: a text sub file including at least a piece of text; an illustration sub file including at least an illustration; and a logic sub file including a plurality of instructions executable by at least the controller to result in the mobile device to: layout at least a screen of a first size, with the screen including at least an area of the piece of materials, at least by keeping the piece of text with the illustration to be presented on the device, wherein to layout includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen; and layout again but for at least a screen of a second size, with the screen including at least the area of the piece of materials, at least by keeping the piece of text with the illustration to be presented on the device, wherein to layout includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen, with the second size being different from the first size, wherein the plurality of instructions executable by at least the controller result in the mobile device to: analyze at least an attribute of the user based on monitoring the user using at least the imaging sensor; identify at least the area of the piece of materials based on at least the analyze; access the area of the piece of materials; and present the area of the piece of materials on the display. ### Claim 2 (depends on 1) A mobile device as recited in claim 1, wherein the mobile device is configured to enable the user to enter an annotation into the device for at least a portion of the piece of materials. ### Claim 3 (depends on 1) A mobile device as recited in claim 1, wherein the plurality of instructions executable by at least the controller result in the mobile device to access an annotation for at least a portion of the piece of materials. ### Claim 4 (depends on 3) A mobile device as recited in claim 3, wherein the annotation is wirelessly received by the mobile device, and wherein the annotation is received by the mobile device not together with the piece of materials. ### Claim 5 (depends on 1) A mobile device as recited in claim 1, wherein the plurality of instructions executable by at least the controller result in the mobile device to: access an annotation with position information regarding the annotation; and layout the annotation with a portion of the piece of materials using at least the position information. ### Claim 6 (depends on 3) A mobile device as recited in claim 3, wherein the plurality of instructions executable by at least the controller result in the mobile device to: determine if there is sufficient space in a screen of the display to present the annotation with the portion of the piece of materials, and layout the annotation with the portion of the piece of materials in the screen if the determine determines that there is sufficient space; and layout the annotation with at least a part of the portion of the piece of materials for another screen if the determine determines that there is insufficient space. ### Claim 7 (depends on 1) A mobile device as recited in claim 1, wherein the plurality of instructions executable by at least the controller result in the mobile device to access a plurality of annotations for the user to identify one or more annotations to be presented on the display with at least a part of the piece of materials. ### Claim 8 (depends on 3) A mobile device as recited in claim 3, wherein the plurality of instructions executable by at least the controller result in the mobile device to: present a notification regarding the annotation on the display; and present the annotation on the display if the mobile device receives a user action regarding the notification. ### Claim 9 (depends on 8) A mobile device as recited in claim 8, wherein the notification includes a symbol. ### Claim 10 (depends on 8) A mobile device as recited in claim 8, wherein the notification includes highlighting at least a part of the portion. ### Claim 11 (depends on 8) A mobile device as recited in claim 8, wherein the plurality of instructions executable by at least the controller result in the mobile device to layout at least the annotation to be presented on the display if the mobile device receives the user action regarding the notification. ### Claim 12 (depends on 3) A mobile device as recited in claim 3, wherein the portion includes at least the piece of text, and wherein the plurality of instructions executable by at least the controller result in the mobile device to alert the user of the annotation by providing at least a notification close to the piece of text, such that at least both the notification and the piece of text can be in the same screen. ### Claim 13 (depends on 1) A mobile device as recited in claim 1, wherein to read materials includes to watch a video. ### Claim 14 (depends on 1) A mobile device as recited in claim 1, wherein the illustration includes a video. ### Claim 15 (depends on 1) A mobile device as recited in claim 1, wherein the illustration includes an animation. ### Claim 16 (depends on 3) A mobile device as recited in claim 3, wherein the annotation includes a video. ### Claim 17 (depends on 3) A mobile device as recited in claim 3, wherein the plurality of instructions executable by at least the controller result in the mobile device to: present on the display a notification regarding the annotation; receive a user action regarding the notification, with the user action indicating to present the annotation on the display; determine if there is sufficient space in a screen of the display to present the annotation with the portion of the piece of materials, and layout the annotation with the portion of the piece of materials in the screen if the determine determines that there is sufficient space; and layout the annotation with at least a part of the portion of the piece of materials for another screen if the determine determines that there is insufficient space. ### Claim 18 (depends on 17) A mobile device as recited in claim 17, wherein the notification is selected from a list including a symbol and highlighting at least a part of the portion, and wherein to read materials includes to watch a video. ### Claim 19 (depends on 18) A mobile device as recited in claim 18, wherein the annotation includes a video, and wherein the illustration includes an animation. ### Claim 20 (depends on 2) A mobile device as recited in claim 2, wherein the plurality of instructions executable by at least the controller result in the mobile device to: determine if there is sufficient space in a screen of the display to present the annotation with the portion of the piece of materials, and layout the annotation with the portion of the piece of materials for the screen if the determine determines that there is sufficient space; and layout the annotation with at least a part of the portion of the piece of materials for another screen if the determine determines that there is insufficient space, wherein to read materials includes to watch a video, and wherein the annotation includes a video. ### Claim 21 (depends on 1) A mobile device as recited in claim 1, wherein the imaging sensor is configured to monitor at least an image regarding the head of the user for analysis. ### Claim 22 A non-transitory computer readable medium of a mobile device, the non-transitory computer readable medium including at least a piece of materials stored therein, the piece of materials comprising a text sub file including at least a piece of text, an illustration sub file including at least an illustration, and a logic sub file including executable computer program code, applicable to present to a user via the mobile device, the mobile device including at least an imaging sensor, the executable computer program code comprising: computer program code to analyze at least an attribute of the user based on monitoring the user using at least the imaging sensor; computer program code to identify an area of the piece of materials based on at least the analyze, with the area including at least the piece of text and the illustration; computer program code to layout at least a screen of a first size, with the screen including at least the area of the piece of materials, at least by keeping the piece of text with the illustration to be presented, wherein to layout includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen; and computer program code to layout again but for at least a screen of a second size, with the screen including at least the area of the piece of materials, at least by keeping the piece of text with the illustration to be presented, wherein to layout includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen, with the second size being different from the first size. ### Claim 23 (depends on 22) A non-transitory computer readable medium as recited in claim 22, wherein the executable computer program code comprises computer program code to enable the user to enter an annotation into the device for at least a portion of the piece of materials. ### Claim 24 (depends on 22) A non-transitory computer readable medium as recited in claim 22, wherein the executable computer program code comprises computer program code to access an annotation for at least a portion of the piece of materials. ### Claim 25 (depends on 24) A non-transitory computer readable medium as recited in claim 24, wherein the executable computer program code comprises: computer program code to access position information regarding the annotation; and computer program code to layout the annotation with the portion of the piece of materials using at least the position information. ### Claim 26 (depends on 22) A non-transitory computer readable medium as recited in claim 22, wherein the executable computer program code comprises computer program code to access a plurality of annotations for the user to identify one or more annotations to be presented with at least a part of the piece of materials. ### Claim 27 (depends on 22) A non-transitory computer readable medium as recited in claim 22, wherein the attribute of the user is related to the head of the user. ### Claim 28 (depends on 22) A non-transitory computer readable medium as recited in claim 22, wherein the attribute of the user is related to a facial feature of the user. ### Claim 29 A method to present to a user via a mobile device, with the mobile device comprising at least a display, at least a sensor applicable to monitor images, and at least a storage medium to store at least a piece of materials, with the piece of materials comprising a text sub file including at least a piece of text, and an illustration sub file including at least an illustration, the method, implemented by the mobile device, comprising: analyzing at least an attribute of the user based on monitoring the user using at least the sensor; identifying an area of the piece of materials based on at least the analyzing, with the area including at least the piece of text and the illustration; laying out at least a screen of a first size, with the screen including at least the area of the piece of materials, at least by keeping the piece of text with the illustration to be presented on the device, wherein to laying out includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen; and laying out again but for at least a screen of a second size, with the screen including at least the area of the piece of materials, at least by keeping the piece of text with the illustration to be presented on the device, wherein to laying out includes to calculate at least one position of the piece of text in the screen and to calculate at least one position of the illustration in the screen, to fit at least both the piece of text and the illustration into the screen, with the second size being different from the first size. ### Claim 30 (depends on 29) A method as recited in claim 29 comprising accessing an annotation for at least a portion of the piece of materials. ### Claim 31 (depends on 30) A method as recited in claim 30, wherein the annotation is entered by the user into the device. ### Claim 32 (depends on 30) A method as recited in claim 30, wherein the annotation is wirelessly received by the mobile device. ### Claim 33 (depends on 30) A method as recited in claim 30 comprising: determining if there is sufficient space in at least the screen of the first size to present the annotation with the portion of the piece of materials, and laying out the annotation with the portion of the piece of materials in the screen if the determining determines that there is sufficient space; and laying out the annotation with at least a part of the portion of the piece of materials for another screen if the determining determines that there is insufficient space. ### Claim 34 (depends on 30) A method as recited in claim 30, wherein the portion includes at least the piece of text, and wherein the method comprises alerting the user of the annotation by providing at least a notification close to the piece of text, such that at least both the notification and the piece of text can be in the same screen. ### Claim 35 (depends on 30) A method as recited in claim 30 comprising: accessing a notification for the user regarding the annotation; receiving a user action regarding the notification, with the user action indicating to present the annotation; and determining if there is sufficient space in at least the screen of the first size to present the annotation with the portion of the piece of materials, and laying out the annotation with the portion of the piece of materials in at least the screen of the first size if the determining determines that there is sufficient space. ### Claim 36 (depends on 35) A method as recited in claim 35 comprising laying out the annotation with at least a part of the portion of the piece of materials for another screen if the determining determines that there is insufficient space.