# US 10,122,888 B2 · Information processing system, terminal device and method of controlling display of secure data using augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,888 B2 - Document ID: US10122888B2 - Status: Granted - Kind: B2 - Application number: 15/292,622 - Filed: 2016-10-13 - Published: 2017-04-27 - Granted: 2018-11-06 - Earliest claimed: 2015-10-26 - Assignees: Ricoh Company, Ltd. - Applicants: Yuuichiroh Tsujiguchi - Inventors: Yuuichiroh Tsujiguchi - Web page: https://patentgrep.com/patent/US10122888B2 ## CPC - H04N 1/32133: {on the same paper sheet, e.g. a facsimile page header} - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06F 3/147: using display panels - G06F 21/6209: {to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself} - G06F 21/84: output devices, e.g. displays or monitors - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/225: {based on a marking or identifier characterising the area} - G09G 5/026: {Control of mixing and/or overlay of colours in general (G09G5/022 and G09G5/024 take precedence)} - G09G 5/14: Display of multiple viewports - H04N 1/00251: {with an apparatus for taking photographic images, e.g. a camera} - H04N 1/32208: {involving changing the magnitude of selected pixels, e.g. overlay of information or super-imposition (H04N1/32219, H04N1/32224 take precedence)} - G06F 2221/2113: Multi-level security, e.g. mandatory access control - G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels - G09G 2358/00: Arrangements for display data security - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract An information processing system includes a server, and a terminal device communicable with the server via a network. The terminal device includes a display, an image capturing device to capture a marker image and a first image formed on a recording medium, the marker image indicating a condition of displaying information on the display, and circuitry to extract an identifier of the marker image from the captured marker image, transmit identification information of the terminal device or user and the extracted identifier of the captured marker image to the server, receive data of a second image associated with the identification information of the terminal device or user and the identifier of the marker image from the server, and superimpose the second image over the first image to cause the display to display the second image and the first image while the recording medium is being captured. ## Claims ### Claim 1 An information processing system comprising: a server; and a terminal device communicably coupled to the server via a network, the terminal device including: a display, an image capturing device configured to capture a recording medium including a marker image and a first image, the marker image indicating a condition of displaying information on the display, and circuitry configured to extract an identifier of the marker image from the marker image captured by the image capturing device, transmit, to the server, the identifier of the marker image and at least one of first identification information for identifying the terminal device, and second identification information for identifying a user of the terminal device, receive a second image associated with the identifier of the marker image and the at least one of the first identification information and the second identification information from the server, the second image being variably set depending on at least one of first information clearance level of the terminal device and second information clearance level of the user of the terminal device; superimpose the second image over the first image to cause the display to collectively display the second image and the first image while the recording medium is being captured by the image capturing device; and set a number of times the extraction, the transmission, the reception, and the superimposition are repeated based on a computing capability of the circuitry of the terminal device. ### Claim 2 (depends on 1) The information processing system of claim 1, wherein the server includes a memory configured to store the second image in association with the identifier of the marker image and the at least one of the first identification information and the second identification information; and circuitry configured to receive the identifier of the marker image and the at least one of the first identification information and second the identification information from the terminal device; identify the second image that matches the at least one of the first information clearance level and the second information clearance level in the memory based on the identifier of the marker image and the at least one of the first identification information and the second identification information received from the terminal device; and transmit the second image to the terminal device. ### Claim 3 (depends on 1) The information processing system of claim 1, further comprising: an information processing apparatus; and an image forming apparatus, wherein the information processing apparatus includes circuitry configured to generate the first image and the marker image, and transmit the first image to the image forming apparatus. ### Claim 4 (depends on 3) The information processing system of claim 3, wherein the image forming apparatus includes circuitry configured to receive the first image and the marker image, and print the first image and the marker image on the recording medium. ### Claim 5 (depends on 3) The information processing system of claim 3, wherein the information processing apparatus further includes a display, and wherein the circuitry of the information processing apparatus is further configured to display a warning message on the display indicating that the marker image is being currently used when the marker image is already in use. ### Claim 6 (depends on 3) The information processing system of claim 3, wherein the information processing apparatus includes further an interface, and wherein, the circuitry of the information processing apparatus is further configured to receive an instruction to update the marker image when the first image and the marker image are to be generated. ### Claim 7 (depends on 3) The information processing system of claim 3, wherein the circuitry of the information processing apparatus is further configured to overwrite information currently set for the marker image with new information when the marker image is already in use. ### Claim 8 (depends on 3) The information processing system of claim 3, wherein the information processing apparatus further includes a display, wherein when the circuitry of the information processing apparatus is further configured to display a warning message on the display requesting re-arrangement of the marker image when a placement position of the marker image overlaps the first image. ### Claim 9 (depends on 3) The information processing system of claim 3, wherein when the circuitry of the information processing apparatus is further configured to search for an image-vacant region, and places the marker image within the image-vacant region when a placement position of the marker image overlaps the first image. ### Claim 10 (depends on 1) The information processing system of claim 1, wherein the display of the terminal device displays an error message when the terminal device does not receive the second image from the server. ### Claim 11 (depends on 1) The information processing system of claim 1, wherein the terminal device includes an interface configured to receive at least one of the first identification information and the second identification information. ### Claim 12 (depends on 2) The information processing system of claim 2, wherein the memory of the server is further configured to store a first security table associating the first information clearance level to the first identification information, and the circuitry of the server is further configured to change at least one of the first information clearance level and the first identification information stored in the first security table when the server receives a corresponding request, wherein the memory of the server is further configured to stores a second security table associating the second information clearance level to the second identification information, and the circuitry of the server is further configured to change at least one of the second information clearance level and the second identification information stored in the second security table when the server receives a corresponding request. ### Claim 13 (depends on 2) The information processing system of claim 2, wherein the circuitry of the server is further configured to change the second image. ### Claim 14 (depends on 1) The information processing system of claim 1, wherein the circuitry of the terminal device is further configured to set the number of times based on a corresponding designation. ### Claim 15 (depends on 1) The information processing system of claim 1, wherein the display of the terminal device is configured to display a message requesting the marker image be set within a capturing range of the image capturing device, when the marker image is not within a capturing range of the image capturing device. ### Claim 16 A terminal device communicably Coupled to a server via a network, the terminal device comprising: a display; an image capturing device configured to capture recording medium including a marker image and a first image, the marker image indicating a condition of displaying information on the display; and circuitry configured to extract an identifier of the marker image from the marker image captured by the image capturing device, transmit, to the server, the identifier of the marker image and at least one of first identification information for identifying the terminal device, and second identification information for identifying a user of the terminal device to the server, receive a second image associated with the identifier of the marker image and the at least one of the first identification information and the second identification information from the server, the second image being variably set depending on at least one of an information clearance level of the terminal device and an information clearance level of the user of the terminal device; superimpose the second image over the first image to cause the display to collectively display the second image and the first image while the recording medium is being captured by the image capturing device; and set a number of times the extraction, the transmission, the reception, and the superimposition are repeated based on a computing capability of the circuitry of the terminal device. ### Claim 17 (depends on 16) The terminal device of claim 16, wherein the second image displayed on the display is variably set depending on at least one of the information clearance level of the terminal device and the information clearance level of the user of the terminal device. ### Claim 18 (depends on 16) The terminal device of claim 16, wherein the circuitry of the terminal device is further configured to set the number of times based on a corresponding designation. ### Claim 19 A method of processing data by a terminal device communicably coupled to a server via a network, the method comprising: capturing a recording medium including a marker image and a first image, the marker image indicating a condition of displaying information on a display; extracting an identifier of the marker image from the marker image in response to the capturing; transmitting, to the server, the identifier of the marker image and at least one of first identification information for identifying the terminal device, and second identification information for identifying a user of the terminal device to the server; receiving a second image associated with the identifier of the marker image and the at least one of the first identification information and the second identification information from the server, the second image being variably set depending on at least one of an information clearance level of the terminal device and an information clearance level of the user of the terminal device; superimposing the second image over the first image to cause the display to collectively display the second image and the first image while the recording medium is being captured; and setting a number of times the extracting, the transmitting, the receiving, and the superimposing are repeated based on a computing capability of the circuitry of the terminal device. ### Claim 20 (depends on 19) The method of claim 19, wherein the setting includes setting the number of times based on a corresponding designation.