Back to results
US 12,608,762 B2Granted
Demosaicking and super-resolution using wobulation
AssigneeVarjo Technologies Oy
InventorMikko Ollila
Life of this patent
Estimating the term…
Abstract
An imaging system includes an image sensor, a wobulator employed to perform sub-pixel shifts; and processor(s) configured to: obtain, in a cycle, two or three sub-images from the image sensor; control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of: same cycle, different cycles, wherein at least one of step sizes is X pixel, wherein X is a fraction lying between 0 and 1, at least one other of step sizes is Y pixels, wherein Y is an integer lying in a range from 1 to Z, Z being equal to a number of pixels of same colour lying along a direction of sub-pixel shift in a smallest repeating M×N array (204a-b) in the image sensor; and process the two or three sub-images to generate image(s).
Claims18
An imaging system comprising:
- an image sensor;
- a wobulator that is to be employed to perform sub-pixel shifts when capturing images with the image sensor;
- and
- at least one processor configured to:
- obtain, in a cycle, two or three sub-images from the image sensor;
- control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- process the two or three sub-images of said cycle, to generate at least one image;
- wherein the at least one image comprises two or three images corresponding to the two or three sub-images, wherein the at least one processor is configured to:
- obtain information indicative of a gaze direction of a user's eye;
- identify a gaze position within a photosensitive surface of the image sensor, based on the gaze direction of the user's eye;
- select a gaze region of the photosensitive surface based on the gaze position, wherein the gaze region of the photosensitive surface includes and surrounds the gaze position, while a peripheral region of the photosensitive surface surrounds the gaze region;
- process parts of the two or three sub-images that have been read out from the gaze region of the photosensitive surface, to generate a single gaze image segment corresponding to the gaze region;
- process respective parts of the two or three sub-images that have been read out from the peripheral region of the photosensitive surface, to generate respective ones of two or three peripheral image segments corresponding to the peripheral region;
- and
- combine each of the two or three peripheral image segments with the single gaze image segment, to generate a respective one of the two or three images.
14 more elements
An imaging system comprising:
- an image sensor;
- a wobulator that is to be employed to perform sub-pixel shifts when capturing images with the image sensor;
- and
- at least one processor configured to:
- obtain, in a cycle, two or three sub-images from the image sensor;
- control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- process the two or three sub-images of said cycle, to generate at least one image;
- wherein the at least one processor is configured to:
- obtain information indicative of a gaze direction of a user's eye;
- determine, based on the gaze direction of the user's eye obtained within a given time period, at least one of:
- a gaze velocity, a gaze acceleration, a predicted gaze direction, of the user's eye;
- detect, based on the at least one of:
- the gaze velocity, the gaze acceleration, the predicted gaze prediction, whether the user's gaze has been fixated and a gaze region of a photosensitive surface of the image sensor where the user's gaze has been fixated;
- and
- perform operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when it is detected that the user's gaze has been fixated.
14 more elements
An imaging system comprising:
- an image sensor;
- a wobulator that is to be employed to perform sub-pixel shifts when capturing images with the image sensor;
- and
- at least one processor configured to:
- obtain, in a cycle, two or three sub-images from the image sensor;
- control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- process the two or three sub-images of said cycle, to generate at least one image;
- the imaging system further comprising pose-tracking means, wherein the at least one processor is configured to:
- process pose-tracking data, obtained from the pose-tracking means, to determine a pose of the image sensor;
- determine a rate at which the pose of the image sensor changes;
- and
- perform operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when the rate at which the pose of the image sensor changes does not exceed a predefined threshold.
11 more elements
An imaging system comprising:
- an image sensor;
- a wobulator that is to be employed to perform sub-pixel shifts when capturing images with the image sensor;
- and
- at least one processor configured to:
- obtain, in a cycle, two or three sub-images from the image sensor;
- control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- process the two or three sub-images of said cycle, to generate at least one image;
- wherein the at least one processor is configured to:
- analyse a sequence of images captured using the image sensor, to detect a movement of objects represented in said sequence of images;
- and
- perform operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when a rate at which the objects move does not exceed a predefined threshold.
10 more elements
A method comprising:
- obtaining, in a cycle, two or three sub-images from an image sensor;
- controlling a wobulator to perform one or two sub-pixel shifts when capturing the two or three sub-images during said cycle, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- processing the two or three sub-images of said cycle, to generate at least one image;
- wherein the at least one image comprises two or three images corresponding to the two or three sub-images, and wherein the method further comprises:
- obtaining information indicative of a gaze direction of a user's eye;
- identifying a gaze position within a photosensitive surface of the image sensor, based on the gaze direction of the user's eye;
- selecting a gaze region of the photosensitive surface based on the gaze position, wherein the gaze region of the photosensitive surface includes and surrounds the gaze position, while a peripheral region of the photosensitive surface surrounds the gaze region;
- processing parts of the two or three sub-images that have been read out from the gaze region of the photosensitive surface, for generating a single gaze image segment corresponding to the gaze region;
- processing respective parts of the two or three sub-images that have been read out from the peripheral region of the photosensitive surface, for generating respective ones of two or three peripheral image segments corresponding to the peripheral region;
- and
- combining each of the two or three peripheral image segments with the single gaze image segment, for generating a respective one of the two or three images.
10 more elements
A method comprising:
- obtaining, in a cycle, two or three sub-images from an image sensor;
- controlling a wobulator to perform one or two sub-pixel shifts when capturing the two or three sub-images during said cycle, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- processing the two or three sub-images of said cycle, to generate at least one image;
- obtaining information indicative of a gaze direction of a user's eye;
- determining, based on the gaze direction of the user's eye obtained within a given time period, at least one of:
- a gaze velocity, a gaze acceleration, a predicted gaze direction, of the user's eye;
- detecting, based on the at least one of:
- the gaze velocity, the gaze acceleration, the predicted gaze prediction, whether the user's gaze has been fixated and a gaze region of a photosensitive surface of the image sensor where the user's gaze has been fixated;
- and
- performing operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when it is detected that the user's gaze has been fixated.
8 more elements
A method comprising:
- obtaining, in a cycle, two or three sub-images from an image sensor;
- controlling a wobulator to perform one or two sub-pixel shifts when capturing the two or three sub-images during said cycle, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- processing the two or three sub-images of said cycle, to generate at least one image;
- processing pose-tracking data, obtained from pose-tracking means, to determine a pose of the image sensor;
- determining a rate at which the pose of the image sensor changes;
- and
- performing operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when the rate at which the pose of the image sensor changes does not exceed a predefined threshold.
5 more elements
A method comprising:
- obtaining, in a cycle, two or three sub-images from an image sensor;
- controlling a wobulator to perform one or two sub-pixel shifts when capturing the two or three sub-images during said cycle, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- processing the two or three sub-images of said cycle, to generate at least one image;
- analysing a sequence of images captured using the image sensor, to detect a movement of objects represented in said sequence of images;
- and
- performing operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when a rate at which the objects move does not exceed a predefined threshold.
4 more elements
An imaging system comprising:
- an image sensor;
- a wobulator that is to be employed to perform sub-pixel shifts when capturing images with the image sensor;
- and
- at least one processor configured to:
- obtain, in a cycle, two or three sub-images from the image sensor;
- control the wobulator to perform, during said cycle, one or two sub-pixel shifts, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- process the two or three sub-images of said cycle, to generate at least one image;
- wherein the at least one processor is configured to:
- obtain information indicative of a gaze direction of a user's eye;
- determine, based on the gaze direction of the user's eye obtained within a given time period, at least one of:
- a gaze velocity, a gaze acceleration, a predicted gaze direction, of the user's eye;
- detect, based on the at least one of:
- the gaze velocity, the gaze acceleration, the predicted gaze prediction, whether the user's gaze has been fixated and a gaze region of a photosensitive surface of the image sensor where the user's gaze has been fixated;
- and
- perform operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when it is detected that the user's gaze has been fixated;
- wherein the step of processing the two or three sub-images is performed for the gaze region only.
15 more elements
A method comprising:
- obtaining, in a cycle, two or three sub-images from an image sensor;
- controlling a wobulator to perform one or two sub-pixel shifts when capturing the two or three sub-images during said cycle, wherein step sizes of sub-pixel shifts vary within at least one of:
- a same cycle, different cycles, wherein at least one of the step sizes is X pixel, wherein X is a fraction that lies between 0 and 1, at least one other of the step sizes is Y pixels, wherein Y is an integer that lies in a range from 1 to Z, Z being equal to a number of pixels of a same colour that lie along a direction of a given sub-pixel shift in a smallest repeating M×N array in the image sensor;
- and
- processing the two or three sub-images of said cycle, to generate at least one image;
- obtaining information indicative of a gaze direction of a user's eye;
- determining, based on the gaze direction of the user's eye obtained within a given time period, at least one of:
- a gaze velocity, a gaze acceleration, a predicted gaze direction, of the user's eye;
- detecting, based on the at least one of:
- the gaze velocity, the gaze acceleration, the predicted gaze prediction, whether the user's gaze has been fixated and a gaze region of a photosensitive surface of the image sensor where the user's gaze has been fixated;
- performing operations of obtaining the two or three sub-images in the cycle, controlling the wobulator to perform the one or two sub-pixel shifts during the cycle, and processing the two or three sub-images to generate the at least one image, only when it is detected that the user's gaze has been fixated;
- wherein the step of processing the two or three sub-images is performed for the gaze region only.
9 more elements
Dependencies are inferred from the claim text.